Processing prosodic boundaries in natural and hummed speech. An fMRI study
Anja K. Ischebeck1,2, Angela D. Friederici2, Kai Alter2,3
1
2
Innsbruck Medical University, Clinical Department of Neurology, Austria
Max Planck Institute of Human Cognitive and Brain Sciences, Leipzig, Germany 3
Newcastle University Medical School, Newcastle upon Tyne, U.K.
Corresponding author: Anja Ischebeck, PhD Innsbruck Medical University Anichstrasse 35 6020 – Innsbruck (Austria) tel.: +43 512 504 23661 e-mail:
[email protected]
Abstract Speech contains prosodic cues such as pauses between different phrases of a sentence. These intonational phrase boundaries (IPBs) elicit a specific component in ERP-studies, the socalled closure-positive-shift (CPS). The aim of the present fMRI study is to identify the neural correlates of this prosody-related component in sentences containing segmental and prosodic information (natural speech) and hummed sentences only containing prosodic information. Sentences with two IPBs both in normal and hummed speech activated the middle superior temporal gyrus, the Rolandic operculum and the gyrus of Heschl more strongly than sentences with one IPB. The results from a region of interest (ROI) analysis of auditory cortex and auditory association areas suggest that the posterior Rolandic operculum, in particular, supports the processing of prosodic information. A comparison of natural speech and hummed sentences revealed a number of left-hemispheric areas within the temporal lobe as well as in the frontal and parietal lobe that were activated more strongly for natural speech than for hummed sentences. These areas constitute the neural network for the processing of natural speech. The finding that no area was activated more strongly for hummed sentences compared to natural speech suggests that prosody is an integrated part of natural speech.
2
Introduction The speech melody of an utterance can carry information that is critically important to understand the meaning of a sentence (see, for a review, Frazier et al., 2006; Friederici & Alter, 2004). In intonational languages such as German, Dutch, and English, prosodic information on sentence-level is mainly conveyed, among others, by the pitch contour of an utterance and the presence of speech pauses. Sentences usually contain one or more major intonational phrases (IPh; Selkirk, 1995) that can be separated by speech pauses. Syntactically relevant speech breaks are also referred to as intonational phrase boundaries (IPB). In studies using an event-related brain potential (ERP) paradigm IPBs were observed to give rise to a positive shift in the EEG-signal that is referred to as the closure positive shift or CPScomponent (Steinhauer et al., 1999). This component has been interpreted as being specifically related to the prosodic information contained in IPBs, as it has been observed using sentence materials that lacked semantic and syntactic information, such as pseudoword sentences (Pannekamp et al., 2005), filtered speech materials (Steinhauer & Friederici, 2001) or hummed speech (Pannekamp et al., 2005). The present fMRI study attempts to identify the brain regions that are involved in the processing of intonational phrase boundaries (IPBs). IPBs are often employed by the speaker to clarify the structure of an otherwise syntactically ambiguous sentence. A sentence like 'Before Ben starts # the day dreaming has to stop' has a different meaning than 'Before Ben starts the day # dreaming has to stop' (# indicating a break). Syntactically relevant speech breaks are also referred to as intonational phrase boundaries (IPB). IPBs often separate major intonational phrases and correspond to major syntactic boundaries (Cooper & Paccia-Cooper, 1981). They represent a high level in the so-called prosodic hierarchy (Selkirk, 1995, 2000; Nespor & Vogel, 1983). In psycholinguistic experiments, IPBs were shown to help resolve ambiguities related to late closure ambiguities (Schafer et al., 2000; Grabe et al., 1994).
3
Experimental evidence suggests that humans can make use of the prosodic information contained in IPBs, that is, in the absence of semantic or syntactic information. Behaviorally, it has been shown that listeners are able to detect major prosodic boundaries in meaningless speech materials, such as, for example, reiterant speech (i.e., a sentence spoken as a string of repeated syllables while preserving the original prosodic contour) (de Rooij, 1976), spectrally scrambled and low-pass filtered speech (de Rooij, 1975, Kreiman, 1982), and hummed sentences (t'Hart & Cohen, 1990). It should be noted, that the rationale for using stimuli of this kind is based on the assumption that speech is separable into layers, such as semantics, syntax and prosody. This assumption, however, may only be an approximation, as it has been argued that the possibility to separate these layers is inherently limited (Austin, 1975; Searle, 1969). In studies using an event-related brain potential (ERP) paradigm IPBs were observed to give rise to a positive shift in the EEG-signal that is referred to as the closure positive shift or CPS-component (Steinhauer et al., 1999). Subsequent studies indicated that the CPS is specifically related to the prosodic aspects of an IPB, as it was also observed for sentence materials that were stripped of semantic information, such as pseudoword sentences, and for sentence materials with reduced or absent segmental information, such as hummed sentences (Pannekamp et al., 2005), and filtered speech materials (Steinhauer & Friederici, 2001). The CPS component is typically distributed bilaterally with a central maximum with a shift to the right for hummed sentences. However, due to the intrinsic difficulty of source localization in EEG studies, it is unclear which brain regions generate the CPS component. To our knowledge, only one imaging study so far investigated the processing of IPBs in speech (Strelnikov et al., 2006). Strelnikov et al. compared sentence materials in Russian that contained an IPB (e.g. 'To chop not # to saw', meaning One should not chop but saw) with sentences that did not contain an IPB ('Father bought him a coat'). Comparing sentences with IPB ('segmented') to sentences without IPB ('not segmented') stronger activation1 was 4
observed within the right posterior prefrontal cortex and an area within the right cerebellum. In the reverse comparison, stronger activation was observed in the gyrus of Heschl, bilaterally, and the left sylvian sulcus. However, the two types of sentence materials were used in two different tasks thus confounding stimulus type and task. Although the comparison between segmented and non-segmented speech materials very likely yielded brain areas that are relevant for prosody processing, it can not be excluded that differences due to the tasks contributed to the results. Ideally, a study investigating the processing of IPBs should compare conditions that do not differ in any other respect than the presence or absence of IPBs, keeping everything else constant. In the electrophysiological studies reviewed above (e.g., Steinhauer et al., 1999), the same task was used on sentence materials that either had one or two IPBs. The sentence materials used in the electrophysiological studies reviewed above were carefully constructed as sentence pairs with the same or similar words. It should be noted that the IPBs in these sentences were obligatory, entailing differences with regard to the syntactic and semantic structure between the two sentences of such a pair. However, these differences only play a role, when the sentence materials are presented naturally spoken, but not when their segmental content is removed by filtering or humming. Hummed speech has the advantage that it preserves the prosody of natural speech while removing major aspects of semantic and syntactic information of the utterance. Human speakers have been shown to be able to selectively preserve the original prosodic contour of an utterance. When asked to produce reiterant speech (i.e., selectively preserving the prosodic contour of a meaningful utterance by repeating a syllable), the resultant utterance preserved the prosodic aspects from normal speech, such as duration and pitch (Larkey, 1983), as well as accentuation and boundary marking (Rilliard & Aubergé, 1998). When utterances are hummed, the hummed version of an utterance has also been shown to preserve pitch contour and duration of the natural spoken version (Pannekamp et al., 2005). Most importantly, the 5
CPS component Pannekamp et al. observed at IPBs within hummed speech materials were similar to the CPS observed for natural speech, but more lateralized to the right hemisphere. Hummed speech has the additional advantage that it is a familiar human vocalization. Different from speech materials that are rendered unintelligible artificially and sound more unfamiliar (Scott et al., 2000; Meyer et al., 2004) the known unintelligibility of humming effectively prevents participants from any attempt to decipher the original speech content of the signal. In the present functional magnetic resonance imaging (fMRI) study, sentence pairs containing one or two IPBs were presented auditorily to the participants as natural speech and as hummed sentences. Similar to previous electrophysiological studies (Steinhauer et al. 1999; Pannekamp et al., 2005; Isel et al., 2005), the sentence pairs were constructed using the same or equivalent content words, except for one or two critical words. All sentences were meaningful, syntactically correct and spoken with natural prosody. To ensure variability in the sentence materials used with regard to the position of the additional IPB, two types of sentence pairs were constructed, one type with the additional IPB at an early position within the sentence (type A), the other type at a later position in the sentence (see, Table 1, for examples of the sentence materials). Similar to previous electrophysiological studies, the IPBs contained in the sentences were obligatory. We had used materials with obligatory IPBs because such materials had been investigated in previous electrophysiological studies. A CPS had been observed for sentences of type A (Steinhauer et al., 1999; Pannekamp et al., 2005) as well as for coordination structures like type B (Steinhauer, 2003). In the case of the naturally spoken sentences with obligatory IPBs, observed activation differences might in part be due to the additional semantic and syntactic differences between the sentences, rather than being solely due to the presence of IPBs. To differentiate the processing of prosody from associated syntactic and semantic processing, also a hummed version of each sentence was
6
produced by a trained speaker who was instructed to preserve the natural prosody of the original sentence. The aim of the present study was to identify the neural structures involved in the processing of sentence-level prosody by comparing sentences with two IPBs to sentences that have only one IPB. To differentiate prosodic processing from syntactic and semantic processing, differences between sentences with a different number of IPBs were investigated separately for natural speech and hummed sentences. With regard to the neural correlates of IPB processing we hypothesized that the primary auditory cortices and the auditory association areas play an important role. These areas are, among others, involved in the processing of complex auditory signals and speech (see, for a review, Griffiths et al., 2004). The superior temporal gyrus has been observed to be involved in processing of prosody (Hesling et al., 2005; Doherty et al., 2004). Activation in regions outside the temporal lobe has also been reported but appears to vary across different studies. These activations were observed to depend on the specifics of the task (Tong et al., 2005; Plante et al., 2002), the type of prosody involved (e.g., affective vs. linguistic, Wildgruber et al., 2004) and the degree of propositional information contained in the stimulus materials (Hesling et al., 2005; Tong et al., 2005; Gandour et al., 2002, 2004). It is possible, that areas outside the temporal lobe are also involved and that the auditory association areas are only part of a more extended processing network. IPBs are realized by variations in the prosody of an utterance. We therefore hypothesized that the superior temporal gyrus, among others, will show a modulation, positive or negative, due to the presence of an additional IPB. Furthermore, given the observed shift of the CPS from a bilateral distribution for natural speech to a more right hemispheric distribution for hummed sentences a more right hemispheric lateralization for prosodic aspect may be observed. Evidence from patients with brain lesions inspired a first raw hypothesis about prosody processing, namely, that the right hemisphere plays a dominant role in prosody processing. Patients with lesions within the left 7
hemisphere often suffer from aphasia, while non-aphasic patients with right hemispheric damage seem to have difficulties to perceive or produce the prosodic aspects of speech (see, for a review, Wong, 2002, but see, Perkins et al., 1996). On the basis of this first raw hypothesis, two more sophisticated classes of hypotheses have been developed. According to the acoustic or cue-dependent class of hypotheses, hemispheric dominance is determined solely by the acoustic properties of the auditory signal. Zatorre and Belin (2001), for example, suggested that the left hemisphere is specialized in processing the high frequency components that generate the vowels and consonants in speech, while the right hemisphere is specialized in processing the low frequency patterns that make up the intonational contour of a syllable or sentence (for a similar view, see, Poeppel, 2003). According to the class of functional or taskdependent hypotheses, hemispheric dominance depends on the function of prosodic information (van Lancker, 1980) or on the attentional focus required for example by an experimental task. A shift from the right hemisphere to the left is assumed to occur, when the task or function of the prosodic information contained in a speech signal involves language processing rather than prosody processing. A review of the available empirical studies suggests that lateralization is stimulus-dependent but can, in addition, vary as a function of task (Friederici & Alter, 2004). It should be noted, however, that the first raw hypothesis of right-hemispheric dominance of prosody is still under debate. If prosodic processes are mainly subserved by the right hemisphere we should find the right hemisphere being activated in both natural and hummed speech and if, moreover, prosody is represented neuronally as a separate layer a direct comparison between hummed minus natural speech should result in more left hemispheric activation for natural speech.
Methods Participants
8
Sixteen healthy right-handed healthy young adults (8 female; mean age: 26.1 years, SD: 4.44) took part in the experiment. The data of two participants were discarded from the analysis, one because of scanner malfunction, the other because of too many errors. All participants were or had been students of the University of Leipzig. They were native speakers of German with normal hearing and had no history of neurological or psychiatric illness. Volunteers were paid for their cooperation and had given written consent before the experiment. Ethical approval to the present study has been provided by the University of Leipzig. Materials Forty-eight German sentence pairs were constructed, that either had one or two intonational phrase boundaries (IPB). Ideally, the materials used should only differ with regard to the number of IPBs they contain. If possible, they should consist of the same words to ensure similar lexical retrieval processes. They should also contain the same number of words and syllables, so that they do not differ in length. To ensure some variability in the sentence materials used with regard to the position of the additional IPB, two types of sentence pairs were constructed. One type (A) had an early additional phrase boundary, one type (B) an additional phrase boundary at a later position in the sentence (see, Table 1, for examples of the sentence materials). The IPBs contained in the sentences were obligatory because such materials had been investigated in previous electrophysiological studies. Sentences of type A had been used in earlier electrophysiological studies where they were observed to elicit the CPS component (Steinhauer et al., 1999). These materials were also found to elicit the CPS component when they were low-pass filtered (Steinhauer & Friederici, 2001) or hummed (Pannekamp et al., 2005). In addition, coordination structures like the type B sentences used here also have been investigated in previous electrophysiological studies (Steinhauer, 2003) and a CPS was observed. The pairs of one and two IPB sentences of types A and B were as similar as possible with regard to the words used. In type B, the same content words were used. Also in type A, the same content words were used, with the exception of the verb, which 9
was either transitive or intransitive. To ensure comparability, the frequency and number of syllables of the verb was matched within a type A sentence pair. We had constructed these sentence materials with the aim to create conditions that do not differ by more than the aspect under scrutiny, namely, the number of IPBs. It should be noted that the sentence materials constructed for this experiment come close to this ideal but are not perfect. The type A (and type B) sentences with one or two IPBs additionally differ from each other with regard to their syntactic structure, and word order (type B). A possibility to circumvent these differences would have been the choice of sentence materials with optional IPBs rather than obligatory IPBs as in the materials used here. While obligatory IPBs correspond to major syntactic boundaries, optional IPBs are on a lower level of the prosodic hierarchy, namely the level of phonological phrases (Selkirk, 1995). They do not necessarily correspond to syntactic phrases (Truckenbrodt, 2005). Furthermore, optional IPBs depend on several factors such as speech rate and hesitations (filled or unfilled pauses) which might make them more difficult to detect for the listener. Although it is highly probable that optional IPBs also elicit a CPS component, this has not yet been investigated. The choice of our materials was primarily motivated to ensure comparability to previous electrophysiological studies where a CPS component was observed. Although not optimal, we think that our materials are reasonably comparable to allow inferences on prosodic processing. --- insert Table 1 about here --The sentences were spoken with natural prosody. Additionally, a hummed version of each sentence was recorded. All materials were recorded from a trained female native speaker of German. The speaker was instructed to speak a hummed version of the sentence after its naturally spoken version, and to take care to preserve its original prosody, speed and total number of syllables of the normal sentence. In Figure 1, spectrograms and fundamental frequency contours are given for type A and type B example sentence (hummed and naturally
10
spoken). The recorded materials were digitized (44.1 KHz) and normalized (70%) with regard to amplitude envelope to ensure an equal volume for all stimuli. --- insert Figure 1 about here --Design and Procedure The task of the participants was to indicate in a two alternative forced choice task whether a probe word presented after the sentence had been contained in the sentence or not. Probe words were chosen for each sentence and belonged to different word classes (e.g., nouns, verbs, determiners). The probes were spoken with final rising pitch, indicating a question. Of the 96 sentences in total, 72 were selected as experimental materials, 18 as probe sentences, and 6 as practice sentences. In each trial participants had to decide whether the probe word had occurred in the sentence or not. 'Yes' ('No') answers were given with the middle (index) finger of the right hand. Sentences from the experimental materials never contained the probe. The probe sentences were not analyzed. In the case of the naturally spoken probe sentences, the probe word was always contained in the sentence and a 'yes' answer was expected. In the case of the hummed experimental sentence materials, the task was very easy. As soon as the hummed sentence presentation finished, participants knew that 'no' had to be the answer. The main purpose of the task in the case of the hummed sentence materials was to have participants attend to the presentation of the sentence while it lasted. In the hummed probe sentences, one word of the hummed sentence was naturally spoken. To prevent that the participants suspended attention as soon as they detected a naturally spoken word in the hummed probe sentence, the probe word was identical to the naturally spoken word within the probe sentence in only half of the trials. This ensured that participants had to wait until the presentation of the probe word. Naturally spoken and hummed sentence materials were presented in alternating blocks. Every block consisted of 2 practice sentences, 24 experimental sentences, 6 probe-sentences, and 6 null events (i.e., trials in which no auditory stimulus was presented). The null events 11
were included to increase the efficiency of the design (Liu et al., 2001) and to provide a baseline condition for analysis. The blocks of hummed materials were of the same structure. The total experiment consisted of 6 blocks (3 hummed, 3 normal), that is, 228 trials in total. Trials were randomized with first-order transition probabilities between conditions held constant. Three randomizations were generated in total. Each trial began with the presentation of a hummed or normal sentence. Then a probe word was presented 500 ms after the offset of the sentence. After the presentation of the probe a waiting time was inserted to ensure a total inter-trial interval (ITI) of 11 seconds (see Figure 1). With a TR of 3 s and an ITI of 11 s, trial presentation and scanner triggering were synchronous every three trials. To ensure synchronization, the beginning of every third trial was synchronized using the scanner trigger. After synchronization a random waiting time of 0 to 600 ms was inserted before the presentation of the sentence. Stimulus presentation and response time recording were controlled by a computer outside the scanner using Presentation software (Neurobehavioral Systems Inc.). The experimental materials were presented over earphones within the scanner. The participants were additionally protected from scanner noise by earplugs. They reported that they had no difficulties understanding the sentences over the scanner noise. A button box compatible with the scanner environment was used to record the responses of the participants. --- insert Figure 2 about here --fMRI data acquisition All imaging data were acquired using a 3T Bruker 30/100 Medspec MR scanner. A high resolution structural scan using the 3D modified driven equilibrium Fourier transform (MDEFT) imaging technique was also obtained (128 sagittal slices, 1.5 mm thickness, inplane resolution: 0.98 x 0.98 mm, field of vision (FOV): 250 mm). Functional images were acquired using a T2* gradient echo-planar imaging (EPI) sequence. For the functional measurement a noise-reduced sequence was chosen to ensure that noise would not impair comprehension. 18 ascending axial slices per volume were continuously obtained, in the plane 12
of the anterior and posterior commissure (in-plane resolution of 3 x 3 mm, FOV: 192 mm, 5 mm thickness, 1 mm gap, echo time (TE): 40 ms, repetition time (TR): 3000 ms). The 18 slices covered the temporal lobes and part of the parietal lobes and cerebellum. According to a short questionnaire given to the participants after the experiment the noise level of the scanner did not critically impair the auditory quality and comprehensibility of the naturally spoken or the hummed stimulus materials. Functional data analysis FMRI data analysis was performed with statistical parametric mapping (SPM2) software (Wellcome Department of Cognitive Neurology, London, U.K.). The six blocks were measured as separate runs with 142 volumes each. The first two images of each functional run were discarded to ensure signal stabilization. The functional data of each participant were motion-corrected. The structural image of each participant was registered to the time series of functional images and normalized using the T1 template provided by SPM2, corresponding approximately to Talairach & Tournoux Space (Talairach & Tournoux, 1988). The functional images were normalized using the normalization parameters of the structural image and then smoothed with a full-width half-maximum (FWHM) Gaussian kernel of 12 mm. A statistical analysis on the basis of the general linear model was performed, as implemented in SPM2. Though hummed and normal sentence materials were presented in blocks, an event-related analysis was chosen. This made it possible to compare the sentences to the null events interspersed within the blocks as a baseline, as well as to discard probe trials and error trials from the analysis. The delta-function of the trial onsets per condition was convolved with the canonical form of the hemodynamic response function as given in SPM2 and its first and second temporal derivative to generate model time courses for the different conditions. Each trial was modelled in SPM2 using the beginning of the sentence as trial onset. Due to the length of the auditorily presented sentence stimuli the BOLD response was modelled with a duration of 5.345 s (average length of the sentences, hummed and natural). Errors and probe 13
sentences were modelled as separate conditions and excluded from the contrasts calculated. The functional time series was high-pass filtered with a frequency cut-off of 1/80 Hz. No global normalization was used. Motion parameters and the lengths of the individual sentences per trial were entered into the analysis as parameters of no interest. For the random effects analysis, the calculated contrast images from the first level analysis of each participant were entered into a second level analysis. ROI-Analysis: Eight regions of interest (ROIs) were created on the basis of the AAL-atlas (automated anatomical labelling (AAL) atlas, TzourioMazoyer et al., 2002): superior temporal gyrus, the Rolandic operculum, the gyrus of Heschl and the inferior frontal gyrus (pars triangularis and pars opercularis). The ROI for the superior temporal gyrus was divided into three parts, anterior (y > -15), middle (-35 < y < -15) and posterior (y < -35), and the Rolandic operculum into two parts, anterior (y > -15) and posterior (y < -15). A visualization of the temporal ROIs is given in Figure 3. The ROIanalysis was performed by averaging the effect-sizes (contrast estimates) over all voxels for the left and right ROIs per participant. --- insert Figure 3 about here --Results Behavioral Participants were excluded from further analysis if they made more than 30% errors in any condition. This led to the exclusion of one participant in addition to the one participant that was excluded due to scanner malfunction. Analyzing the data of the remaining fourteen participants yielded 144 errors (5.7 %) in total. Response times were measured from the presentation of the probe word. Only correct responses to experimental trials, not probe trials, within the interval of 200 to 2000 ms were analyzed. Response times were entered into a repeated measures analysis of variance with SPEECH TYPE (natural vs. hummed) and IPB (one vs. two IPBs) as factors. Participants reacted significantly faster to the hummed (473 ms, SD = 46.05) than to the naturally spoken sentence materials (753 ms, SD = 85.36), yielding a 14
significant main effect of SPEECH TYPE (F(1,13) = 30.64, MSE = 35691, p < 0.001). The number of IPBs did not influence response times; the main effect of IPB and the interaction SPEECH TYPE x IPB was not significant. FMRI-data: whole brain analysis Comparisons of sentence materials with a different number of IPBs. For natural speech, sentences with two IPBs activated the superior temporal gyrus, extending to the Rolandic operculum and gyrus of Heschl, bilaterally, more strongly than sentences with one IPB (Figure 4a, Table 2). In the corresponding comparison for hummed sentences, a significant cluster was observed in the left supramarginal gyrus, extending to the left superior temporal gyrus and the left gyrus of Heschl. No significant clusters were observed in the reverse comparisons, for natural speech as well as for hummed sentences. --- insert Figure 4 & Table 2 about here --Comparisons of the two basic types of materials (hummed sentences and natural speech) to baseline When natural speech was compared to baseline (null events), the strongest activations were observed within the superior temporal gyrus, bilaterally, and the SMA (Table 2). Further activations were observed within the right precentral gyrus, the right insula, the left superior parietal lobule and in the cerebellum. When hummed sentences were compared to baseline, activations were observed within the superior temporal gyrus, bilaterally, the precentral gyrus, bilaterally, the SMA and the right middle frontal gyrus, as well as the left inferior parietal lobule. Activations were also observed within the cerebellum, bilaterally. Comparisons of hummed sentences to natural speech Compared to hummed sentences, natural speech activated the frontal gyrus and middle temporal gyrus, bilaterally but with a left-hemispheric predominance, as well as the left angular gyrus and the left thalamus and caudate, more strongly (Figure 4b, Table 2). No brain area was significantly more activated for hummed sentences than for natural speech. 15
FMRI-data: ROI-Analysis To further investigate and compare the behaviour of the brain areas potentially involved in IPB processing, we conducted an ROI-analysis over eight ROIs (pars opercularis and triangularis of the inferior frontal gyrus, anterior, middle and posterior part of the superior temporal gyrus, gyrus of Heschl, and anterior and posterior Rolandic operculum). The effect sizes for each condition were averaged over all voxels contained within each ROI per side and participant. The results per ROI, hemisphere and condition are shown in Figure 5. They were entered into a repeated measures analysis of variance with ROI (8), HEMISPHERE (left vs. right), IPB (1 vs. 2 IPS), and SPEECH TYPE (naturally spoken vs. hummed) as factors. The left hemisphere was on average more strongly activated than the right hemisphere, which is reflected in a significant main effect of HEMISPHERE (F(1,13) = 49.53, MSE = 2.279, p < 0.001). Natural speech activated the brain areas investigated more strongly than hummed speech, yielding a significant main effect of SPEECH TYPE (F(1,13) = 11.93, MSE = 4.415, p < 0.01). More activation was observed for sentences containing two intonational phrase boundaries than sentences containing one intonational phrase boundary giving a main effect of IPB (F(1,13) = 17.45, MSE = 0.385, p < 0.01). The different overall activation levels within the ROIs yielded a significant main effect of ROI (F(7,91) = 72.71, MSE = 1.435, p < 0.001). SPEECH TYPE and IPB yielded additive effects, as none of the interactions containing both factors reached significance. The type of speech materials (hummed or natural) influenced lateralization, yielding a significant SPEECH TYPE x HEMISPHERE interaction (F(1,13) = 30.59, MSE = 0.419, p < 0.001) and depended on the ROI giving a significant triple ROI x SPEECH TYPE x HEMISPHERE interaction (F(7,91) = 3.92, MSE = 0.103, p < 0.001). The lateralization, type of materials, and number of IPBs, influenced activation differently in different ROIs, giving the significant two-way interactions ROI x HEMISPHERE (F(7,91) = 19.83, MSE = 0.437, p < 0.001), ROI x SPEECH TYPE (F(7,91) = 6.40, MSE = 0.175, p < 0.001) and ROI x IPB (F(7,91) = 16.00, MSE = 0.018, p < 0.001). 16
The significant triple interactions between ROI x HEMISPHERE x IPB (F(7,91) = 6.14, MSE = 0.007, p < 0.001) and ROI x HEMISPHERE x SPEECH TYPE (F(7,91) = 3.92, MSE = 0.103, p < 0.001) indicate that SPEECH TYPE and IPB each interacted with hemisphere differently in different ROIs. The remaining interactions were not significant. To identify these differences between ROIs more closely we conducted additional ANOVAs separately for different ROIs. Significant main effects for IPB were observed in all ROIs except for the inferior frontal gyrus ROIs. The main effect SPEECH TYPE was significant in all ROIs but the anterior and posterior Rolandic operculum. Significant main effects for HEMISPHERE were observed in all ROIs with the exception of the anterior STG and the anterior Rolandic operculum. The HEMISPHERE x SPEECH TYPE interaction was significant in all but three ROIs, the anterior and posterior Rolandic operculum and the gyrus of Heschl. A HEMISPHERE x IPB interaction was observed in the posterior STG and the gyrus of Heschl. No ROI showed a significant SPEECH TYPE x IPB interaction. The triple interaction HEMISPHERE x SPEECH TYPE x IPB was significant only in the anterior Rolandic operculum. To investigate differences with regard to lateralization for the two types of material in every ROI more closely, Newman-Keuls post-hoc tests were calculated for the interaction HEMISPHERE x SPEECH TYPE. A stronger activation within the left hemisphere compared to the right was observed for natural speech in nearly all ROIs, except the anterior Rolandic operculum and the anterior part of the superior temporal gyrus. For hummed speech a lefthemispheric dominance was observed in only four ROIs, the gyrus of Heschl, the middle and posterior part of the superior temporal gyrus and the posterior Rolandic operculum. No ROI showed a significantly stronger activation of the right hemisphere for hummed sentences compared to natural speech. --- insert Figure 3 about here --Discussion 17
The aim of the present study was to identify the brain areas involved in the processing of sentence-level prosody by investigating the processing of intonational phrase boundaries (IPBs). We will first discuss our results with regard to differences in the processing of sentences with two IPBs as compared to sentences with one IPB, and later turn to the general differences between natural and hummed speech. Brain areas involved in IPB processing In the whole brain analysis stronger activation was observed for sentences with two IPBs than sentences with one IPB within the left superior temporal gyrus, for natural speech as well as for hummed sentences. An additional focus within the superior temporal gyrus on the right side was significant for natural speech but failed to reach significance for hummed speech. This indicates that processing an additional IPB activates the superior temporal gyrus similarly for natural speech and hummed sentences. In the ROI-analysis (pars opercularis and triangularis of the inferior frontal gyrus, anterior, middle and posterior part of the superior temporal gyrus, gyrus of Heschl, and anterior and posterior Rolandic operculum), more activation for two than for one IPB was observed only in the temporal ROIs, but not within the inferior frontal gyrus. This indicates that prosody processing mainly involves brain areas related to auditory processing. In addition, all but one ROIs, the posterior Rolandic operculum, showed a significant main effect or an interaction with SPEECH TYPE (natural speech or hummed sentences). This result could be taken to indicate that these regions are involved in the processing of the more complex segmental information contained in natural speech. The posterior Rolandic operculum (bilaterally), on the other hand, did not show stronger activation for natural speech compared to hummed sentences: there was no main effect or any interaction with SPEECH TYPE. This suggests that this region might play a less pronounced role in the processing of the specific spectral composition and the additional linguistic information (semantics, syntax) contained in natural speech. As this region, however, was more strongly activated by 18
materials containing two IPBs rather than one (main effect of IPB) it could be speculated that this region might be specifically involved in the processing of the prosodic information. Interestingly, there was no interaction of IPB and SPEECH TYPE in any of the ROIs investigated. This indicates that the activation elicited by the presence of an additional IPB in the ROIs investigated did not depend on the comprehensibility of the utterance. Although IPBs can aid the understanding of an utterance, we did not observe any evidence for an interaction in the respective ROIs, not even within the left inferior frontal gyrus, a region assumed to be involved in syntactic processing. This finding should not lead to the conclusion, however, that prosody and syntax do not interact. It is possible that syntax and prosody are processed independently from each other in different brain regions and that the interaction between both types of information occurs in higher associative areas within the brain. This is also suggested by a recent ERP study with patients suffering from lesions in the corpus callosum (Friederici et al., 2007). So far, only one neuroimaging study investigated the perception of IPBs (Strelnikov et al., 2006). In the condition with IPB, the IPB could be in one of two positions, changing the meaning of the sentence. 'To chop not # to saw' ('Rubit nelzya, pilit') means: 'not to chop but to saw', whereas 'To chop # not to saw' ('Rubit, nelzya pilit') means 'to chop and not to saw'. This is a possible construction in Russian because, different from Germanic languages such as German, Dutch or English, the negative word ‘not’ may have scope to its left or right depending on the position of an IPB. In this condition, participants were first presented with one of the two versions and had to select the appropriate alternative (one needs to: chop/saw). In the condition without IPB, participants were first presented with a simple statement ('Father bought him a coat') and then had to select the appropriate alternative ('His father bought him: a coat/a watch'). Strelnikov and colleagues reported stronger activation for sentences containing an IPB (e.g., 'To chop not # to saw', or 'To chop # not to saw') than for sentences without an IPB (e.g., 'Father bought him a coat') within the right posterior prefrontal cortex 19
and an area within the right cerebellum. Different from our results, Strelnikov et al. did not observe stronger activation within the superior temporal gyrus. However, as already outlined in the introduction, the task used was different for both sentences types. Although the task consisted in both cases of a visually presented question with two response alternatives ('Father bought him: a coat / a watch' and 'One needs to: saw / chop', for the two examples, respectively), the first type of materials (sentences without IPBs) required a semantic judgement based on the segmental content of the utterance while the second type of materials (sentences with an IPB) required a semantic judgement based on the segmental content as well as the prosodic information (IPB position) of the utterance. It is therefore possible that the differences observed by Strelnikov et al. (2006) might not only be due to the presence or absence of IPBs in the materials but also to task variables. To summarize our results so far, we observed a stronger activation within the superior temporal gyrus for sentences containing two IPBs compared to sentences containing one IPB. The activation extended to the gyrus of Heschl and the Rolandic operculum. To find out whether one of these areas might show a specialization with regard to prosody processing as compared to the processing of the segmental content of speech we also conducted an ROI analysis. In the ROI-analysis, the posterior part of the Rolandic operculum was the only brain region that showed a modulation of activation due to the presence of an additional IPB independent of the amount of segmental information contained in natural speech as compared to hummed sentences. The processing of hummed sentences compared to natural speech Hummed sentences also represent a human vocalization and a complex auditory signal. Comparing the processing of natural speech with hummed sentences can therefore yield information with regard to the brain areas processing the segmental content of speech as compared to brain areas processing the more basic aspects of speech. Comparing the processing of hummed sentences to baseline gives an initial overview of the brain areas 20
involved in the processing of hummed speech, as well as, potentially, in the processing the prosodic aspects of speech. The strongest activations were observed within the superior temporal gyrus, extending to the Rolandic operculum and the gyrus of Heschl, areas involved in auditory processing. Strong activations were also observed within the SMA, the right precentral gyrus and the cerebellum. These activations are most likely due to the manual response required by the task. On the basis of the logic of cognitive subtraction, subtracting hummed sentences from natural speech should yield brain areas that are involved in the processing of segmental information, lexico-semantics and syntax. Stronger activations for natural speech than for hummed sentences were observed in the middle temporal gyrus, bilaterally, and within the left opercular and triangular part of the inferior frontal gyrus, corresponding approximately to BA 44 and 45. These fronto-temporal activations are related to the processing of syntactic and semantic information contained in natural speech. Similar activations have been reported in other studies comparing natural speech to unintelligible speech (Spitsyna et al., 2006). Natural speech also activated the superior temporal gyrus more strongly than hummed speech, possibly reflecting the involvement of the auditory cortex in processing the more complex spectral components carrying the segmental information of natural speech. It should be noted that the probe monitoring task used here might have had some influence on the brain activation patterns observed. Our task induced an attentional focus on lexico-semantic processing rather than pure prosody processing that might also have influenced the activations we observed in temporal areas for hummed speech. In the following, we will further discuss our findings on the background of the organization of the auditory system with regard to its relation to speech and prosody processing. The brain areas involved in the processing of complex sounds and speech The auditory association cortex seems to represent mainly spectral and temporal properties of auditory stimuli rather than more abstract high-level properties such as auditory objects (see, for reviews, Griffiths et al., 2004; Griffiths & Giraud, 2004). The auditory 21
association areas are assumed to be processing more abstract properties than primary auditory cortex. Some of these properties are assumed to be processed in separate regions. Rauschecker and Tian (2000) proposed that anterior regions subserve auditory object identification while posterior regions process spatial information, similar to the ventral ('what') and dorsal ('where') pathways of the visual system (Ungerleider & Mishkin, 1982). Although such an anterior-posterior distinction is supported in part by single cell recordings in monkeys (Recanzone, 2000; Tian et al., 2001) as well as by lesion (Adriani et al., 2003; Clarke et al., 2000) and brain imaging studies (Maeder et al., 2001), the distinction is not clear cut. The functional nature of this anterior-posterior division is therefore still under debate (Zatorre et al., 2002). With regard to the processing of speech, the dual pathway model (Rauschecker & Tian, 2000) suggests that speech should activate regions anterior to the primary auditory cortex, as it is a familiar and highly complex auditory signal with no relation to spatial information. When intelligible speech is compared to non-speech, stronger left-lateralized activations obtain in anterior and middle parts of the superior temporal sulcus, for single words as well as for sentences (Binder et al., 2000; Narain et al., 2003; Scott al., 2000; Meyer et al., 2004). However, often also activation within the posterior temporal and inferior parietal regions is observed in imaging studies (Meyer et al., 2002; Spitsyna et al., 2006). Although not directly derivable from the dual pathway model, this finding is not altogether surprising. It has long been known from lesion studies that posterior temporal and inferior parietal regions are critical for supramodal language comprehension (Geschwind, 1965). The stronger activations for natural speech than for hummed sentences observed in the present study within the superior temporal gyrus are compatible with previous results. We observed stronger activation for natural than for hummed speech in middle and posterior parts of the superior temporal gyrus and the posterior part of the Rolandic operculum. With regard to the lack of activation in anterior parts of the superior temporal lobe, it could be speculated 22
that hummed speech is recognized as a familiar human vocalization. This familiarity might explain differences to results from studies using speech materials that are rendered unintelligible artificially and sound more unfamiliar (Scott et al., 2000; Meyer et al., 2004). It should be noted, however, that the activations observed within the temporal cortex for speech may not be specific for speech. Similar activations have been observed for other complex auditory stimuli such as musical sounds and melodies (Griffiths et al., 1998; Patterson et al., 2001; see, for a review, Price et al., 2005). It is possible that differences between speech and other complex auditory stimuli are subtle and go beyond simple localization. A recent study by Tervaniemi et al. (2006), for example, showed that the superior temporal region reacts differently to changes in pitch or duration for speech syllables than for musical sounds. The brain areas involved in the perception of prosody Activations within the superior temporal gyri have been regularly observed in imaging studies when speech with or without prosodic modulation is presented and compared to baseline (e.g. Wildgruber et al., 2004; Hesling et al., 2005; Gandour et al., 2003, 2004). Regions outside the temporal lobe have also been observed to show a modulation of activation dependent on prosodic properties but appear to vary considerably across different studies. Prosody-related activations were observed to depend on the specifics of the task (Tong et al., 2005; Plante et al., 2002), the type of prosody involved (e.g., affective vs. linguistic, Wildgruber et al., 2004) and the degree of propositional information contained in the stimulus materials (Hesling et al., 2005; Tong et al., 2005; Gandour et al., 2003, 2004). Although the superior temporal region seems to be the main candidate for the perception of prosodic modulation in speech, other areas outside the temporal lobe are likely to be involved and the auditory association areas are likely to be only part of a more extended processing network for prosodic aspects of speech. Imaging studies investigating the processing of sentence-level prosody often compared natural speech to speech stimuli either with no segmental information or with no or little 23
prosodic information. One possibility is to remove the segmental information by filtering, preserving the intonational contour of the sentence (Meyer et al., 2002, 2004). The results from these studies point towards a fronto-temporal network with a right hemispheric dominance. The Rolandic operculum, in particular in the right hemisphere, has been identified as part of this network. Another approach is to remove the intonational contour of sentences, for example, by high-pass filtering (Meyer et al., 2004; Hermann et al., 2003), or to reduce prosody information by speaking with little prosodic expressiveness (Hesling et al., 2005). The rationale for this approach is to identify prosody-related activations by subtracting lowprosody speech from normal speech, thus ideally subtracting away activations related to the processing of the segmental content of speech while leaving prosody-related activations. The results from these studies, however, are mixed. Meyer et al. (2004) did not observe stronger activations for normal speech than for low-prosody speech. It is possible, that this is finding is due to the task that required a prosody comparison between two successive sentences stimuli in an experimental setting in which fattened speech [-prosodic information] and degraded speech [-segmental information were presented in a pseudo-randomized order. Hesling et al. (2005) observed different results for intelligible and nonintelligible (i.e., low-pass filtered) speech. In the case of high-prosody intelligible speech, the right superior temporal gyrus was more strongly activated than in the case of low-prosody intelligible speech. High-prosody nonintelligible speech did not activate any brain regions more strongly than low-prosody nonintelligible speech. These results indicate that the additional prosodic information contained in high-prosodic speech activates the superior temporal gyrus, although not very strongly. Finally, a study by Doherty and colleagues (2004) compared sentences with a rising pitch (question) to sentences with a falling pitch (statement). Although their results might not exclusively reflect prosody processing, they observed, among others, a stronger activation within the superior temporal gyrus, bilaterally, for sentences with a rising pitch. These findings also indicate that the superior temporal gyrus might play a dominant role in the first 24
stages of prosody processing. The results of the present study, namely, that areas within and around the superior temporal gyrus are involved in IPB processing therefore agree well with previous results. The lateralization of prosody processing Although a right hemispheric dominance in prosody processing was initially suggested based on evidence from patients, the results with regard to the perception of sentence-level nonaffective prosody show a mixed picture. Some studies show greater impairment in RHD patients compared to LHD and controls (Bryan, 1989), whereas others find greater impairment for LHD (Pell & Baum, 1997; Perkins et al., 1996), or a more complicated pattern of preserved function alongside impairments for both patient groups compared to controls (Baum & Dwivedi, 2003; Baum et al., 1997; Walker et al., 2001; Imaizumi et al., 1998). A possible reason for these differences might be that the patient groups comprise patients with non-overlapping patterns of brain damage, often in fronto-parietal areas. Cognitive functioning in these patients might therefore be compromised in a number of domains, such as, for example, working memory and drawing inferences, faculties required in complex tasks such as prosodic disambiguation of sentences. Imaging studies investigating the perception of sentence-level prosody with healthy adults circumvent this problem and allow a separate assessment of lateralization for different brain regions. A number of imaging studies have investigated the lateralization of prosodic processing (Hesling et al., 2005; Meyer et al., 2002, 2004; Kotz et al., 2003). Meyer et al. (2002, 2004) observed stronger right-hemispheric activations within fronto-temporal areas for low-pass filtered speech than for natural speech whereas Kotz et al. (2003) using natural speech stimuli observed predominantly bilateral activations. Hesling et al. (2005) found activation within the right superior temporal gyrus when high-prosody intelligible speech was compared to low-prosody intelligible speech. Another way to remove the propositional content of speech while preserving its prosodic contour is to present foreign language 25
materials to listeners with no knowledge of this language. In an experiment by Tong et al. (2005), bilateral activation was observed for English speakers listening to Chinese language materials for eight out of nine ROIs investigated. A stronger right-hemispheric activation was observed for only one ROI within the middle frontal gyrus. Tong et al. also found mostly bilateral activation (six out of nine ROIs) for Chinese participants listening to Chinese, and a dependence of lateralization on the type of task. This evidence indicates that prosody when presented together with segmental information is subserved by a bilaterally distributed network of fronto-temporo-parietal brain areas. In the present study, hummed sentence materials were presented as well as natural speech. Under the hypothesis that the right hemisphere is specialized in the processing of sentence-level prosody it was hypothesized that the processing of hummed speech compared to baseline shows right-hemispheric lateralization. In the whole brain analysis, bilateral activation was observed in the superior temporal gyrus and the precentral gyrus, and stronger activation of the right hemisphere was observed within the middle frontal gyrus for hummed sentences. In the ROI-analysis, six out of eight ROIs showed stronger activation within the left hemisphere for natural speech, whereas only four out of eight ROIs show a lefthemispheric dominance for hummed sentences. This suggests that prosodic processing is organized more bilaterally than the processing of other properties of speech, such as syntax or lexico-semantics. Further insights about the lateralization with regard to the processing of specific linguistic aspects of prosody can be derived from the lateralization of IPB processing. An interaction between the number of IPBs and lateralization was observed only in two ROIs (posterior STG, gyrus of Heschl) with both ROIs showing a stronger modulation of activation due to the number of IPBs in the left hemisphere, for natural speech as well as hummed sentences. This could be taken to indicate that these two temporal areas show a lefthemispheric dominance for prosody processing.
26
The left-hemispheric dominance in the temporal regions observed in the present study might be due to some degree to the task employed. While other studies required participants to attend directly to the prosodic information of the speech stimuli (Tong et al. 2005; Meyer et al., 2002, 2004), a probe detection task was used here that required participants, even in the hummed speech condition, to attend to and memorize a potentially appearing naturally spoken word carrying segmental information. The results of the present study are therefore compatible with the functional or task-dependent class of hypotheses. According to this class of hypotheses, the lateralization of prosody processing could shift from the right to the left hemisphere when the task or the stimulus materials promote attention to syntactic-semantic rather than prosodic properties of the materials.
Conclusion This study aimed at identifying the brain areas involved in the processing of sentence-level prosody, and in particular, the processing of intonational phrase boundaries (IPBs). Sentences with two IPBs activated the superior temporal gyrus, bilaterally, more strongly than sentences with one intonational phrase boundary. This pattern of activation was very similar for natural speech and hummed sentences. The results from the ROI analysis suggest that the posterior Rolandic operculum might play a specific role in the processing of prosodic information, because it was the only ROI not the showing an influence of the type of speech materials (hummed sentences or natural speech). When comparing natural speech and hummed sentence materials, we found natural speech to activate a number of areas in the left hemisphere more strongly than hummed sentences. The left-hemispheric dominance of temporal activations observed for hummed sentences, however, might be due to the attentional focus on segmental information required by the task employed in the present study.
27
References Adriani, M., Maeder, P., Meuli, R., Bellmann, A., Frischknecht, R., Villemure, J.-G., Mayer, J., Annoni, J.-M., Bogousslavsky, J., Fornari, E., Thiran, J.-P., Clarke, S. 2003. Sound recognition and localization in man: specialized cortical networks and effects of acute circumscribed lesions. Exp Brain Res 153:591-604. Austin J. 1975. How to do things with words. (William James Lectures). 2nd edition. M. Sbisa, J. Urmsson (Eds). Cambridge, Mass: Harvard University Press. Baum, S. R., Dwivedi, V. D. 2003. Sensitivity to prosodic structure in left-and righthemisphere-damaged individuals. Brain Lang 87:278-289. Baum, S., Pell, M., Leonard, C., Gordon, J. 1997. The ability of right- and left-hemispheredamaged individuals to produce and interpret prosodic cues marking phrasal boundaries. Lang Speech 40:313–330. Binder, J. R., Frost, J. A., Hammeke, T. A., Bellgowan, P. S. F., Springer, J. A., Kaufman, J. N., Possing, E. T. 2000. Human temporal lobe activation by speech and nonspeech sounds. Cereb Cortex 10:512–528. Brett, M., Christoff, K., Cusack, R., Lancaster, J. 2001. Using the Talairach atlas with the MNI template. Neuroimage 13:S85. Bryan, K. 1989. Language prosody and the right hemisphere. Aphasiology. 3:285–299. Clarke, S., Bellman, A., Meuli, R., Assal, G., Steck, A. J. 2000. Auditory agnosia and auditory spatial deficits following left hemispheric lesions: evidence for distinct processing pathways. Neuropsychologia 38:797-807. Cooper, WE., Paccia-Cooper, J. 1981. Syntax and Speech. Harvard University Press. Doherty, C. P., West, W. C., Dilley, L. C., Shattuck-Hufnagel, S., Caplan, D. 2004. Question/statement judgments: An fMRI study of intonation processing. Human Brain Mapp 23:85–98.
28
Frazier, L., Carlson, K., Clifton, C. 2006. Prosodic phrasing is central to language comprehension. Trends Cogn Sci 6:244-249. Friederici, A. D., Alter, K. 2004. Lateralization of auditory language functions: A dynamic dual pathway model. Brain Lang 89:267-276. Friederici, A. D., von Cramon, D.Y., Kotz, S. A. 2007. Role of the corpus callosum in speech comprehension: Interfacing syntax and prosdody. Neuron 53:135-145. Gandour, J., Dzemidzic, M., Wong, D., Lowe, M., Tong, Y., Hsieh, L., Satthamnuwong, N., Lurito, J. 2003. Temporal integration of speech prosody is shaped by language experience: An fMRI study. Brain Lang 84:318-336. Gandour, J., Wong, D., Lowe, M., Dzemidzic, M., Satthamnuwong, N., Tong, Y., Li, X. 2002. A cross-linguistic fMRI study of spectral and temporal cues underlying phonological processing. J Cogn Neurosci 14:1076–1087. Gandour, J., Tong, Y., Wong, D., Talavage, T., Dzemidzic, M., Xu., Y., Li., X., Lowe, M. 2004. Hemispheric roles in the perception of speech prosody. Neuroimage, 23:344357. Geschwind, N. 1965. Disconnexion syndromes in animals and man: Part I. Brain 88:237–294. Grabe, E., Warren, P., Nolan, F. 1994. Resolving category ambiguities - evidence from stress shift. Speech Communication 15:101-114. Griffiths, T. D., Giraud, A. L. 2004. Auditory Function. In: Human Brian Function. 2nd ed. R. S. J. Frackowiak, K. J. Friston, C. D. Frith, R. J. Dolan, C. J. Price, S. Zeki, J. Ashburner, W. Penny (Eds). Amsterdam, NL: Elsevier. pp. 61-75. Griffiths, T. D., Warren, J. D., Scott, S., K., Nelken, I., King, A. J. 2004. Cortical processing of complex sound: a way forward? Trends Neurosci. 27:181-185. Griffiths, T.D., Büchel, C., Frackowiak, R. S. J., Patterson, R. D. 1998. Analysis of temporal structure in sound by the human brain. Nat Neurosci. 1:422–427.
29
Grosjean, F., Hirt, C. 1996. Using prosody to predict the end of sentences in English and French: Normal and brain-damaged subjects. Lang Cogn Proc. 11:107-134. Herrmann, C. S., Friederici, A.D., Oertel, U., Maess, B., Hahne, A., Alter, K. 2003. The brain generates its own sentence melody: A Gestalt phenomenon in speech perception. Brain Lang, 85:396-401. Hesling, I., Clement, S., Bordessoules, M., Allard, M. 2005. Cerebral mechanisms of prosodic integration: evidence from connected speech. Neuroimage. 24:937-947. Imaizumi, S., Mori, K., Kiritani, S., Hiroshi, H., Tonoike, M. 1998. Task-dependent laterality for cue decoding during spoken language processing. Neuroreport. 9:899–903. Isel, F., Alter, K., Friederici, A. D. 2005. Influence of prosodic information on the processing of split particles: ERP evidence from spoken German. J Cogn Neurosci. 17:154-167. Kotz, S. A., Meyer, M., Alter, K., Besson, M., von Cramon, D. Y., Friederici, A. D. 2003. On the lateralization of emotional prosody: an event-related functional MR investigation. Brain Lang. 86:366– 376. Kreiman, J. 1982. Perception of sentence and paragraph boundaries in natural conversation. Journal of Phonetics, 10:163-175. Larkey, L. S. 1983. Reiterant speech: an acoustic and perceptual validation. J Acoust Soc Am. 73:1337-1345. Liu, T. T., Frank, L. R., Wong, E. C., Buxton, R. B. 2001. Detection power, estimation efficiency and predictability in eventrelated fMRI. Neuroimage. 13: 759–773. Maeder, P. P., Meuli, R. A., Adriani, M., Bellmann, A., Fornari, E.Thiran, J.-P., Pittet, A., Clarke, S. 2001. Distinct Pathways Involved in Sound Recognition and Localization: A Human fMRI Study. Neuroimage. 14:802-816. Meyer, M., Alter, K., Friederici, A. D., Lohmann, G., & von Cramon, D. Y. 2002. FMRI reveals brain regions mediating slow prosodic modulations in spoken sentences. Hum Brain Mapp. 17:73–88. 30
Meyer, M., Steinhauer, K., Alter, K., Friederici, A. D., von Cramon, D. Y. 2004. Brain activity varies with modulation of dynamic pitch variance in sentence melody. Brain Lang. 89:277-289. Narain, C., Scott, S. K., Wise, R. J. S., Rosen, S., Leff, A., Iversen, S. D., Matthews, P. M. 2003. Defining a left-lateralized response specific to intelligible speech using fMRI. Cereb. Cortex. 13:1362–1368. Nespor, M., Vogel I. 1983. Prosodic structure above the word. In: Prosody: Models and Measurements. A. Cutler, D. R. Ladd (Eds.) Berlin: Springer Verlag. pp. 123-140. Pannekamp, A., Toepel, U., Alter, K., Hahne, A., Friederici, A.D. 2005. Prosody-driven Sentence Processing: An Event-related Brain Potential Study. J Cogn Neurosci 17:407-421. Patterson, R. D., Uppenkamp, S., Johnsrude, I., Griffiths, T. D. 2002. The Processing of Temporal Pitch and Melody Information in Auditory Cortex. Neuron 36:767–776. Pell, M., Baum, S. 1997. The ability to perceive and comprehend intonation in linguistic and affective contexts by brain-damaged adults. Brain Lang 57:80–99. Perkins, J. M., Baran, J. A., Gandour, J. 1996. Hemispheric specialization in processing intonation contours. Aphasiology 10:343–362. Plante, E., Creusere, M., Sabin, C. 2002. Dissociating sentential prosody from sentence processing: Activation interacts with task demands. Neuroimage. 17:401–410. Poeppel, D. 2003. The analysis of speech in different temporal integration windows: Cerebral lateralization as asymmetric sampling in time. Speech Comm. 41:245–255. Price, C., Thierry, G., Griffiths, T. 2005. Speech-specific auditory processing: where is it? Trends Cogn Sci. 9:271-276. Rauschecker, J. P., Tian, B. 2000. Mechanisms and streams for processing of ‘what’ and ‘where’ in auditory cortex. Proc Natl Acad Sci 97:11800–11806.
31
Recanzone, G. H. 2000. Spatial processing in the auditory cortex of the macaque monkey. Proc Natl Acad Sci. 97:11829-11835. Rilliard, A., Aubergé, V. 1998. Reiterant Speech for the Evaluation of Natural vs. Synthetic Prosody. In: Third ESCA/COCOSDA Workshop on Speech Synthesis. Australia. 1998. 87-92. Rooij, J. J. de, 1975. Prosody and the perception of syntactic boundaries. IPO Annual Progress Report 10:36-39. Rooij, J. J. de, 1976. Perception of prosodic boundaries. IPO Annual Progress Report 11:2024. Schafer, A. J., Speer, S. R., Warren, P., White, S. D. 2000. Intonational disambiguation in sentence production and comprehension. J Psycholing Res 29:169-182. Schmahmann, J. D., Doyon, J., McDonald, D., Holmes, C., Lavoie, K., Hurwitza, A. S., Kabanic, N., Toga, A., Evans, A., Petrides, M. 1999. Three-Dimensional MRI Atlas of the Human Cerebellum in Proportional Stereotaxic Space. Neuroimage 10:233-260. Scott, S.K., Blank, C. C., Rosen, S., Wise, R. J. S. 2000. Identification of a pathway for intelligible speech in the left temporal lobe. Brain. 123:2400–2406. Searle J. 1969. An essay in the philosophy of language. Cambridge, MA: Cambridge University Press. Selkirk, E. 1995. Sentence prosody: intonation, stress, and phrasing. In The Handbook of Phonological Theory, J. A. Goldsmith (Ed.). Cambridge, MA: Blackwell. pp. 550-569. Selkirk, E. 2000. The interaction of constraints on prosodic phrasing. In Prosody: Theory and Experiment M. Horne (Ed.), Dordrecht: Kluwer Academic Publishing, pp. 231-262. Spitsyna, G., Warren, J. E., Scott, S. K., Turkheimer, F. E., Wise, R. J. S. 2006. Converging Language Streams in the Human Temporal Lobe. J Neurosci 26:7328-7336. Stark, C. E. L., Squire, L. R. 2001. When zero is not zero: The problem of ambiguous baseline conditions in fMRI. Proc Natl Acad Sci 98:12760–12766. 32
Steinhauer, K. 2003. Electrophysiological correlates of prosody and punctuation. Brain Lang 86:142-164. Steinhauer, K., Alter, K., Friederici, A. D. 1999. Brain potentials indicate immediate use of prosodic cues in natural speech processing. Nat Neurosci 2:191–196. Steinhauer, K., Friederici, A. D. 2001. Prosodic boundaries, comma rules, and brain responses: The closure positive shift in ERPs as a universal marker for prosodic phrasing in listeners and readers. J Psycholing Res 30:267–295. Strelnikov, K. N., Vorobyev, V. A., Chernigovskaya, T. V., Medvedeva, S. V. 2006. Prosodic clues to syntactic processing - a PET and ERP study. Neuroimage 29:1127-1134. Talairach, J., Tournoux, P. 1988. Co-planar stereotaxic atlas of the human brain. New York: Thieme. Tervaniemi, M., Szameitat, A., Kruck, S., Schröger, E., Alter, K., De Baene, W. Friederici, A. D. 2006. From air oscillations to music and speech: functional magnetic resonance imaging evidence for fine-tuned neural networks in audition. J Neurosci 23:8647– 8652. t'Hart, R. C. & Cohen, A. 1990. A perceptual study of intonation: an experimental-phonetic approach to speech melody. Cambridge, England: Cambridge Univ. Press Tian, B., Reser, D., Durham, A., Kustov, A., Rauschecker, J. 2001. Functional specialization in rhesus monkey auditory cortex. Science 292:290–293. Tomasi, D., Ernst, T., Caparelli, E. C., Chang, L. 2006. Common deactivation patterns during working memory and visual attention tasks: an intra-subject fMRI study at 4 Tesla. Human Brain Mapp 27:694-705. Tong, Y., Gandour, J., Talavage, T., Wong, D., Dzemidzic, M., Xu, Y., Li, X., Lowe, M. 2005. Neural circuitry underlying sentence-level linguistic prosody. Neuroimage 28:417-428.
33
Truckenbrodt, H. 2005. A short report on intonation phrase boundaries in German. Linguistische Berichte 203:273-296. Tzourio-Mazoyer, N., Landeau, B., Papathanassiou, D., Crivello, F., Etard, O., Delcroix, N., Mazoyer, B., Joliot, M. 2002. Automated anatomical labelling of activations in spm using a macroscopic anatomical parcellation of the MNI MRI single subject brain. Neuroimage 15:273-289. Ungerleider, L., Mishkin, M. 1982. Two cortical visual systems. In: Analysis of visual behavior. D. J. Ingle, M. A. Goodale, R. J. W. (Eds) Cambridge, MA: MIT Press. pp 549-586. Van Lancker, D. 1980. Cerebral lateralization of pitch cues in the linguistic signal. Papers in Linguistic: Int. J. Human Communication 13:200–277. Walker, J. P., Fongemie, K., Daigle, T. 2001. Prosodic facilitation in the resolution of syntactic ambiguities in subjects with left and right hemisphere damage. Brain Lang 78:169-196. Wildgruber, D., Hertrich, I., Riecker, A., Erb, M., Anders, S., Grodd, W., Ackermann, H., 2004. Distinct frontal regions subserve evaluation of linguistic and emotional aspects of speech intonation. Cereb Cortex 14:1384–1389. Wong, P. C. M. 2002. Hemispheric specialization of linguistic pitch patterns. Brain Res Bull 59:83–95. Zatorre, R. J., Belin, P. 2001. Spectral and temporal processing in human auditory cortex. Cereb Cortex. 11:946–953. Zatorre, R. J., Bouffard, M., Ahad, P., Belin, P. 2002. Where is ‘where’ in the human auditory cortex? Nat Neurosci 5:905–909.
34
Acknowledgement.
This work was supported by a grant of the Human Frontier Science
Program (HFSP RGP5300/2002-C102) awarded to Kai Alter.
35
Table 1. Example of the stimulus materials and mean length of the sentences in secs.
Sentences (Naturally spoken and hummed)
Natural Hummed
Type A 1 IPB
Peter verspricht Anna zu arbeiten # und das Büro zu putzen Peter promises Anna to work and to clean the office
2 IPBs
4.42
4.64
4.74
5.06
5.73
5.76
6.12
5.85
Peter verspricht # Anna zu entlasten # und das Büro zu putzen Peter promises to support Anna and to clean the office
Type B 1 IPB
Otto bringt Fleisch, # Ute und Georg kaufen Salat und Säfte für das Grillfest. Otto contributes meat, Ute and Georg buy salad and soft drinks to the barbecue.
2 IBPs
Otto bringt Fleisch, # Ute kauft Salat # und Georg kauft Säfte für das Grillfest. Otto contributes meat, Ute buys salad and Georg buys soft drinks to the barbecue.
36
Table 2. Contrasts are thresholded at p < 0.001 uncorrected, p < 0.05 corrected on cluster level. Coordinates are reported as given by SPM2 (MNI space), corresponding only approximately to Talairach Turnoux space (Talairach and Tournoux, 1988; Brett et al., 2001). Anatomical labels are given on the basis of the classification of the AAL (automated anatomical labeling) atlas (Tzourio-Mazoyer et al., 2002). Cerebellar labels within the AALatlas are based on Schmahmann et al. (1999). The first label denotes the location of the maximum, the following labels denote further areas containing a majority of voxels of the activated cluster. Abbreviations: IFG_oper = inferior frontal gyrus, pars opercularis, IFG_tri = inferior frontal gyrus, pars triangularis, MFG = middle frontal gyrus, SFG = superior frontal gyrus, SMA = supplemental motor area, TP = temporal pole, MTG = middle temporal gyrus, STG = superior temporal gyrus, AG = angular gyrus, ROp = Rolandic operculum, HeschlG = gyrus of Heschl, SPL = superior parietal lobule, IPL = inferior parietal lobule, PrecG = precentral gyrus, PostG = postcentral gyrus, supramG = surpamarginal gyrus, k = cluster size. *Activation is part of a bigger cluster. 1 masked with natural speech > baseline (hummed sentences > baseline) at p < 0.05, inclusive. Side
x Natural Speech > Baseline
y
Frontal Left Right SMA 3 15 Right Insula* 36 24 Right PrecG* 30 -6 Right PrecG* 54 3 Temporal Left STG -60 -24 Right STG* 66 -18 Parietal Left SPL -30 -57 Cerebellum Left Right Right Crus1 42 -57 Natural > Hummed Sentences1 Frontal Left IFG_tri , IFG_oper -39 30 Right SMA 6 15 Right IFG_tri 45 27 Right MFG 33 3 Temporal Left TP, MTG, STG -57 9 Right MTG, STG 63 3 Parietal Left AG, SPL -27 -63 Basal Ganglia Left Thalamus, Caudate -15 -12 Natural Speech: 2 pauses > 1 pause Temporo-parietal Left HeschlG, STG, ROp -45 -18 Right STG, ROp, HeschlG 42 3 Natural Speech: 1 pauses > 2 pause ns
z
k
Z
x Hummed Sentences > Baseline
y
Z
k
Z
54 6 30 48
7897 7897 7897 7897
5.36 4.77 4.63 4.09
PrecG*
-36
-12
66
4928
4.78
SMA* MFG* PrecG
3 27 51
-3 36 -3
66 30 0
4928 4928 4928
5.81 4.42 6.40
6 6
7897 7897
6.25 6.03
STG* STG
-54 51
-6 3
-6 54
4928 47
5.90 4.96
48
7897
4.68
IPL
27
-39
30
4928
3.63
-54 -36 -57
-36 -36 -36
92 93 87
4.00 4.04 3.82
24
105
3.61
-36
7897
4.87
0 54 18 54
600 209 50 65
5.13 5.42 4.13 4.30
-24 -18
583 150
5.01 4.43
42
83
4.00
12
347
4.13
Crus1 -42 VI 33 Crus1 45 Hummed > Natural Speech1 ns
Hummed sentences: 2 pauses > 1 pause 12 -12
287 242
4.51 4.64
SupramG, STG, HeschlG
-63
-39
Hummed sentences: 1 pauses > 2 pause ns
37
Figure Captions
Figure 1. Spectrograms (0 - 5000 Hz) and fundamental frequency contours (pitch: 0 - 500 Hz) for sample stimuli of the sentence materials used for all conditions of the experiment. Natural speech is presented on the left, the respective hummed version of the sentence on the right. Spectograms and pitch contours are given for each pair (1 and 2 IPBs) of sentence materials, for type A sentences in the top half, for type B sentences in the bottom half. Figure 2. Trial timing scheme. Figure 3. A visualization of the six temporal ROIs as an ascending series of axial slices (shown only for the left hemishpere). ROIs: anterior (y > -15, light blue), middle (-15 > y > 35, yellow) and posterior (y < -35, brown) part of the superior temporal gyrus, gyrus of Heschl (dark blue), anterior (y > -35, medium blue) and posterior (y < -35, orange) part of the Rolandic operculum. Figure 4. a) Comparison of sentences with 2 IPBs to sentences with 1 IPB: 2 IPBs > 1 IPB. Natural speech (left) and hummed sentences (right). b) Comparison of speech types: Natural speech > hummed sentences. Threshold: p < 0.001, uncorrected, showing only clusters with more than 40 voxels, corresponding to a p < 0.05 corrected on cluster-level. Left is left in the image. Figure 5. Results of the ROI-analysis (bars with black stripes = left hemisphere, white bars = right hemisphere). Effect sizes were averaged over all voxels of each ROI. Error bars represent the standard error of the mean. Abbreviations: n1 = natural speech with 2 IPBs, n2 = natural speech with 2 IPBs, h1 = hummed sentences with 1 IPB, h2 = hummed sentences with 2 IPBs, IFG oper = inferior frontal gyrus, pars opercularis, IFG tri = inferior frontal gyrus, pars triangularis, ant/post RolOp = anterior/posterior part of the Rolandic operculum, ant/mid/post STG = anterior/middle/posterior part of the superior temporal gyrus, Heschl = gyrus of Heschl. 38
39
Footnotes 1. Although the term 'activation' suggest an absolute value, it is, in the case of fMRI data, always relative. This is due to two reasons. First, the statistical analysis only evaluates differences between conditions. Second, it is generally difficult to define an absolute baseline in brain activation for physiological reasons (Stark & Squire, 2001; Tomasi et al., 2006). In the present article, the term 'activation' is used only, when an experimental condition is compared to a low-level baseline. For comparisons between conditions the term 'activation difference' is used, or the term 'activation' with an adjective indicating the direction of the difference (e.g., 'stronger').
40
