The role of emotionality and typicality in ... - Wiley Online Library

Scandinavian Journal of Psychology, 1995, 36, 189-200

The role of emotionality and typicality in speechreading KARMA JOHANSSON and JERKER RdNNBERG Department of Education and Psychology, Linkdping University, Link6ping, Sweden

Johansson, K. & Rhnberg, J. (1995) The role of emotionality and typicality in speechreading. Scandinavian Journal of Psychology, 36, 189-200. Two experiments were carried out to investigate how emotionality and typicality could influence speechreading performance. Typical and atypical sentences for a certain script, and emotional sentences (happy and sad content, presented with happy, sad or neutral facial expression) were shown without sound on a TV-screen. Two different scripts (restaurant and doctor) were used. In Experiment 1, hearing-impaired subjects participated and in Experiment 2, normal-hearing subjects participated. Experiment 2 also evaluated the effects of tactile information. The results from both experiments showed that typical restaurant Sentences were the easiest to speechread of all sentence-types, in line with script-theory (Abelson, 1981; Anderson, 1983; Bell& Bower, 1981; Nottenburg & Shoben, 1980; Yekovich & Walker, 1986). For the doctor script, sad sentences were better speechread than happy sentences, also according to script-theory. In addition, perception of emotional content was enhanced by tactile information. Generally. both the cognitive and emotional effects are scriptdependent, suggesting important social constraints on speechreading accuracy. Key words: Emotionality, typicality, facial expression, speechreading. Karina J o h n s o n , Department of Education and Psychology, Link6ping University, S-581 83 Linkdping, Sweden

Human social processes involve emotional communication to a large extent (Buck, 1984). Amongst different situations of emotional communication the most prominent one is face-to-face interaction. Emotional expressions are important since they convey, in most cases, important aspects of the speaker’s message. When having to rely on speechreading, facial expressions might be even more important as they may compensate for the lack of auditory cues in speechreading. A number of fundamental or biologically basic emotions such as anger, fear, sadness, disgust, and happiness, are often distinctly manifested in different facial expressions (Ekman, 1973; 1992; bard, 1977). There is also high agreement among judges about emotions shown in the human face (Ekman et al., 1982). Emotions shown in the speaker’s facial expression are supposedly important to gain a better understanding of the spoken message. The general prediction, then, is that emotional sentences, with a distinct facial expression ought to be easier to speechread than emotional sentences accompanied by a more neutral facial expression. A further important source of information for speechreaders is their schematic knowledge about a given communicative situation. A simple form of cognitive schema is the script, embodying knowledge of stereotyped event sequences (Abelson, 1981). A script involves common information about a specific situation or a set of actions within a special framework. These actions are so well practised in everyday life that their performance is stereotyped (Schank & Abelson, 1977), and the script functions as guidance to one’s actions and expectations during the sequence (Nottenburg & Shoben, 1980). This guidance can, to a substantial degree, facilitate speechreading and understanding of the message (Samuelsson & RBnnberg, 1993). Consequently, as sentences typical of a specific script should be expected more frequenctly than atypical sentences, typical sentences should be better speechread than atypical sentences for the same script.

0 1995 Scandinavian

University Press. ISSN 0036-5564

190

K. Johansson and J. Ronnberg

Scan J Psycho1 36 (1995)

In order to assess the separate contributions of facial expression and schematic knowledge, two different scripts were chosen, a visit to a doctor and a visit to a restaurant. When given information of a script certain expectations of the script is brought into mind, activating possible messages and actions of the script. The doctor script is assumed to be an emotionally negatively charged script, since visiting a doctor is hardly a pleasant experience. The script is not a stereotyped one as the events, and thus the sentences spoken by a doctor, are hard to predict depending on the outcome of the visit. Therefore, emotional sentences (is. sad sentences) and facial expression were thought to be delimiting factors and would thus facilitate interpretation of this script. The restaurant script, on the other hand, is a cognitive script involving stereotyped distinct events and highly predictable sentences, and therefore, sentences involving schematic knowledge (i.e., typical sentences) of the script were thought to be a facilitating factor. To test the predictions and to compare the effects of emotionality (i.e., happy vs. sad) and trpicality (i.e., typical vs. atypical sentences for a script)’ factors on speechreading accuracy, Experiment 1 was conducted. As hearingimparied persons have to rely on speechreading in their everyday communication they were considered a suitable group for Experiment 1. EXPERIMENT 1 Method Overview of the experiment. Two well-known simple scripts, were used (a visit to a restaurant and a visit to a doctor), and three different facial expressions: sad. happy and neutral. The sentences for each

script were of two types: emotional sentences (sad and happy content, presented with a sad, happy or neutral face) and cognitive sentences (typical and atypical, with a neutral face). The sentences wen recorded, with sound on a videotape, but presented without sound. To create expectations of the sentence-tcFbe-specchd a sign was also shown to the subjects, stating that the sentence would be sad or happy, typical or atypical. In half of the cases the signs were correct and in half of the cases false with respect to the content of the sentence. (The emotions shown on the sign and the facial expression werc always matched.) When a correct sign was shown the spccchrrading performance was hypothesized to be higher compared to when a falsc sign was shown. Half of the emotional sentences were accompanied by a facial expression. The remaining half of the emotional sentences, as well as the cognitive sentenccs, were accompanied by a neutral face. For an overview of the design. see Fig. 1. Subjects. Twenty-one (fourteen men) moderately hearing-impaid subjects participated in the experiment. Their mean age was 54.05 years (SD= 9.08 yean; range = 36-68 years). Their average hearing loss calculated for the “best ear” over the frequencies 500, 1000 and 2000 Hz,was 49.43dB (SD= 13.30; range = 30-75). MareriaLs. Two scripts were used in the experiment. a visit to a doctor and a visit to a restaurant. For each mipt, 60 simple sentences were constructed. T h s sentences were judged by i8 subjects on a 7-point scale typidity-diicnsion. from atypical (I) to typical (9, as well as for the emotion-dimension, from sad (1) to happy (7). For each script, the four sentences rated as the most typical, and also judged as emotionally neutral. and, the four sentences rated as the kast typical, and also judged as emotionally neutral, were chosen as the two levels of the typicality sentence variable. Two of the typical and two of the atypical sentcnca were randomly assigned to a false/comct sign condition. For each script, the eight sentences judged as the most happy and also judged as cognitively neutral, and the eight sentences judged as the most sad and also judged as cognitivdy neutral, wen chosen as the two levels of the emotional sentence variable. Of the 16 “emotional” sentences, half of the happy and half of the sad sentences were randomly assigned to the conditions with happy or sad facial expression or to the neutral face. Within these two sets of sentences, four sentences wcrc randomly assigned to correct sign/false sign conditions (for examples of sentences, see Appendix). The comct/false conditions were created by showing the sign, stating that the sentence would be happy, sad, typical or atypical. The sign was correct in half of the cases and false in the other half with respect to the content of the sentence. The order of the sentences was randomized with the restriction ’For syntactical reasons “typicality” has sometimes been exchanged with “cognitive”.

Emotionality and typicality in speechreading

S a n d J Psycho1 36 ( 1995)

EXPERIMENTAL DESIGN Script: A vlslt to a reStaurantlA vlslt to a doctor HappylSad faclel 0Xp~SSlOn HapWfaca Emotional sentences

1

Sadface

2

Happy sentence Sad sentence Sadface

3

Happyface

Neutral taclal exprerslon Neutralface

5

Happy sentence 4

Neutralface

7

Neutralface

6

Sad sentence Neutralface

8

fake sign Happy sentence Sad sentence

correct sign Typlcallatyplcal sentences

Happy sentence

Sad sentence

Neutralface

Neutralface

9

Typical sentence Neutral face

I

10

Atrpical sentence

1 1 Neutral face

12

fake sign Typical sentence

Atypical sentence

Fig. 1. Layout of experimental design, with the conditions numbered from 1 to 12. that only two sentences from one and the same condition were allowed to occur in succession. By reversing the random order, both for the scripts and the sentences, and by switching the correct sign/false sign conditions, a second random order was created. The rated mean (on the 7-pint scale) differences for the happy vs. sad, and typical vs. atypical sentences were found to be significant for both scripts (ps < 0.01). The mean differences for the happy vs. emotionally neutral and sad vs. emotionally neutral sentences were also significant for both scripts ( p s < 0.01). and, there was a significant difference between the means for typical and atypical, compared to the cognitively neutral sentences (ps < 0.04). The frequency of the words in each sentence, based on the Occurrence out of one million m n h g words in Swedish newspaper texts (Allen, 1972), was counted in order to control for a possible confounding with condition. T-tests revealed no differences betweeo the contrast typical vs. atypical, t (14) = 0.31, p > 0.05, the contrast happy vs. sad. r(30) = 1.69, p > 0.05, the contrast emotional vs. cognitive, r ( 4 6 ) = 1.27, p > 0.05, or between the sentences in the doctor script vs. those in the restaurant script, t(46) = 0.86, p > 0.05. Number of propositions (Kintsch, 1974) was also counted in order to control for a possible confounding with condition. T-tests revealed no difference between the contrast typical vs. atypical, r( 14) = 1.42, p > 0.05, the contrast happy vs. sad, r( 30) = 0.23, p > 0.05, the contrast emotional vs. cognitive: r ( 4 6 ) 0.45, p > 0.05, or between sentences in the doctor script vs. those in the restaurant script t (46)= 1.73, p > 0.05. As the sentences were simple declarative statements or simple questions, with no inserted subordinate clauses, grammatical complexity was deemed equal across experimental conditions. Thus, the manipulations made arc neither confounded by word frequency, or propositional content, nor by grammatical complexity. In addition, realiability estimates were computed. The 18 judges were split into two halves by random assignment and their mean judgements were correlated. For the typicality dimension, the means for each subject were computed for the typical (four values) and atypical (four values) sentences pooled over scripts, resulting in r = 0.88. For the motional dimension, the means for each subject were computed for the happy (eight values) and the sad (eight values) sentences, pooled over scripts, resulting in r =0.93. Thus, the overall reliabilities are satisfactorily high. Procedure. In the experimental situation, the subjects were seated at a 3 m distance from the TV screen. They were told to read the instructions, which were written on the first page in the answer shect, and informed that all the sentences would be presented without sound. They were then informed about the script, embedding the sentences-to-be-sphread: half of the subjects started with the restaurant script and the other half with the doctor script. Before the actor was shown on the TV screen they were f

19 1

192

K. Johansson and J. Rdnnberg


instructed to read a sentence, printed on the answer sheet, which offered extra contextual help, in that it described the content in the sentence-to-be-speechread(Lyxell& Rdnnberg, 1987; 1989). In the general context of, for example, the restaurant script, the sentences consisted of a phrase (e.g., “this is about”) and a cue word (e.g., ‘‘ordering”). This was done because a poor context generally gives a too low speechrcadiq performance to observe diKercntial effects of manipulated variables (Samuelsson & RBnnberg. 1991). Finally, the subjects were told that the written signs, which the experimenter would show, announced whether the sentence-to-&-speechread would be sad or happy, typical or atypical. The video-tape was organised as follows: first, a red colour was shown on the screen for eight seconds, then the act03 was shown (face and shoulder) on the screen. presenting the sentence with the sound turned off, after which a blue colour was shown for 15 seconds. This also constituted the response interval. During the response intmal, the subjects were instructed to write down all they had been able to speechread on the answer sheet, and they were encouraged to guess if they did not decode the who& sentence. Aher the response interval, the red colour was shown again for eight seconds and during this time the subjects were instruced to read the extra contextual sentence and the handwritten sign. The experimental session started with two practice sentences. Immeditely after that, the experiment started, and when all sentences from the first script were completed the experimenter stopped the video. At this point, subjects were given instructions about the new script. They were also informed that the continuation of the experiment would follow the same procedure apart from the new script. After this short break. the exgmiment continued until it was completed. Design. The general design (see Fig. 1) of Experiment 1 is a factorial design where the first factor refers to the script variable (restaurant and doctor). The facial expression variable (happylsad face or neutral face) constitutes the second factor, the third factor refers to the emotional or typicality status of the sentence (with happylsad face the sentences were always emotional, and with neutral face the sentences could be either emotional or cognitive), and the fourth factor refers to the sign shown (correct vs. false). Each condition was replicated once and the mean value of the two observations constituted the raw data in each a l l . All factors are within-subjects variables.

Results and discussion

The sentences were scored on a content basis, as the content words best reflect whether the sentence is happy, sad, typical or atypical. The proportion correct content words was obtained by dividing all content words the subject had been able to speechread with all content words per sentence, pooled over both observations for each subject and condition. Two separate ANOVAs (SPSSx) were computed with the main purposes (1) to evaluate the effects of happy/sad facial expression and (2) to evaluate the effects of all types of sentences, with a neutral facial expression. Note that the design is not completely crossed as cognitive sentences do not carry any unequivocal happy/sad facial expression. Therefore, no overall ANOVA was computed. ANOVA 1. The first ANOVA evaluated the emotional sentences, both with happy, sad and neutral facial expression (i.e., conditions 1-8, see Fig. 1, adding script), including four variables: script, type of sign, emotional facial expression vs. neutral facial expression, and emotional content (sad/happy). A main effect of type of sign was obtained, F( 1,20) = 7.83, Mse = 0.03, p < 0.05, where correct sentences (x = 0.13) were found to be better speechread than false sentences (x = 0.07). There was also a main effect of emotional content, F( 1.20) = 10.03; Mse = 0.03, p c 0.05. Sad sentences (x = 0.13) were superior to happy sentences (x = 0.06). As seen in Table 1, the two-way interaction between script and type of sign was significant, F(1,20) =4.85, Mse =0.01, p ~ 0 . 0 5 .Correct restaurant sentences were not easier to speechread than false restaurant sentences, t(40) = 1.86, p > 0.05, but correct doctor sentences were better speechread than false doctor sentences, t ( @ ) = 4.93, p < 0.05. When the subjects’ expectations of the sentence-to-be-speechread (induced by the correct sign) were compatible with the actual sentences, performance was enhanced. In the doctor rIhe actor was a professional who was hired to record the sentences.


Scand J Psycho1 36 ( 1995)

Table 1. Mean performance OR the speechreding test in Ekperiment I as a f i c t i o n of script (doctor1 restaurani) and type of sign (correctrake)

Correct False

Doctor

Restaurant

0.16 0.08

0.10

0.07

script, it is likely that the subjects can imagine more vaned messages as the script is not a very stereotyped one. Here, the correct sign also constrains the number of possible inferences, and hence, facilitates speechreading. Furthermore, as the doctor script is more emotionally loaded than the restaurant script, expectations of emotional sentences are more likely in the doctor script. As the restaurant script is not primarily an emotional one, the expectations of emotional sentences in the restaurant script are less likely. Instead, the subjects presumably had expectations of a stereotyped script with specific events, and therefore, they were not mislead by a false sign concerning the emotional sentences. As can be seen in Table 2, there was also a significant two-way interaction between type of script and type of emotional content, F( 1.20) = 15.97, Mse = 0.04, p < 0.05, hence qualifying the main effect of emotional content. Sad doctor sentences were easier to speechread than happy doctor sentences, t(40) = 7.17, p < 0.05, but happy restaurant sentences were not better speechread than sad restaurant sentences, t (40) = 1.33, p > 0.05. Finally, it should be noted that no significant effects attributable to facial expression emerged. According to script theory (Abelson, 1981; Anderson, 1983; Belleiza & Bower, 1981; Nottenburg & Shoben, 1980; Yekovich & Walker, 1986), sentences typical of a certain script are better speechread than atypical sentences for the same script. For the doctor script, Sentences with a sad emotional content are more frequent (and thus typical) than sentences with a happy emotional content. In the present study sad doctor sentences were better speechread than happy sentences. But facial expression was not necessary for the activation of the sad sentences. As the restaurant script is a very stereotyped one, none of the emotional sentences were easier to speechread than the other emotional sentences. Table 2. Mean performance on the speechreading test in Experiment 1 as a function of emotional content (happy/sod) and script (doctorlrestaurant)

Happy Sad

Doctor

Restaurant

0.05 0.19

0.07

0.10

ANOVA 2. The second ANOVA evaluated the sentences presented with a neutral facial expression, both emotional and cognitive sentences (i.e., conditions 5-12, see Fig. 1, adding script). The ANOVA included three variables: script, type of sign and type of sentence. In order to study each sentence type separately, the type of sentence variable was split into the four levels: sad, happy, typical and atypical. There was a main effect of type of sentence, F(3,19) = 38.53, Mse = 0.05, p < 0.05. The means for this variable were: typical sentences (x = 0.30), atypical sentences ( x = 0.09,sad sentences (x = 0.13), and happy sentences (x = 0.07). Typical sentences were significantly easier to speechread than all other sentences ( p s < 0.05). Sad sentences were not si@cantly easier to speechread than happy sentences (p >0.05).

193

194

K. Johansson and J. Rcnnberg


Table 3. Mean per$ormance on the speechreadmg test m Experiment J as a function of script (doctor1 restaurant) and type of sentences (typicallatypicallhappylsad)

Typical Atypical Happy Sad

Doctor

Restaurant

0.18 0.11 0.06 0.19

0.42 0.03 0.01 0.08

As can be seen in Table 3, there was also a sigmficant two-way interaction effect between script and type of sentence, F(3,19) = 6.25, Mse = 0.04, p < 0.05, such that typical restaurant sentences were easier to speechread than all other restaurant sentences ( p s < 0.05). Again, sad doctor sentences were easier to speechread than happy doctor sentences, f ( 4 0 ) = 2.73, p < 0.05. With respect to the prediction that typical sentences should be better speechread than atypical sentences, this could be confirmed for the restaurant script, thus replicating Sainuelsson and RBnnberg (1993). It seems that we are guided by the script and are expecting typical sentences to occur (Samuelsson & RBnnberg, 1993). The restaurant script in itself may be more familiar to the subjects than the doctor script. The restaurant script, in this study, also involved distinct events, from which very typical sentences evolved. For the doctor script sad sentences were the easiest to speechread but sentences typical for the doctor script were speechread almost as good as the sad sentences. Thus, even for this highly emotional script, typical sentences were important in promoting speechreading. The main conclusions from Experiment 1 are P t for the less stereotyped doctor script, correct cueing and the expected sad sentences represent important deliminations for speechreading. For the more stereotyped restaurant script, these delimitations were not necessary. Also, effective support for speechreading of typicality sentences are provided by a neutral face presenting messages that are deemed typical of a particular script.

EXPERIMENT 2 Experiment 2 was conducted (a) to explore the hypothesis that sentences with a congruent facial expression and emotional content are better speechread than sentences with a neutral facial expression, (b) to replicate and generalize the main results of Experiment 1 to a group of normal hearing students, and (c) to evaluate the potential benefits of a tactile hearing-aid, with the specific hypothesis that tactile supported speechreading would allow for an even more efficient pick-up of emotional aspects of the spoken message, that is, introducing a condition with both information from facial expression and information derived directly from the auditory speech signal. The rationale behind this hypothesis is that the time-intensity information that is extracted and t r a n s d u d onto a surface on the skin (in this case the wrist) would represent some of the prosodic elements that are important for the perception of emotion (aster & Risberg, 1986). Method Subjects. Twenty normal-hearing (twelve men) students participated in the experiment. Their mean age was 28.8 yean (SD = 8.48 years; range = 20-50 years). Materinls, procedure and &sign. Experiment 2 had the same basic structure as Experiment 1. The material used was the same as in Experiment 1. No signs were used in this experiment. Instead the correct sign both for the emotional and the cognitive sentences were “ r e p l d ’ with a tactile


Scand J Psvchol 36 (1995)

“hearing-aid” (Minivib 3). The Minivib 3 is a simple device, which presents an amplitude-modulated 250-Hz sinusoidal signal to a vibratory transducer that can be held in the hand or worn on the wrist. The resulting amplitude-modulatedsignal can be used to provide the wearer with information about selected aspects of intensity and duration of input sound. Potentially, such devices would be useful in conveying information about the presence of an acoustic stimulus and its rhythm and stress pattern, and might also provide fine-structure phoneme-level information of a temporal nature (e.g., vowel duration) (Weisenberger & Russell, 1989). The Minivib 3 transducer is rectangular in shape, with dimensions of 6.5 x 4.3 x 1.7 cm.In this experiment the Minivib 3 was wrist-worn and controlled by the experimenter. It was connected to the video such that only the sound waves of the actor’s voice were transmitted to the aid. Thus, no potential background noise was allowed to interfere. The experimenter informed the subjects, before the sentence, when the aid was to be used. The procedure then was identical to ’ Experiment 1, except for the replacement of the sign with the tactile “hearing-aid”.

Results and discussion

The scoring procedure was identical to that of Experiment 1, and two analogous ANOVAs (SPSSx) were computed. A third ANOVA (SPSSx) was also computed with the purpose to evaluate the effects of congruent facial expressions. ANOVA 1. The first ANOVA evaluated the emotional sentence conditions with happy, sad and neutral facial expression (i.e., conditions 1-8, adding the script variable). The ANOVA thus included four variables: script, emotional facial expression vs. neutral facial expression, emotional content (sad/happy), and presencelabsence of the tactile information. Two main effects emerged: Speechreading of sentences supported by the tactile “hearing aid” ( x = 0.19) was facilitiated compared to sentences with no tactile support (x = 0.14), F( 1, 19) = 5.62, Mse = 0.05, p < 0.05, and sad sentences ( x = 0.21) were superior to happy sentences ( x = 0.13), F( 1, 19) = 16.91, Mse = 0.03, p < 0.05, as in Experiment 1. There was a general enhancing effect of the Minivib 3 for the emotional sentences, where the speaker’s tempo, rhythm and intonation etc. seemed to be picked up and reinforced. Emotional information added to the time-intensity information that the Minivib 3 is designed to pick up. As can be seen in Table 4, there was a significant two-way interaction between script and facial expression, F( 1, 19) = 15.16, Mse = 0.03, p < 0.05. Restaurant sentences with neutral facial expression were easier to speechread than restaurant sentences with emotional facial expression t( 38) = 2.23, p < 0.05, and doctor sentences with emotional facial expression were easier than doctor sentences with neutral facial expression t(38) = 3.17 p < 0.05. This interaction constitutes a further indication that the restaurant script might be conceived of as a cognitive script. It even appears as if facial expression is a distracting factor in an otherwise stereotyped, cognitive script. Contrary to this, the doctor script is an emotional script, and an accompanying facial expression is more likely to be expected (cf. Experiment 1). As can be seen in Table 5 there was also a significant two-way interaction between script and emotional content, F( 1, 19) = 25.85, Mse = 0.04,p < 0.05. Again, sad doctor sentences were better speechread than happy doctor sentences, t(38) = 6.51, p < 0.05, whereas happy restaurant sentences were not easier to speechread than sad restaurant sentences, t(38) = 1.02, p > 0.05. Table 4. Mean performance on the speechreading tesr in Experiment 2 as a fwction of script (doctor/ restaurant) and facial expression (hcrppylsad us. neutral)

Happy/sad expression Neutral expression

Doctor

Restaurant

0.21 0.13

0.13 0.19

195

196

K. Johansson and J. Rtinnberg


Table 5 . Mean performance on the speechreading test in Experiment 2 as a function of emotional content (hnppylsad) and script (doctor/restaurant) Doctor

Restaurant

Happy

0.07

Sad

0.26

0.18 0.15

The result that sad doctor sentences were easier to speechread than happy doctor sentences is reasonable since sad messages are likely to be expected when visiting a doctor. This result was straightforward and it also constitutes a replication of Experiment 1. Thus, we can generalize the main results to be valid for normal-hearing subjects. ANOVA 2. The second ANOVA evaluated the sentences with neutral facial expression, both emotional and cognitive sentences (i.e., conditions 5-12, adding the script variable). The ANOVA thus included three variables: script, presencelabsence of the tactile information, and type of sentence. Two main effects yielded significance in ANOVA 2. The restaurant script (x =0.25) was better speechread than the doctor script (x = 0.17), F(1, 19) = 9.88, Mse = 0.05, p < 0.05. The main effect of type of sentences was significant, F( 1.19) = 54.52, Mse = 0.05, p < 0.05. The means were: typical sentences ( x = 0.42), atypical sentences (x = 0.1 I), happy sentences (x =0.13) and sad sentences ( x -0.19). Typical sentences were easier than all other sentences, hence replicating Experiment 1. Also, note the absence of effect of the tactile aid. There was a main effect of script, with restaurant sentences being better speechread than doctor sentences. This effect was found for the sentences without facial expression (see ANOVA 1). Typical sentences were also much easier to speechread in the restaurant script than in the doctor script, r (38) = 11.24, p < 0.05 (see Table 6). The doctor script was more dependent on facial expression, as sentences accompanied with a facial expression were easier to speechread than sentences without facial expression (see ANOVA 1). However, this facial expression by script interaction did not replicate the data from Experiment 1, and hence, this aspect of the above reasoning must be seen as tentative. What is certain is that facial expression did not show any general facilitative effect in either experiment. As can be seen in Table 6, there was a significant two-way interaction between script and tw of sentences, F(3, 19) = 5.94, Mse = 0.03, p < 0.05. This interaction is mainly due to the fact that typical restaurant sentences were easier than all other restaurant sentences and that sad doctor sentences were easier to speechread than happy doctor sentences, r(38) = 3.59, p 0.05. The result for the restaurant script that typical sentences were best speechread was a replication of the result in Experiment 1. This result supports the script theory (Abelson, Table 6. Mean performance on the speechreading test in Experiment 2 as a f i c t i o n of script (doctor/ restaurant) and type of sentences (typicallatypical/happy.ppvlJad)

.

Typical Atypical Happy Sad

Doctor

Restaurant

0.28 0.16 0.05 0.21

0.56

0.05 0.2 1 0.17


Scand J Psycho1 36 (1995)

1981; Anderson, 1983; Bellezza & Bower, 1981; Nottenburg & Shoben, 1980; Yekovich & Walker, 1986). Sad doctor sentences were easier than happy doctor sentences, also replicating Experiment 1. ANOVA 3. The third ANOVA evaluated the sentences with a congruent facial expression (i.e., conditions 1 2 versus conditions 5 6, see Fig. 1, adding script), including three variables: script, emotional facial expression vs. neutral facial expression and emotional content (sad/happy). As seen in Table 7, the two-way interaction between script and congruent facial expression was sigIuficant, F( 1,19) = 15.30, Mse = 0.06, p < 0.05. Doctor sentences with a congruent facial expression (and emotional content) were easier to speechread than doctor sentences with a neutral facial expression, t(38) = 3.19, p < 0.05, and restaurant sentences with a neutral expression were not better speechread than restaurant sentences with a congruent facial expression (and emotional content), t( 38) = 0.50, p > 0.05. As the doctor script is less stereotyped, messages with a congruent facial expression can be a valuable delimitation in promoting speechreading. For the stereotyped, cognitive restaurant script, with a congruent facial expression, the differences between these sentences and the sentences with a neutral facial expression disappeared (compred to ANOVA 1, when a false facial expression also was present). Thus, in general terms it seems as emotional content (i.e., sad sentences) is important for the doctor script and typicality is important for the restaurant script and this holds true for both hearing-impaired (Experiment 1) and normal-hearing subjects (Experiment 2). Also, at least for the normal-hearing, when the emotional content in the sentences was congruent with the emotional expression of the actor’s face, the subjects’ performance was higher than for the sentences without any facial expression. Two further aspects of the data were explored: First, to control that differences in word frequency were not responsible for the differences in speechreading performance across the various conditions, correlations between the word-frequency and speechreading performance were computed, for each group (hearing-impaired and normal-hearing), and for both groups together. There were no significant correlations on an overall basis, neither for typicality sentences nor for sad sentences. The only sigruficant correlation was found for happy sentences, r( 14) = 0.69, p c 0.05, for the hearingimpaired group. However, the correlation for the happy sentences does not explain why sad sentences were better speechread, given that the overall frequencies did not differ. Consequently, the effects obtained are not due to word-frequency per se. Secondly, in the restaurant script the typical sentences were better speechread than both atypical and emotional sentences. But the 18 subjects’ ratings for the restaurant script, on the typicalitydimension scale, (SD = 1.02), did not conform to the mean more closely than the ratings for other restaurant sentences (SDs = 0.84, 0.84, 1.32 for happy, sad and atypical, respectively), or for doctor sentences (SDs = 1.13, 0.90, 1.32, 1.08 for happy, sad, typical and atypical respectively). Thus, the typical restaurant sentences were not “tighter” or more “precise” in their alleged properties than the other sentences. Therefore, this potential confounding effect is not responsible for the obtained typicality effect.

+

+

Table 7. Mean perfonrvmce on the speechreudmg test m Experiment 2 as n function of script (doctor/ restaurant) and fiicrl expression (congtuentfiial expressionlneutralfacial expression)

Congruent facial expression Neutral facial expression

Doctor

Restaurant

0:27 0.12

0.18 0.20

197

198

K. Johansson and J. R6nnberg

Scan J Rychol 36 (1995)

GENERAL DISCUSSION The general prediction raised in the present study that sentences with emotional content (happy and sad) accompanied by a distinct facial expression should be better speechread than sentences accompanied by a more neutral facial expression could not be confirmed. Instead, information about a specific script brings Frtain expectations for the script into mind and activates possible messages and actions of this script, thereby preparing the actors for speechreading of certain messages. The doctor script is a less stereotyped and less f d a r script than the restaurant script. It can be argued that the subjects did not have much control over the situation as the sentences were spoken by the doctor. Therefore, it was difficult 'to anticipate what sentences that could appear. With a correct sign, then, the possible messages to be expected were delimited. Still, emotionally charged messages (i.e., sad sentences) represent sentences that are likely to occur in a doctor script. The data for the doctor script also suggest that sad sentences were easier to speechread than happy sentences. Another factor relevant for promoting speechreading in the doctor script was facial expression. For the normal-hearing subjects, sentences accompanied with a facial expression were better speechread than sentences without facial expression. This effect was present only when facial expressison was congruent with the message: The hearing-imparied group speechread the doctor script just as well, independently of facial expression. Apparently, the content of the message was more important for the hearing-impaired than the accompanying expression. As the doctor script can be considered an emotional script, normal-hearing persons may expect accompanying facial expressions to OCCUT in this situation. Therefore, they speechread the sentences better with facial expression than without. The hearing-imparied, on the other hand, still expecting facial expression, seem to have developed a strategy that focuses on the scripts themselves (e.g., emotional scripts). Therefore, when concentrating on the spoken message they appear to disregard the surrounding environment, including the accompanying fadial expression. Also, they are probably more conscious of the possibility that facial expressions and other environmental aspects can be misleading, or at least confusing. Their communication tactic is to turn to the source of information (i.e., to watch the speaker's lips) and not to depend on information that occasionally is not there (i.e., facial expression). In MtUrd communication, face and message content w i l l t y p i d y agree, so it is probably not the facial expression per se that facilitates the understanding of the message. Instead, it is the congruent combination of face and message content that is crucial. For the restaurant script. typical sentences were much better speechread than atypical sentences. This result was in line with the prediction from cognitive script-theories (Abelson, 1981; Anderson, 1983; Bellezza & Bower, 1981; Nottenburg & Shoben, 1980; Yekovich & Walker, 1986). The restaurant script is a very stereotyped one and the subjects are more familiar with the role of the guest. It thus appears that knowledge of the script content is very &dent in promoting speechreading success (Samuelsson & RGnnberg, 1991; 1993), and that type of script i4 crucial for the level of speechreading obtained. Also, for the restaurant script, the relative contribution to speechreading from typicality is larger than that from the emotional flavour of the message. But here, too, emotional sentences accompanied by a congruent facial expression were speechread as &ciently as emotional sentences without any facial expression. So, even if facial expression is not expected to be part of this more cognitive script, it does not hinder the speechreading process. To summarize, for the less stereotyped doctor script several deliminations are necessary for efficientspeechreading: (a) when the expectations of the-message-to-be-speechreadare in line with the actual message (through the correct sign), (b) when sad sentences are in line with the doctor script, and (c). when the accompanying facial expression is congruent with the

Sand J Psycho1 36 ( 1995)

Emotionality and typicality in speechreadrhg

spoken message. For the stereotyped restaurant script none of these delimitations is necessary. When given information of this script, typical sentences were expected, and hence, speechreading of the typical sentences were superior compared to atypical and emotional sentences. Nevertheless, the general role of facial expression must not be overlooked. For normal-hearing it can be important in assuring the subjects that they have grasped the overall situation, thereby making the speechreading sitution more relaxed and less energyconsuming (Lyxell er ul., 1992). The effect of tactile-supported speechreading was only present in conditions with emotional sentences. This result is important in its own right, because emotional sentences supply important prosodic aspects that obviously are at the core of the perceptual possibilities provided by the Minivib 3. Presumably, this mutual interaction supports the decoding of the visual speech signal (Ohngren ef d.,1992). Thus, it does seem as if there are “more” prosodic aspects that the Minivib 3 is able to pick up in the emotional sentences compared to the cognitive sentences. Also, there did not seem to be any difference between the two types of emotional sentences, at least concerning the speaker‘s tempo, rhythm and intonation picked up by the Minivib 3. This finding is important as it opens up the possibility that the application and development of tactile aids must take into account other aspects of communication than the speech signal per se. Finally, it must be noted that there is a clear difference between the hearing-hnpaxied who rely on speechreading and deaf persons who rely on sign-language. Sign-language is a language constituted of gestures and facial expressions, and naturally, facial expressions are important. These factors can also be found in spoken communication, but they do not seem to be of a general facilitating kind, at least not in the present study. Despite the fact that spoken language has certain aspects in common with sign-language (Pettito C Marentette, 1991; SWerfeldt et al., in press), the present data do not suggest that there is some automatic transfer of factors important in.perception of sign-language to visual perception of spoken language. This study was supported by a grant from the Swedish Council for Social Research awarded to the second author (No E 90/163). Ulla-Britt Persson and Runa Pate1 are thanked for checking the language.

REFERENCES Abelson, R. P. (1981). Psychological status of the script concept. American Psychologist, 36 (7). 715-729. Anderson, J. R. (1983). A spreading activation theory of memory. Journal of Verbal Learning and Verbal Behavior, 22, 26 1-295. Allen, S . (1972). Top ten thousand. Word frequencies in Swedirh newspaper text. Almqvist & Wiksell. Bellezza, F. S., & Bower, G. H. (1981). The representational and processing characteristics of scripts. Bulletin of the Psychonomic Society, 18, 1-4. Buck, R. (1984). The Communication of D o t i o n . New York: Guilford. Ekman, P. (1973). Cross-cultural studies of facial expression. In. P. Ekman (Ed.), Darwin Md facial expression (pp. 169-222). New York Academic Press. Ekman, P. (1992). Are there basic emotions? Psychological Review, Vol. 59,No 3, 550-553. Ekman, P., Friesen, W. V. & Ellsworth, P. (1982). Emotions in the human face. Cambridge, England Cambridge University Press. Izard, C. E. (1977). Human emotions. New York: Plenum. Kintsch, W. (1974). The representation of menaing in memory. Hillsdale: LEA. LyxeU, B. & Rennbcrg, J. (1987). Guessing and speechreading. British J o d of Audiology, 21, 13-20. Lyxell, B. & ReMberg, J. (1989). Information-processing skill and speechreading. British Journal oj Audiology, 23, 339-347. Lyxell, B., Lidestam, B., Johansson, K. & RGnnberg, J. (1992). Face expression and speechreading skill. Subm. m.

199

200

K. Johanrson and J. Rtinnberg

Scan J Psycho1 36 ( 1 9 5 )

Notteaburg, G. & Shoben, E. J. (1980). Scripts as hear orders. Journal of Experimental Social P~chology,16, 329-341. Pettito, L. A. C Marantdte, P. F (1991). Babbling in the manual mode: Evidence for the ontogeny of language. Science, March, 1493- 1496. Samuelsson, S. & Riimberg, J. (1991). Script activation in lipreading. Scandinavian J o m l of PsycholO ~ Y , 32, 124-143. Samuclsson. S. & Riinabcrg, J. (1993). Impfiat and explicit use of scripted constraints in lipreading. European Journal of Ggnitiue Psychology, 5, 201-233. Schaak, R. C. & Abelson, R. B. ( 1977). Scripts, plans, goals and understanding. Wsdalc. N.J.:Erlbaum. SMerfeldt, B., RGaaberg, J. & Risberg, J. (in prtss). Regional cerebral blood flow in sign-language users. Brain and Language. Weisenberger, J. M. & Russel, A. F..(1989). Comparison of two siagle-chanael vibrotaaile aids for the hearing-impaired. Journal of Speech and Hearing Research, 32, 83-92. Yekovich, F. R. & Walker, C. H. (1986). Retrieval of scripted concepts. Joumul of Memory and h g u a g e , 25, 627-644. (Ihagrm, G.. Riinnberg, J. & Lyxell, B. (1992). Tactiliag: A usable support system for speechding? British Jowml of Audiology, 26, 167-173. b e r , A-M. & Risberg, A. (1986). The identification of the mood of a speaker by hearing-impaired listeners. Speech Transmhwn Luboratory-Quarterly Progress and Status Report, 4, 79-90. Received 2 June 1993 Accepted 18 Jaauary 1994

APPENDIX Translated examples of sentences used in Experiment 1 and Experiment 2 Sad doctor sentences I am afraid we have to keep you for observation. This does not look good. Happy doctor sentences The medicine seems to be having effect. Almost everyone recovers completely after treatment. Typical doctor sentences I will give you a prescription. Please sit down. Atypical doctor sentences Are there many patients in the waiting-room? The treatment you will have is very expensive.

Sad restaurant sentences This restaurant is very smoky. I ordered my steak well-done and not rare. Happy restaurant sentences We would like to order a bottle of champagne. The food here is really worth its price. Typical restaurant sentences Can we pay with our creditcard? We would like to see the menu. . Atypical restaurant sentences We would like to have some flowers on our table. Can you hang my coat next to the black coat?