Forceville, Charles

7 downloads 0 Views 75KB Size Report
Terry D. Royce, Wendy L. Bowcher (Eds), Lawrence Erlbaum, Mahwah, NJ, 2007, 403 pp.,. ISBN 0-8058-5106-2 (cloth). A rapidly waxing flow of “multimodality” ...
1

The following is a pre-proof version of a book review that was subsequently published as: Forceville, Charles (2009). Review of Terry D. Royce & Wendy L. Bowcher (eds), New Directions in the Analysis of Multimodal Discourse (Lawrence Erlbaum 2007). Journal of Pragmatics 41(7): 1459-1463 ( doi:10.1016/j.pragma.2009.01.007). If you want to quote from it, please check the published version. [ChF]

New Directions in the Analysis of Multimodal Discourse Terry D. Royce, Wendy L. Bowcher (Eds), Lawrence Erlbaum, Mahwah, NJ, 2007, 403 pp., ISBN 0-8058-5106-2 (cloth)

A rapidly waxing flow of “multimodality” floods publishers’ catalogues, journals, and conferences. This is understandable, since discourses that draw on more than just the good old language modality present themselves in ever more emphatic ways, due to developments such as the growing multifunctionality of cel phones, the rise of computer games, and the huge popularity of Youtube and of surfing on Internet in general. In this monomodal review I discuss a book that promises innovative perspectives on multimodality, Royce and Bowcher’s edited volume New Directions in the Analysis of Multimodal Discourse. The coherence between the twelve chapters in the book resides mainly in the adherence to the Systemic Functional Linguistics (SFL) approach rooted in Hallidayan grammar, while Kress and Van Leeuwen’s (1996) pioneering application of this approach to the visual realm is cited in every chapter. I should say straightaway that I have no first-hand knowledge of Halliday’s work. This has the drawback that I cannot verify how true to the letter or spirit the authors are to their primary sources of inspiration, and the advantage that I can assess with relative freshness what promising tools SFL has to offer for analyzing

2

multimodality. Let me further add that my own primary affiliation is to Cognitive Linguistics and Relevance Theory models, as developed by such scholars as Lakoff, Johnson, Gibbs, Kövecses, Turner, Fauconnier, and Sperber and Wilson. However, I consider myself a critical and eclectic follower of these paradigms. The tree is known by its fruits, and I will happily make use of any theoretical insight that can help understand multimodality, irrespective of its source. I will first briefly, and with no claim to exhaustiveness, discuss each chapter separately, and end by charting and evaluating shared tendencies. The 60 page-long first chapter by Matthiessen is devoted to “The multimodal page: A Systemic Functional exploration.” After some preliminary remarks, the author proposes first to “examine multimodality within language” (p. 11), because allegedly “the ideational resources of language provide us with a range of strategies for modeling […] all modes and media of expression” (p. 21). Then I more or less lose track. I cannot help feeling that the many technical terms Matthiessen introduces come as a substitute for explanation, rather than as its prerequisite. The complex panels and figures (figure 1.14 has more than 80 boxes!) are not particularly encouraging either. And while the author makes some interesting observations on “articulation” and “prosody” as distinguishable modalities in spoken language, and on the rhetorical patterns in verbal texts, he has very little to say about the visual realm, let alone about interrelations between word and image. Royce usefully begins his chapter “Intersemiotic complementarity: A framework for multimodal discourse analysis” by briefly outlining the three metafunctions Halliday distinguishes in language: the ideational (pertaining to “meaning” as it arises out of the relation between text and world), the interpersonal (pertaining to the relation between communicator and addressee), and the textual (pertaining to the internal coherence of a discourse and the way it links up to extratextual context). Royce argues that these metafunctions are applicable to non-verbal communication as well and uses an Economist

3

article, accompanied by a cartoon and a diagram, to demonstrate that “repetition,” “antonymy,” “hyponymy,” “meronymy,” and “collocation” are concepts that define relations both within and straddling the verbal and visual modalities on the ideational level. He comes up with similar modality-independent concepts for the interpersonal and textual levels. Moreover, he introduces the concept of “visual message elements” (VMEs), “visual features which carry semantic properties […] potentially realized by a variety of visual techniques at the disposal of the visual designers” (p. 70). The in-depth analysis of a single case-study in the light of clearly outlined theoretical concepts provides a useful starting point for applications by other scholars to other (types of) texts. Thibault’s chapter focuses on the visual dimensions of language. He proposes a typology of graphic dimensions of verbal elements (e.g., a “grapheme” is the visual expression of the verbal “letter,” and “logogram” corresponds to “word”). This is insightful, but it is a pity that the author tends to be rather verbose, and to use an overdose of jargon to explain sensible but uncomplicated ideas. He goes on to illustrate how the levels of content and graphic expression combined create meaning in a Duracell ad, drawing on Gestalt theory concepts (figure-ground, spatial orientation, distance, similarity, continuity of direction, directionality or orientation, closure). Thibault ends his chapter by showing how visually deviant language (as in comics and cartoons) can resonate with visuals in a way that enhances a sense of rhythm that is difficult to capture in words alone. In a fine chapter, Bateman, Delin, and Henschel convincingly expound how the impact of content shifts when it is presented in different media. They come up with lists of verbal, visual, and production/consumption related dimensions that need minimally to be taken into account for an adequate description of how multimodal documents communicate, and that allow for comparison between documents belonging to the same genre. What I particularly like in this research, which is part of a larger “Genre and multimodality” project, is that it

4

combines commonsense theoretical concepts with honest, systematic data analysis, the central idea being that one can only discover interesting patterns after identifying and quantifying salient variables. The case study, which compares traditional and electronic versions of newspapers, elegantly underscores this point. Baldry reports about the benefits and possibilities of the Multimodal Corpus Authoring (MCA) system he helped develop. He, too, emphasizes the importance of corpus research, and advances proposals for the labelling and description of pertinent variables in multimodal discourse – indispensable for enabling generalizations and formulating testable hypotheses. Among other things, he discusses the issue of segmenting a text, taking into account linguistic, prosodic, and visual features. A case study exemplifies the approach, and further cooperation is invited via the MCA, accessible online. Lim takes his cue from fellow contributor Martin ’s earlier work to develop visual counterparts of verbal organization mechanisms, specifically visual taxonomy, visual configuration, visual reference, and visual taxis, and applies them to comics. He relies on McCloud’s (1993, pp. 70-72) six types of frame-transition, even though these types are sometimes difficult to distinguish. Among the concepts he develops and discusses are the “Associating Element” (e,g., a cornflakes package connotes “breakfast” and a single tree connotes “this is an outdoor scene”), and the “visual linking device” (VLD), the latter enabling a viewer to understand how the action in one panel relates to that in a previous panel, e.g., the repetition of (part of) a character or scene. Martin and Stenglin describe how spatial organization and layout in a “post-colonial” museum in New Zealand, bridging indigenous and “white” perspectives on history and culture, is made communicatively significant. Among the “tools” they deploy in their analyses are the “given-new” dichotomy, “bound” versus “unbound” spaces,” and “orbital” versus “serial” navigation paths. The authors focus on a single modality only, but persuasively argue

5

that this modality – space – needs to be taken into account in communication that involves physical motion. Bowcher draws on her co-editor’s model for “A multimodal analysis of good guys and bad guys in Rugby League Week.” Her analysis, of a newspaper article, is dauntingly detailed. Although mainly focusing on verbal aspects, the author also discusses photographs and headings, and their location on the page. Importantly, she tries to show how information in the verbal and visual modalities are connected. Her tentative claim that the direction in which the rugby players in the photographs look correlates with where in the verbal text “goodness” and “badness” are discussed is suggestive. But even if correct much more case-studies wil have to be conducted to assess whether this mechanism is generalizable. Mohan et al., claiming that “the mere fact that similar graphics occur in textbooks [i.e., scientific textbooks in different languages, ChF] does not mean that readers of those textbooks interpret them in the same way” (p. 277), asked Chinese, English, and Japanese readers to respond in their own language to a diagram representing the “water cycle.” A sample of the texts was subsequently analyzed in more detail, revealing that “some participants produced texts which had a sequential line of meaning, while others produced texts that explicitly reflected causal relationships” (p. 278). The authors then hypothesize that “causal line texts exemplify the features of scientific discourse more than do time line texts” (ibid.). The hypothesis is confirmed – but what has been proven? Apparently, that the “causal line” subjects used more typically “scientific” constructions, such as nominalization. But surely there is something circular going on here: we can only conclude that the subjects who produced the causal texts (“Y happens because of X”) understood the water cycle better than those who produced timeline texts (“Y happens after X”), and that language and cultural background had no effect on this. Put differently, some English, Japanese, and Chinese

6

subjects would fail their their Physics test – but I am not sure that we can conclude much more than that from the experiment. “Japanese semiotic vernaculars in ESP multiliteracies projects” is Ferreira’s report of a classroom task performed by his Japanese students, in two consecutive years, to produce a multimodal tourist brochure aimed at an English-speaking audience. He notes, among other things, that Japanese right-left writing and the possibilities for vertical layout of text can have (sometimes: unintended) effects on the visuals in the brochure. Unsworth investigates how stories are affected by their “translation” into a different medium. Here, this classic remediation research assumes some urgency because the target audience consists of children. Unsworth emphatically presents the comparative multimodal analyses – of Stellaluna, Shrek, and The Little Prince and their CD-ROM/filmic counterparts – in terms of fostering a critical “pedagogic practice” (p. 333). He unveils a number of salient differences, both in the language and visuals used, which particularly in Shrek reveal that much of the prickly unconventionality of the original has beeen smoothed out in the bigscreen version. Royce’s programmatic “Multimodal communicative competence in second language contexts” would have made a good opening chapter. He takes his inspiration from the “New London Group” to promote the teaching of multimodal competences by extending Dell Hymes’ four criteria for effective communication (grammatical/formal correctness, ease of production and processing, appropriateness to context, and effects in the real world). For the analysis of graphic elements he draws on a list by Donis Dondis which consists of dot, line, shape, direction, tone, color, texture, scale/proportion, and dimension/motion. He ends the chapter by presenting a number of classroom tasks that can aid multimodal communicative competence.

7

Let me signal and comment on some themes and patterns in the book. First, the positive and promising developments I discern: 1. Most chapters insist that teaching multimodal communicative competence is a necessary component in (secondary) education. Having to teach multimodality forces researchers and educators to develop clear-cut models and approaches, whose usefulness is demonstrated by sample analyses but which are flexible enough to be applicable to new texts. Focusing on multimodal discourses also sharpens pupils’ awareness of the relative strengths and weaknesses of different modalities. At the very least, this work outlines an important task for the contemporary humanities’ curriculum. 2. The laudable use of corpora, though sometimes small ones, surfaces in several chapters. By definition, detecting patterns requires comparing things. Even comparing two things using the same measuring tool is already infinitely more useful for theorizing than an application to a single case-study. Comparing more than two items (dozens, hundreds) is better, of course, but given that qualitative research of multimodal texts can be very time-consuming, this is not always feasible. A bonus is that, given a clear theoretical framework, comparative work can be divided over a group of students who are to earn credits for course work. 3. In the visual modality, spatial dimensions are crucial. Moreover, given that space involves movement it is not surprising that several chapters mention “embodiment” – a central concept in the Cognitive Linguistics approach. Specifically the “source-pathgoal schema” (discussed by Mohan et al.) has overlap with the importance attributed to “vectors” in the SFL framework, and provides provides fruitful angles for analyzing audiovisual modalities (see Johnson, 1987).

8

4. An often-quoted term in the volume is Gibson’s (1979, chapter 8) notion of “affordance” (roughly: that what you can do with a thing or concept). It makes sense to add its opposite, “constraints” (that what you can not do with a thing or concept). My view is that this pair is very helpful in the assessment of the opportunities and impossibilities pertaining to media, modalities, and genres (see Forceville, 2006a). I also see serious problems: 1. Many chapters subscribe to the idea that the visual modality constitutes a “system” comparable to that exemplified in the grammar and vocabulary of the verbal modality. This confidence, typical of old-style semiotics, is not warranted, at least not to the degree voiced in this book. The notion of a visual “grammar” (Kress and Van Leeuwen 1996) needs to be used with extreme caution. The term betrays the extent to which work on multimodality is undertaken by scholars trained as linguists, and entails the risk that claims about the need to acquire visual competence are either exaggerated or misdirected – the former because understanding pictures comes to humans a lot more naturally than understanding language; the latter because the language model may blind researchers to the medium-specific affordances that, by definition, are absent or downplayed in that model. Moreover, while many of the concepts proposed by Kress and Van Leeuwen (1996), such as given-new, centremargin, and vectors, are exciting and potentially fruitful, I am worried that they are simply considered as facts that can be unproblematically used to “prove” things, whereas in reality these concepts are at present no more than hypotheses requiring further critical evaluation as well as empirical testing (see Forceville, 1999). 2. Apart from the fact that “modality” itself is not adequately defined, this book focuses predominantly on the (interrelations between) two modes only: the verbal and the visual. Martin and Stenglin are absolutely right to point out that the two-dimensional

9

page “affords” representing language and (static) pictures at the expense of, for instance, space, sound, music, moving images, and gestures. We run the risk of being blind to the specificities of these other modalities. 3. Even the visual modality gets little attention, compared to the verbal modality. As to the verbal modality: analysis in this vein continues work in the venerable tradition of “stylistics,” associated with pioneering work by Leech and Short (Leech, 1966, 1969; Leech and Short, 1981), and the journal Language and Literature. As to the short shrift paid to the visual modality: to some extent, this is inevitable, since there are few tried-and-tested tools for the analysis of contemporary visual culture (film theory is an exception; see Bordwell and Thompson (2008), while McCloud’s work (1993, 2006) holds promise for comics). But in order to discuss how different modalities interact in multimodal discourse, basic knowledge of how they function on their own is indispensable. We all need to work hard to develop this non-linguistic knowledge; otherwise what is bound to happen – a problem surfacing in many of the chapters of the current book – is that the visual modality is seen as always merely supportive of the verbal. For my own recent attempts to practise what I preach, see Forceville (2005, 2006b), Forceville and Jeulink (2007), Eggertsson and Forceville (forthcoming), and Forceville et al. (in press).

10

References

Bordwell, David, Thompson, Kristin, 2008. Film Art: An Introduction, 8th edn. McGraw-Hill, Boston. Eggertsson, Gunnar Théodor, Forceville, Charles, forthcoming. The HUMAN VICTIM IS ANIMAL metaphor in horror films. In: Forceville, C., Urios-Aparisi, E. (Eds), Multimodal Metaphor, Mouton de Gruyter, Berlin. Forceville, Charles, 1999. Educating the eye? Kress & Van Leeuwen's Reading Images: The Grammar of Visual Design. Language and Literature 8, 163-78. Forceville, Charles, 2005. Visual representations of the Idealized Cognitive Model of anger in the Asterix album La Zizanie. Journal of Pragmatics 37, 69-88. Forceville, Charles, 2005. Addressing an audience: time, place, and genre in Peter Van Straaten’s calendar cartoons. Humor 18, 247-278. Forceville, Charles, 2006a. Non-verbal and multimodal metaphor in a cognitivist framework: agendas for research. In: Kristiansen, G., Achard, M., Dirven, R., Ruiz de Mendoza Ibáñez, F. (Eds), Cognitive Linguistics: Current Applications and Future Perspectives, Mouton de Gruyter, Berlin, pp.379-402. Forceville, Charles, 2006b. The source-path-goal schema in the autobiographical journey documentary: McElwee, Van der Keuken, Cole. The New Review of Film and Television Studies 4, 241-261. Forceville, Charles, Jeulink, Marloes, 2007. The source-path-goal schema in animation film. Paper presented at the International Cognitive Linguistics Conference, Jagiellonian University Krákow, Poland, 15-20 July.

11

Forceville, Charles, Veale, Tony, Feyaerts, Kurt (in press). Balloonics: The visuals of balloons in comics. In: Goggins, J., Hassler-Forest, D. (Eds), Out of the Gutter, McFarland, Jefferson NC. Gibson, James J., 1979. The Ecological Approach to Visual Perception. Houghton Mifflin, Boston. Johnson, Mark, 1987. The Body in the Mind: The Bodily Basis of Meaning, Imagination and Reason. University of Chicago Press, Chicago. Kress, Gunther, Van Leeuwen, Theo, 1996. Reading Images: The Grammar of Visual Design. Routledge, London. Revised edition published in 2006. Leech, Geoffrey N., 1966. English in Advertising: A Linguistic Study of Advertising in Great Britain. Longman, London. Leech, Geoffrey N., 1969. A Linguistic Guide to English Poetry. Longman, London. Leech, Geoffrey N., Short, Michael H., 1981. Style in Fiction: A Linguistic Introduction to English Fictional Prose. Longman, London. McCloud, Scott, 1993. Understanding Comics. Paradox, New York. McCloud, Scott, 2006. Making Comics: Storytelling Secrets of Comics, Manga and Graphic Novels. HarperCollins, New York.

Charles Forceville studied English language and literature at the Vrije Universiteit, Amsterdam, where he subsequently taught in the departments of English, comparative literature, and Word & Image. Since 1999 he has been employed in the department of Media Studies of the Universiteit van Amsterdam, where he is now associate professor and directs the department’s Research Master programme. In 1996, he published Pictorial Metaphor in Advertising (Routledge); the volume Multimodal Metaphor (Mouton de Gruyter), co-edited

12

with Eduardo Urios-Aparisi, is forthcoming. His teaching and research interests focus on the structure and rhetoric of multimodal discourse in various media and genres (documentary, animation, comics, cartoons). Between 1987-2007 Forceville wrote some 200 literary reviews for the Dutch newspaper Trouw. Ch.J. Forceville* Dept. of Media Studies, University of Amsterdam Turfdraagsterpad 9, 1012 XT Amsterdam, The Netherlands

*Tel.: +31 20 5254596; fax: +31 20 5254599 E-mail address: [email protected] URL: http://home.medewerker.uva.nl/c.j.forceville/