Assessing translation competence

© 2013 Actualizaciones en Comunicación Social Centro de Lingüística Aplicada, Santiago de Cuba

JUNE EYCKMANS* PHILIPPE ANCKAERT** WINIBERT SEGERS*** Ghent University* Ghent, Belgium Haut Ecole Francisco Ferrer** Brussels, Belgium Lessius University College*** Antwerp, Belgium [email protected], [email protected], [email protected]

Assessing translation competence 1. Introduction In recent years the field of translation studies has professed the need for more empirical evidence for the quality of translation tests (Waddington 2004; Anckaert et al. 2008). Although educational as well as professional organizations have implemented certification of translation skills on the basis of test administration, the reliability and validity of those tests remain underexplored. It seems that translation assessment around the world is more dependent on codes of practice than on empirical research. Translation tests have a long history of serving as an indicator of language proficiency in schools, universities, and colleges around the world, although some language testers have raised serious objections to this practice (e.g. Klein-Braley 1987). With the current revival of the contrastive approach in Second Language Acquisition (Kuiken 2001; Laufer 2005) one might even say that translation is back in vogue as a measure of foreign language abilities. Ironically, it was remarked by Robert Lado in the sixties that “translation tests that are so common in testing language proficiency skills are not available as tests of the ability to translate” (Lado 1961: 261). To our knowledge, no method has been developed or disseminated to relate performance indicators to the underlying translation competence in a psychometrically controlled way (Anckaert et al. 2008). At least two explanations might account for this lack of psychometrically sound test development when assessing translation ability. First of all, the lack of validity of the translation test as a measure of language proficiency caused a certain loss of popularity of the format during the years of the Communicative Approach to language teaching (Widdowson 1978). As translation tests were not considered to be valid test formats for measuring language proficiency, they were not subjected to the same psychometric scrutiny as other language testing formats (such as the cloze or c-test). A second explanation concerns the epistemological gap that is experienced between protagonists of hard sciences (i.e. biology, chemistry, etc.) versus the humanities (literature, linguistics, translation and interpreting studies, etc.). The preconception that it is impossible to objectify language knowledge or translation quality without surrendering its essence is very tenacious among translator trainers and language teachers whose corporate culture exhibits a marked reticence towards the use of statistics (Anckaert et al. 2008). The fact that the teaching and testing of both translation and interpreting skills has generally been more in the hands of practitioners than of researchers has not helped in this regard either.Thanks to the introduction of psychometrics, numerous studies on the reliability and validity of language tests have been carried out in the domain of language assessment, but the domain of translation and interpreting studies has lagged behind in this respect. 2. Current methods in translation assessment Today’s practice of assessing translations is still largely characterized by a criterion-referenced approach. Both in the educational and the professional world, assessment grids are used in an attempt to make the evaluation more consistent and reliable (Waddington 2001; House 1981; Horton 1998; Al-Qinai 2000; Schmitt 2005). These grids reflect the criteria the evaluator (or the organization) sees as essential for determining the quality of the translation. They traditionally consist of a near-exhaustive taxonomy of different kinds of mistakes and/or bonuses (i.e. grammaticality, text cohesion, word choice, etc.) combined with a relative weight that is attributed to these categories (major or minor mistake). Although the use of assessment grids is motivated by the evaluator’s wish to take the different dimensions of translation competence into account, one could argue that they fall short in adequately reducing the subjectivity of the evaluation, since the identification of dimensions of translation competence in itself is pre-eminently subjective (Anckaert et al. 2008). Aside from the inherent subjective nature of establishing sub-competences of translation ability, there are other factors that threaten the reliability of the test. Let’s start with the evaluator’s challenge of being consistent in her/his assessment when confronted with the task of scoring several tens of translations within a certain time span. Not only will the scoring be susceptible to order effects (contrast effects as well as halo effects, i.e. unintentional or unconscious preconceptions versus students with a weak/strong reputation), it is also difficult to maintain a sound testing practice in which one does not only distinguish the really good from the really bad translations, but where one can also discriminate within the large group of ‘average quality’ translations. Moreover, all scores have to be justifiable, and nowadays students exercise their rights (as they should) on equitable assessment of their performances. Fortunately, researchers from the fields of translation studies (Pacte 2000; Waddington 2001; Conde Ruano 2005) as well as applied linguistics (Anckaert et al. 2008) are now taking up issues such as inter- and intra-rater reliability of translation assessment, and the construct validity of translation tests. Gradually the methodology of

513


educational measurement together with the insights of language testing theory are being transferred to the field of translation and interpreting studies in order to arrive at valid and reliable ways to measure translation competence.It is within this context that we put forward a norm-referenced method for assessing translations, which we have developed with the aim of freeing translation assessment from construct-irrelevant variables that arise in both analytic (i.e. evaluating by means of pre-conceived criteria) and holistic (i.e. evaluating the performance as a whole in a so-called “impressionistic” or “global intuitive” way) scoring. 3. A norm-referenced method for assessing translation competence The norm-referenced approach largely consists of transferring the well-known “item”-concept of traditional language testing theory and practice to the domain of translation studies and translation assessment. This implies a rupture with traditional methods of translation assessment, where the evaluator judges translation quality according to a series of pre-established criteria. A norm-referenced method is independent of subjective a priori judgments about the source text and the translation challenges it may encompass. Instead, the performance of a representative sample of the student population is used in order to identify those text segments that have discriminating power. Every element of the text that contributes to the measurement of differences in translation ability acquires the “item”-status. These items are selected in a pre-test procedure. Afterwards these items function as the sole instances on which to evaluate the translation performance in the actual summative/diagnostic test. As in language testing, the norm-referenced method presupposes a dichotomous approach of the text segments: a translated segment is either acceptable or not (i.e. correct or not). There is no weighing of mistakes and/or bonuses against other alternatives. This does not imply that there is only one appropriate translation for the text segment in question, it merely means that for each translated segment it is agreed between graders which alternatives are acceptable and which are not. Since the method is based on the practice of calibrating segments of a translation, we call it the Calibration of Dichotomous Items-method (CDImethod). The different steps that lead to the calibration of items and tests (pre-testing, establishing discrimination power and estimating the test reliability) allow the construction of standardized tests of translation (across languages). It is clear that this does not constitute a time-saving procedure; therefore the method is only to be promoted for use in summative contexts (high stake situations where decisions have to be made). 4. Conclusion The norm-referenced CDI-method holds the promise that it is a stable and evaluator-independent measurement that bridges the gap between language testing theory and the specific epistemological characteristics of translation studies. In our paper presentation we will report an experiment in which three methods for assessing translation competence are compared: the holistic method, the analytic method (both criterion-referenced approaches), and the CDI-method (norm-referenced approach). The central research question centers on the reliability and the validity of these methods with reference to each other and homes in on the quintessential issue of text independent measurement of translation competence.The results of the experiment indicate that translation performance indicators can be related to the underlying translation competence in a psychometrically controlled way. However, collaboration between translation teachers and test developers is indispensable if we want to rise to the methodological challenge of achieving equivalent standards across languages.The dissemination of norm-referenced methodsties in with the European call for test standardisation in the sense that it allows a reliable and valid certification of translation competence across Europe. It also permits an exploration of text robustness so that tests can be validated for different language combinations. References Al-Qinai, Jamal. 2000. Translation Quality Assessment. Strategies, Parameters and Procedures. Meta45(3): 497519. Anckaert, Philippe, Eyckmans, June and Segers, Winibert. 2008. Pour une évaluation normative de la compétence de traduction. ITL International Journal of AppliedLinguistics 155: 53-76. Conde Ruano, Tomás. 2005. No me parece mal. Comportamiento y resultados de estudiantes al evaluar traducciones. University of Granade: Unpublished doctoral dissertation. Horton, David. 1998. Translation assessment: Notes on the interlingual transfer of an advertising text. IRAL36(2): 95-119. House, Juliane. 1981. A Model for Translation Quality Assessment. Tübingen: Gunter Narr. Klein-Braley, Christine. 1987. Fossil at large: Translation as a language testing procedure. In Taking Their Measure: The Validity and Validation of Language Tests, Grotjahn, Rüdiger, Klein-Braley, Christine and Stevenson Douglas K. (eds.). 111-132. Bochum: Brockmeyer. Kuiken, Folkert. 2001. Contrastief en taakgericht: een contrast. In Perspectieven voor de internationale neerlandistiek in de 21ste eeuw: handelingen Veertiende Colloquium Neerlandicum, Katholieke Universiteit Leuven, 27 August-2 September 2000 / IVN, Internationale Vereniging voor Neerlandistiek, Elshout, Gerard et al. (eds.). 353-362. Münster: Nodus-Publ. Lado, Robert. 1961. Language Testing. The Construction and Use of Foreign Language Tests. A Teacher’s Book. London: Longmans. Laufer, Batia. 2005. Focus on form in second language vocabulary learning. In EUROSLA Yearbook Volume 5, Foster-Cohen Susan H., Garcia-Mayo María and Cenoz, Jasone (eds.). 223-250. Amsterdam/Philadelphia: John Benjamins Publishing Company.

514


Pacte. 2000. Acquiring translation competence: hypotheses and methodological problems in a research project. In Investigating Translation, Beeby, Allison.,Ensinger, Doris, Presas, Marisa (eds.). 99-106. Amsterdam: John Benjamins. Schmitt, Peter A. 2005. Qualitätsbeurteilung von Fachübersetzungen in der Übersetzerausbildung. Problemeund Methoden, paper presented at Vertaaldagen Hoger Instituut voor Vertalers en Tolken, 16-17 March 2005, http://www.hivt.be/onderzoek/vertaaldagen2005_verslag.htm. Waddington, Christopher. 2001. Different methods of evaluating student translations: the question of validity. Meta46(2): 331-325. Waddington, Christopher. 2004. Should student translations be assessed holistically or through error analysis? LebendeSprachen49(1): 28-35. Widdowson, Henry G. 1978. Teaching Language as Communication. Oxford: Oxford University Press.

515