Linguistic simplification of mathematics items: effects

Eur J Psychol Educ DOI 10.1007/s10212-014-0233-6

Linguistic simplification of mathematics items: effects for language minority students in Germany Nicole Haag & Birgit Heppt & Alexander Roppelt & Petra Stanat

Received: 15 July 2014 / Accepted: 20 October 2014 # Instituto Superior de Psicologia Aplicada, Lisboa, Portugal and Springer Science+Business Media Dordrecht 2014

Abstract In large-scale assessment studies, language minority students typically obtain lower test scores in mathematics than native speakers. Although this performance difference was related to the linguistic complexity of test items in some studies, other studies did not find linguistically demanding math items to be disproportionally more difficult for language minority students than for native speakers. Furthermore, previous studies investigating the effectiveness of linguistic simplification of test items have yielded inconsistent results. We tested the impact of linguistic simplifications of mathematics test items in Germany with a specific focus on the role of academic language features included in the items. Specifically, we carried out a large-scale linguistic simplification study to test whether the performance gap between language minority students and German monolinguals is smaller when students are assessed with linguistically simplified items. We additionally determined whether students’ socioeconomic status (SES) or their language proficiency moderate the effect of linguistic simplification. The study uses data from 17,738 fourth graders, 17 % of whom sometimes or always spoke a language other than German at home. Although differences between language minority students and German monolinguals in mathematics achievement were related to differences in their language proficiency and SES, we found no significant main effects of linguistic simplification. Differential effects for language minority students emerged, however, when we took the moderator effects of SES and language N. Haag (*) : A. Roppelt : P. Stanat Institute for Educational Quality Improvement (IQB), Humboldt-Universität zu Berlin, Unter den Linden 6, 10099 Berlin, Germany e-mail: [email protected] URL: http://www.iqb.hu-berlin.de A. Roppelt e-mail: [email protected] URL: http://www.iqb.hu-berlin.de P. Stanat e-mail: [email protected] URL: http://www.iqb.hu-berlin.de B. Heppt Humboldt-Universität zu Berlin, Unter den Linden 6, 10099 Berlin, Germany e-mail: [email protected]

N. Haag et al.

proficiency into account, indicating that some language minority students may profit from linguistic simplification in elementary school. Keywords Educational measurement . Second language learners . Academic language . Accommodation . Linguistic simplification

Introduction Large-scale assessment studies have repeatedly documented performance disadvantages of second-language learners in elementary and secondary schools (Haag et al. 2012; Martin et al. 2008; Mullis et al. 2008; Schwippert et al. 2012; Stanat and Christensen 2006; Tarelli et al. 2012). It is assumed that these differences may, in part, be due to test items with high linguistic demands. These items may selectively disadvantage second-language learners in the testing situation by measuring construct-irrelevant linguistic competencies (Abedi 2002). Several studies found that linguistically complex test items tend to be disproportionally more difficult for second-language learners even when mean performance differences are controlled (Abedi et al. 2008; Haag et al. 2013; Martiniello 2008, 2009; Wolf and Leon 2009), although the effect tends to be small. Test accommodations aim, among other things, at decreasing these disproportional difficulties by removing construct-irrelevant task components, such as the unnecessary use of academic language in mathematics test items (Sireci et al. 2005). However, most studies on linguistic simplification as an accommodation for second-language learners have been conducted in middle school, even though elementary school children are already confronted with the academic language register. Therefore, the present study tests whether linguistic simplification of test items is a valid accommodation in elementary schools.

Theoretical and emprical background Academic language Most previous studies use the concept of academic language to describe the linguistic features of test items. Academic language is defined as the language that is spoken in the classroom to impart and acquire knowledge (Chamot and O'Malley 1994) and contains a larger number of specific lexical and syntactical features than everyday language. Two crucial lexical features are general and specialized academic vocabulary (Bailey 2000/2005; Bailey et al. 2007; Butler et al. 2004). While general academic vocabulary (e.g., report, synthesize) is used across disciplines, specialized academic vocabulary (e.g., denominator, multiplication) is associated with a specific school subject. Additional lexical features include long words with three or more syllables and clause connectors. In terms of grammar, academic language is more likely than everyday language to entail long and syntactically complex sentences, passive voice constructions, and larger numbers of long noun phrases and prepositional phrases. For second-language learners, gaining proficiency in academic language is expected to be more challenging than gaining proficiency in everyday language (Cummins 1979, 2000). This assumption is based on the notion that—except for specialized academic vocabulary—the various features of academic language are only rarely explained in classroom discourse and, therefore, have to be acquired outside of school (Komor and Reich 2008; Schleppegrell 2012). Given the expected disadvantages of second-language learners in linguistically complex test items and their assumed difficulties with academic language more generally, it has been

Linguistic simplification of mathematics items

proposed to apply linguistic simplification to test items as an accommodation for secondlanguage learners (e.g., Abedi et al. 2004). Linguistic simplification as a test accommodation In general, test accommodations are alterations to testing procedures designed to provide support for students based on their particular needs without changing the construct the test intends to measure. Possible alterations can pertain to the presentation of items to the students (e.g., using a test that has been translated into the student’s native language), the way students respond to the items (e.g., dictating their responses to a scribe), or an increase of testing time (American Educational Research Association, American Psychological Association, and National Council on Measurement in Education 1999; Koenig and Bachman 2004). A good accommodation should result in an interaction effect: students who need the accommodation should benefit from it whereas students who do not need the accommodation should not (or at least not to the same amount), thus ensuring the validity of the test for all students (e.g., Fuchs et al. 2000; Koenig and Bachman 2004; Sireci et al. 2005). Accommodations for students with disabilities are widely used (for a review, see Sireci et al. 2005), however, at least in the US, accommodations for English language learners (ELLs) are also common (Abedi et al. 2004; Rivera et al. 2000). Potential moderators of linguistic simplification A number of factors have been shown to influence second-language learners’ school success (for a review, see Genesee and Lindholm-Leary 2012). Two of these factors, namely their families’ socioeconomic status (SES) and their language proficiency in the language of the test, have also been discussed as moderators of the effects linguistic complexity of test items may have on students’ performance in content assessments (Cummins 1979, 2000; Townsend et al. 2012). Large-scale assessment studies revealed considerable social disparities in academic achievement, indicating that students with low SES perform significantly below students with average or high SES in mathematics, science, reading, and listening comprehension (Ehmke and Jude 2010; Martin et al. 2008; Mullis et al. 2008; Richter et al. 2012; Stubbe et al. 2012). These differences emerge early in students’ educational careers. This is at least partly due to low-SES children’s restricted exposure to literacy activities in their homes which seems to hamper the development of the language skills required for school success (cf., Hart and Risely 1995; Scheele et al. 2012; Snow 1983). In addition, second-language learners from families with low SES may be particularly disadvantaged in their early reading and mathematics achievement trajectories, due to a limited amount and quality of input in the language of instruction (Kieffer 2008; Roberts and Bryant 2011). As immigrant students often grow up in socially disadvantaged families (Genesee and Lindholm-Leary 2012; Segeritz et al. 2010), it is unclear whether their lower academic achievement is due to their language background or to social disparities. Therefore, the effectiveness of linguistic simplification of test items might not only depend on students’ language background but also on their SES. However, most previous studies on the effects of academic language on test performance have not systematically taken students’ SES into account. The extent to which linguistic simplification affects students’ test performance in content assessments should also be moderated by their level of language proficiency (Aguirre-Muñoz 2000; Ardasheva et al. 2012; Sato et al. 2010). More specifically, language proficiency should affect mathematical performance only until a certain level of language proficiency is reached. After this, the relationship between language proficiency and mathematical performance weakens or disappears (cf., Cummins 1979, 2000). Technically speaking, this suggests a

N. Haag et al.

nonlinear relationship—namely a combination of a linear and a negative quadratic effect— between language proficiency and mathematical performance in the probability of solving an item. Thus, the effect of linguistic simplification should be more pronounced for students with intermediate language proficiency than for students with either low or high language proficiency. Research on linguistic simplification of mathematics tests Several studies have investigated the effectiveness of linguistic simplifications for ELLs in mathematics. The focus of this research was on middle school, i.e., on grades 7 and 8. While some of the studies did not reveal any overall effects of linguistic accommodations at all (Abedi et al. 2006; Johnson and Monroe 2004), some found that, under certain conditions, both ELLs and non-ELLs performed better on linguistically simplified items (Abedi et al. 2006; Hofstetter 2003). Another set of studies found a differential improvement of ELLs on the simplified items (Abedi et al. 2001; Abedi and Lord 2001; Sato et al. 2010), indicating that ELLs profited more from the accommodations than non-ELLs. One of these studies only found differential effects for ELLs under the 1-PL item response theory model, not when analyzing the raw scores. This indicates that using IRT to obtain person ability estimates may be a more sensitive approach to detecting possible effects of linguistic simplification than comparing the mean number of correctly answered items between simplified and original test booklets. Three meta-analyses synthesized the studies on accommodations for ELLs in different school subjects with different study samples. While most primary studies were on accommodations for science and mathematics testing, a smaller number of studies also dealt with accommodations for reading or history tests. The first meta-analysis (Kieffer et al. 2012, which is an updated version of Kieffer et al. 2009) revealed a significant mean effect of linguistic simplification (d=0.14) based on 24 effect sizes. The effects were moderated by grade level and subject with larger effect sizes in grades 7 and 8 compared to grades 4–6 and larger effect sizes in history and mathematics than in science. These differences are driven by particularly large effect sizes of two primary studies in middle school history (Aguirre-Muñoz 2000) and mathematics (Sato et al. 2010) tests and thus should be interpreted with caution. This meta-analysis did not explore whether students with varying levels of English language proficiency or SES differentially profit from linguistic simplification. Another meta-analysis by Li and Suen (2012) systematically explored student variables—namely ethnicity and English language proficiency—as potential moderators of accommodation effects for ELLs based on 85 effect sizes from 19 studies. In this metaanalysis, all types of accommodations were clearly more effective for ELLs with low English proficiency than for ELLs with high English proficiency. In a third meta-analysis, PennockRoman and Rivera (2011) aimed at determining the influence of students’ language proficiency on the effectiveness of linguistic simplification as a test accommodation based on 50 effect sizes from 14 studies. Their results suggested that the effectiveness of linguistic simplification as an accommodation depends on ELLs’ skills in the test language, with the largest effects occurring for students with an intermediate level of language proficiency. These results were, however, based on only three effect sizes that all came from the same study on accommodations in a history explanation task (Aguirre-Muñoz 2000). Therefore, it is unclear whether the finding is robust and can be generalized to other assessment domains and grade levels. To sum up, research on the effectiveness of linguistic simplification as a test accommodation is still inconclusive. While linguistic simplification does not seem to yield consistent

Linguistic simplification of mathematics items

effects on second-language learners’ scores overall, there is some evidence that the effects of linguistic simplification are highest for students with intermediate language proficiency. It is also still unclear how students’ SES affects the impact of linguistic simplification accommodations. Based on the considerable disparities associated with SES in both academic achievement and academic language development and the fact that second-language learners often grow up in families with low SES, it seems plausible to assume that the effects of linguistic simplification are moderated by students’ SES. However, very few of the previous studies have systematically addressed this issue. In addition, the method used to calculate the person ability estimates seems to be important for detecting possible effects of linguistic simplification. However, most of the available studies only used differences between the percent-correct scores for accommodated and unaccommodated students as their dependent variable. Finally, even though considerable performance differences exist between second-language learners and native speakers in elementary school, there seem to be no published studies on the effectiveness of linguistic simplification accommodations in that age group. Already in elementary school, texts and test items may contain a considerable amount of academic language (Schleppegrell 2001) and high academic language demands of elementary school mathematics test items are related to disproportionally higher item difficulties for second-language learners than for native speakers, both in English and in German (Haag et al. 2013; Martiniello 2009). Reducing constructirrelevant linguistic complexity of test items should therefore be a suitable accommodation for second-language learners in elementary school.

Research question and hypotheses The purpose of the present study is twofold: First, it tests the effects of linguistic simplification of mathematics test items on the performance of German secondlanguage learners in a mathematics achievement test in elementary school. Second, we test the extent to which the effectiveness of the accommodation depends on students’ SES and language proficiency. Based on the differential boost and interaction hypotheses (Fuchs et al. 2000; Koenig and Bachman 2004; Sireci et al. 2005), we expect the linguistically simplified items to be easier than the original items and we predict this effect to be more pronounced for second-language learners than for native speakers. Moreover, we expect both language proficiency and SES to moderate the effect of linguistic simplification. For the effect of language proficiency, we specifically expect second-language learners with intermediate language proficiency to profit most from the accommodation.

Methods Design In the present study, we employed a 2×2 experimental design with the within-person factor item status (original, simplified) and the between-person factor home language (German monolingual [GM], language minority [LM]). Additionally, we included students’ SES as well as linear and quadratic effects of reading performance—as an indicator for German language proficiency—as continuous moderator variables.

N. Haag et al.

The analyses are based on a subsample of the German National Assessment Study in elementary schools (Stanat et al. 2012). This nationally representative large-scale assessment study is part of the German educational monitoring system and is conducted every 5 years to compare the 16 German states (Länder) in their attainment of the national educational standards in mathematics and German (reading, listening comprehension, and orthography) in grade 4. The test items were developed in close cooperation with content experts, educational scientists, and psychometricians. The items are of both multiple choice and constructed response format and have been piloted and normed in nationally representative samples (see Granzer et al. 2009). The assessment took place on two consecutive days, with students either taking the mathematics test on the first day and the German test on the second day or the other way around. Additionally, students and their parents completed questionnaires including questions on their language background, SES, and parental education. Linguistic simplification procedure We constructed linguistically simplified versions of 23 of the linguistically most challenging mathematics test items used in the German National Assessment Study. We had to focus on three out of the five content strands due to the restricted space available for linguistically simplified items in the assessment design. Specifically, items pertaining to one of the three content strands with the highest number of overall words (Space and Shape, Patterns and Structures, and Probability) were modified. Simplifications were based on the concept of academic language (e.g., Bailey 2007) and only items containing a substantial number of academic language features were included in the simplification process. We simplified both lexical and grammatical features that are assumed to contribute to construct-irrelevant academic language complexity (Abedi and Lord 2001; Haag et al. 2013; Kopriva 2008). We left the original mathematical vocabulary untouched and did not include any additional visuals or graphs in the modified items, in order to leave the construct-relevant aspects of the items intact. The predominant lexical characteristics in our items were long words with three and more syllables, including compounds and general academic vocabulary. These were replaced by shorter and more familiar vocabulary. In terms of grammar, we reduced average sentence length by avoiding subordinate clauses and infinitive constructions. Passive voice and other impersonal constructions (e.g., sth. is to be seen) occurred in seven items and were also eliminated. Five items contained long noun phrases of four or five words which were shortened. Figure 1 shows an example of a simplified item and its original counterpart. As a manipulation check, eight trained raters coded the academic language features of the items (see Heppt et al. 2014, for details of the rating procedure). For each item, two out of the eight raters identified the frequency of each academic language feature present in each item. Overall, interrater reliability determined by a one-way random effects model of intraclass correlation coefficients was high, ranging from 0.73 for math-relevant vocabulary words to 0.96 for measurement-related vocabulary words. Table 1 shows the means and standard deviations of the ratings for the academic language features targeted by the linguistic simplification as well as for the mathematics-specific academic language features for original and simplified items.

Linguistic simplification of mathematics items

Item modification status

German

English translation

Original

In einer zylinderförmigen Regentonne stehen 50 cm Wasser. Nun setzt Dauerregen ein. Nach 30 min befinden sich 65 cm Wasser in der Tonne. Wie voll ist die Tonne nach 50 min?

In a cylindrical barrel the height of the water is 50 cm. Then continuous rainfall sets in. After 30 min the barrel contains 65 cm of water. How full is the water barrel after 50 min?

Simplified

In einer zylinderförmigen Regentonne steht das Wasser 50 cm hoch. Jetzt fängt es zu regnen an. Nach 30 min steht das Wasser in der Tonne 65 cm hoch. Wie hoch steht das Wasser in der Tonne nach 50 min?

The height of the water in a cylindrical barrel is 50 cm. Now it starts to rain. After 30 min the water is 65 cm high. How high is the water in the barrel after 50 min?

Fig. 1 Example of an original item and the corresponding simplified item. The original items used in the German National Assessment Study in elementary schools on which the present study is based (Stanat et al. 2012) are not released for publication. Therefore, the example item is a simplified version of a previously released item from the same item pool. It is functionally equivalent to the assessment items

According to paired t tests, our linguistic simplification procedure significantly reduced the number of general academic vocabulary words, words with three or more syllables, passive voice verb phrases, complex sentences, mean sentence length in words, and mean noun phrase length in words while keeping the number of math-relevant vocabulary words and measurement-related vocabulary words constant.

Measures Student characteristics In line with the common practice in Germany, we used parents’ ratings of their home language as an indicator of students’ language minority status. The parents indicated the frequency with which they speak German at home with their children (always, sometimes, or never). In cases with missing parent ratings, the corresponding student rating was used. As the frequency of the never-category was very low (0.65 %), we collapsed the categories sometimes and never, resulting in a dichotomous variable for LM status (always speaks German at home, sometimes or always speaks a language other than German at home). For the assessment of the families’ SES, we asked the parents to report their occupation. The responses were coded according to the International Standard Classification of Occupation (ISCO-08; International Labour Office 2012) and subsequently mapped on the International Socio-Economic Index (ISEI; Ganzeboom et al. 1992). The ISEI is an indicator of a person’s occupation taking into account the salary as well as the typically required educational qualification, and it ranges from 16 to 90 with low values representing a low SES. In the present analyses, the family’s SES was represented by the highest ISEI (HISEI) of the parents.

N. Haag et al. Table 1 Means and standard deviations of academic language ratings Academic language feature

Number of words

Original items

Simplified items

M

SD

M

SD

toriginal–simplified

32.96

17.76

31.57

16.64

Number of general academic vocabulary words

1.70

1.66

0.72

1.27

1.80 5.94**

Number of words with three or more syllables

2.78

1.91

1.63

1.32

4.97**

Number of passive voice verb phrases

0.59

0.83

0.02

0.10

3.27**

Number of compounds

0.72

0.69

0.59

0.62

1.06

Number of complex sentences

0.87

0.89

0.24

0.75

5.14**

Mean sentence length in words Mean noun phrase length in words

9.13 2.22

3.97 0.53

6.96 2.01

1.77 0.50

3.21** 3.11**

Number of math-relevant vocabulary words

1.28

1.27

1.39

1.64

−0.89

Number of measurement-related vocabulary words

0.30

0.47

0.37

0.53

−1.37

Noriginal =23; Nsimplified =23. t values result from paired t tests for original and simplified items **p