GEORGE MASON UNIVERSITY SCHOOL OF PUBLIC POLICY RESEARCH PAPER NO. 2010-25
IMPACT OF FAMILY VS. SCHOOL FACTORS ON CROSS-NATIONAL DISPARITIES IN ACADEMIC ACHIEVEMENT: EVIDENCE FROM THE 2006 PISA SURVEY Sonia Sousa George Mason University - School of Public Policy David J. Armor George Mason University - School of Public Policy
October 5, 2010
1
Impact of family vs. school factors on cross-national disparities in academic achievement: Evidence from the 2006 PISA survey
Sonia Sousa PhD student, School of Public Policy, George Mason University 3401 Fairfax Drive, Arlington, VA 22201
[email protected] David J. Armor University Professor, School of Public Policy, George Mason University 3401 Fairfax Drive, Arlington, VA 22201
[email protected]
Abstract This paper investigates whether international differences in math and science achievement, and specifically the lower US scores, can be explained by school programmatic, institutional, and resource differences after controlling for family SES factors. Using 2006 PISA student-level data for the 10 largest developed OECD countries, the results show that, while family SES has a strong impact on students’ achievement, it does not explain the US achievement gap with other developed OECD countries. In contrast, a substantial number of school variables not only have significant impacts on math and science achievement, but they contribute more to these gaps than SES differences. Of particular importance for policy purposes are the lower amounts of time devoted to studying math and science in the US, as well as student-centered pedagogical techniques which are emphasized in the US but not in such high-scoring countries such as Korea.
Electronic copy available at: http://ssrn.com/abstract=1688131
2
Introduction Previous studies have established wide disparities in cross-national comparisons of academic achievement, with the US scoring much lower than most other developed countries (Baldi et al. 2007, Juvonen et al. 2004, Lemke et al. 2001, Lemke et al. 2004, Provasnik et al. 2009). Other studies have demonstrated that family socioeconomic status (SES) and school resources have significant impacts on students’ academic achievement (Christenson et al. 1992, Henderson and Mapp 2002, Sirin 2005, Willms 2010). To date, however, few studies have investigated the extent to which SES and school factors might explain these cross-national achievement disparities, and particularly the relatively low performance of the US. This paper seeks to fill this gap by assessing the extent to which family SES and school factors help explain international differences in students’ achievement in math and science. In so doing, this research seeks to shed light on several major public policy questions. First, to what extent can cross-national differences in achievement be explained by a combination of family SES and school factors differences? Second, can the lower performance of the US be explained by school factors differences between it and countries with higher scores? Third, in an international context, which set of factors are more important in explaining achievement disparities– family SES or school characteristics? The international comparison focuses on the 10 largest developed countries that are members of the Organization for Economic Cooperation and Development (OECD) – United States, Japan, Germany, France, United Kingdom, Italy, Korea, Spain, Canada, and Australia.1 The research is based on a sample of 55,888 fifteen-year-old students from the OECD’s 2006 Program for International Student Assessment (PISA). The 2006 PISA datasets provide 1
This group of countries was selected in order to include only countries comparable with the US in terms of both size and level of economic development with fairly complete data from the same source – the 2006 PISA survey.
Electronic copy available at: http://ssrn.com/abstract=1688131
3
achievement test scores in math and science, as well as numerous measures of family background and school factors from student and school context questionnaires.
Background Until recently, evidence on the impact of family SES and school factors on students’ achievement derived predominantly from within-country studies. Following the seminal studies of Coleman et al. (1966) using data for the US and Peaker (1971) for the UK, numerous withincountry studies have explored the impact of family SES on students’ achievement. Several meta-analyses and research syntheses have concluded that family SES has a strong and consistent impact on students’ academic achievement2 (Christenson et al. 1992, Henderson 1987, Henderson and Berla 1994, Henderson and Mapp 2002, Sirin 2005). Other within-country studies have emphasized the importance of school factors after taking account of family SES. We identify four general conditions that enhance the effects of school factors on achievement: (1) More accurate outcome measures that better reflect schools’ curricula (Brookover et al. 1978, Madaus et al. 1979); (2) More detailed measures of programmatic factors such as teaching practices, teacher quality, curriculum, length of school year, ability grouping and tracking, interactions among students and teachers, and how time and resources are used (Alexander, 1982, Alexander et al. 2007, Dreeben and Gamoran 1986, Gage and Needels 1989, Ho Sui-Chu and Willms 1996, Mortimore et al. 1988, Pallas 1988, Rutter 1983, Sammons et al. 1995, Scheerens 1992, Slavin 1990, 1994); (3) Research designs that take into account students’ learning capacity at different stages (Alexander et al. 1981, Bryk and Raudenbush 1988); and 2
They also show that there is general consensus that family SES is defined as parental income, education, and occupation and that using all three is better than using only one. There is less agreement, though, on which measures should be used to assess each of these three components and whether it is preferable to use a composite measure or single indicators for each component (Schulz 2005).
4
(4) more sophisticated statistical techniques that take into account the hierarchical structure of educational data (Lee and Bryk 1989, Aitkin and Longford 1986, Raudenbush and Willms 1995, Raudenbush and Bryk 1986). In short, there is compelling empirical evidence that within a country, particularly within the US, both family SES and school factors have significant effects on students’ academic achievement. Nevertheless, a long-lasting debate persists in the literature regarding the relative importance of family and school factors on students’ achievement (Jenks 1972, NonoyamaTarumi and Willms 2010, Willms 2010). Cross-national studies on SES versus school effects are scarce largely because of the limited availability of internationally comparable data. More recently, the advent of large cross-country achievement studies, particularly the Trends in International Mathematics and Science Study (TIMSS) in the 1990s3 and the OECD’s PISA study in the 2000s4 have been providing data on these issues. To this point, however, cross-national studies have focused more on achievement comparisons rather than family and school effects on students’ achievement.5 An early study of 24 high- and low-income countries using 1970s achievement data finds that family SES effects are weaker than school factors in low-income countries but family SES effects are stronger than school factors in high-income countries (Heyneman and Loxley, 1983). This study has the limitation of pooling data from 6 different sources raising issues of data comparability across countries, particularly in the way family SES is measured (Xia 2009). More recently, using 1994-95 TIMSS data Baker et al. (2002) found that family SES effects are stronger than school resources effects regardless of a country’s economic development, 3
The International Association for the Evaluation of Education (IEA) has collected TIMSS data in 1994-95, 199899, 2002-03, and 2006-07. The next cycle will be conducted in 2010/11. 4 To date, PISA data has been collected in 2000, 2003, 2006, and 2009 (full database not available yet). 5 This review does not include studies using international comparable data to perform two-country comparisons or regional comparisons either within a country or between countries.
5
leading the authors to conclude that the relative effect of family SES and school resources on achievement across countries no longer depends on national income levels. Nonoyama (2005) concurs using data from the 2000 and 2003 PISA studies and a more complete array of family SES measures than Baker et al. (2002). This finding is further corroborated by the recent study of Nonoyama-Tarumi and Willms (2010) using 2000 PISA data. One limitation in the Heyneman and Loxley (1983), Baker et al. (2002), Nonoyama (2005), and Nonoyama-Tarumi and Willms (2010) studies is that they restrict school factors to a relatively limited number of school resources. School variables such as curriculum, teacher quality, and institutional factors are not included. Woessmann (2004) using 1994-95 TIMMS data also concluded that SES factors have strong effects on math and science achievement in the US and 17 Western European countries, although the number of family background measures is more limited than those available in the PISA data. In another study Baker et al. (2001) used the 1994-95 TIMSS data to show that outside-school instruction (or private tutoring) which he called “shadow education” did not appear to influence math achievement. Woessmann using 1994-95 TIMMS data (2003), Fuchs and Woessmann using PISA 2000 data (2007), and Woessmann, et al. using 2003 Pisa results (2009) examined the impact of school institutional factors on student achievement. The three studies found that both family background and school institutional factors have strong effects on student achievement, whereas the effects of school resources are mixed. In addition, Fuchs and Woessmann (2007) found that family background, home incentives, student characteristics, school resources, teachers’ quality, and school institutional factors account for more than 85% of the variation in test scores at the national level in all three subjects and between 26% (science) and 32% (reading) of the variation at the student level.
6
The study most similar to our analysis is one by Xia (2009), who uses the PISA 2006 data to compare the impact of family factors on math and science achievement in the US versus 20 other developed countries.6 The results suggest that family process factors (e.g., learning structure, learning resource availability, and parental involvement) play an important role in explaining the achievement gaps between the US and the other 20 developed countries, after controlling for family SES, school inputs, student characteristics, and other country-specific factors.7 He also finds that after taking student, family, and school factors into account, the US still ranks 18th in math and 17th in science.8 This paper attempts to extend previous research in two ways. First, we analyze the relative importance of family SES and school factors among comparable countries in terms of both size and level of economic development. The major concerns here are (1) to investigate the extent to which the performance gap between the US and other large developed countries can be explained by a combination of family SES and school factors and (2) which set of factors are more important in explaining such achievement gaps. Second, this research uses more comprehensive measures of both family background and school factors than many prior cross-country studies by exploiting the richness of the internationally comparable data offered by the 2006 PISA student and school context questionnaires. In particular, it uses a broader array of school factors than Xia (2009) by including public vs. private school, hours of regular math and science classes, teacher quality, an extensive range of school program factors, and several school institutional factors, e.g., ability grouping, academic selectivity and school accountability (see Table 3). 6
15 Western European countries and 5 East Asia countries and jurisdictions. The present analysis uses learning resources but not one learning structure item (time spent on homework) and not parental involvement (number of out-of-school lessons in math & science). 8 For school inputs Xia included class size, resource shortage, public vs. private, and hours of regular math and science classes, but he did not examine the effect of school programmatic variables available from the student questionnaire (see Table 2), the effect of school institutional factors other than public vs. private and the effect of teacher education (see Table 3). 7
7
Data and methods This research uses data from the 2006 PISA study, a cross-national survey sponsored by the OECD in which 57 countries participated, the 30 OECD country members and 27 partner countries. In each country, a nationally representative two-stage stratified sample with a minimum of around 4400 students was selected. In the first stage, schools were sampled proportionally to size. In the second stage, around 35 fifteen-year-old students were randomly selected within each sampled school.9 In some countries some regions were oversampled and larger sample sizes were obtained (OECD 2009). This study selects a subset of 10 countries, the largest developed country members of the OECD – United States, Japan, Germany, France, United Kingdom, Italy, Korea, Spain, Canada, and Australia. In total, 117,691 students from these 10 countries participated in PISA 2006. Given that five of them are over-sampled by region (UK, Italy, Spain, Canada, and Australia) this research uses a subsample of 55,888 students that was obtained by randomly selecting 25% of the UK and Australian students and 50% of the Italian, Spanish, and Canadian students, drawn by school and stratified by region. Throughout the analysis sampling weights are used to account for stratification, oversampling of certain subgroups or regions, and non-response adjustments. The PISA study assesses the knowledge and life skills of 15-year-old students in three subject areas: reading, mathematics, and science. Each participating student takes a two-hour pencil-and-paper test designed to assess students’ ability to apply their knowledge and skills in mathematics, science, and reading to real-life problems, rather than how well they master specific school curricula. In addition to students’ test scores in these three subjects, the PISA survey also collects a range of family, student, and school information using context 9
In schools with less than 35 fifteen-year-old students all students were sampled. In this way, the sampling procedure did not exclude small schools and ensured the representativeness of rural areas in the sample.
8
questionnaires filled out by students and school principals. The student context questionnaire includes questions on student characteristics, family background, home environment, study habits, and school/classroom climate. The school questionnaire completed by principals or head administrators collects information on the demographics of school, school staffing, the school environment, human and material school resources, and school governance (OECD 2007). The Achievement Variables Our two dependent variables are achievement in mathematics and achievement in science.10 Both math and science achievement scores are standardized in a way that the mean score is 500 and the standard deviation is 100 (OECD 2009). PISA estimates student test scores as plausible values with each student having five plausible values for each subject area.11 In this research the sample mean of the five plausible score values is used as a measure of the student’s achievement in math and science. Family Socioeconomic Status (SES) variables Family SES is assessed by six variables measuring parental occupation, parental education, and family’s economic means, all obtained from the student context questionnaire. Parental occupation corresponds to the higher occupation status of mother and father as measured by the International Socio-Economic Index of occupational status (ISEI) developed by Granzeboom et al. (1992).
10
The PISA survey measures students’ achievement in reading as well, but reading is not used here because it is not available for the US due to printing errors in the test booklets (Sean 2007). 11 Since each student completes only a subset of the achievement questions, test scores area estimated as plausible values using Item Response Theory (IRT). IRT identifies patterns of correct, incorrect, and omitted responses in the subset of questions completed and uses statistical models to predict the probabilities of a student answering correctly to the non-completed set of questions as a function of the student’s proficiency in the completed questions. Therefore, plausible values represent the distribution of potential scores for all students in the population with similar characteristics and identical patterns of item response (OECD 2009).
9
Parental education corresponds to the highest number of years of schooling of the father or the mother. In the PISA Survey students are asked to identify their fathers’ and mothers’ highest level of education on the basis of national qualifications. PISA study codes these qualifications in accordance with the International Standard Classification of Education (ISCED). Then, the highest level of educational attainment of the father or the mother is converted into years of schooling. The conversion coefficients used can be found in OECD (2007, p. 336). The student questionnaire does not have a direct measure of parental income or parental wealth. Although a question about parental income is included in a parent questionnaire, this data is available only in a limited number of countries – just 3 out of the 10 countries used here. However, students are asked if they have access to a set of relevant household items, which taken together can be used as a proxy for students’ family economic means.12 In this research 4 variables are used to assess family’s economic means: (1) index of home possessions, (2) number of books at home, (3) number of cars at home, and (4) number of computers at home. The index of home possessions is a summary index of 14 household items.13 School variables This research uses 29 school variables, 8 obtained from the student questionnaire and 21 from the school questionnaire, representing three broad types of school factors: programmatic factors, institutional factors, and school resources including teacher-related factors.
12
Home resources are not used as commonly as parental income, parental education, and parental occupation, to measure family SES. However, researchers have emphasized the significance of various home resources as indicators of family’s economic means (Coleman 1988, Duncan & Brooks-Gunn 1997). Additionally, some researchers argue that family wealth, as measured by home assets, is a better measure of family’s economic means than income (Bradley and Corwyn 2002, Filmer and Pritchett 1999). 13 The household items are: a desk to use for studying, a room of their own, a quite place to study, a computer they can use for school, educational software, a link to the internet, their own calculator, classic literature, books of poetry, works of art, textbooks, dictionary, dishwasher, and DVD player or VCR.
10
School programmatic factors are measured in the student questionnaire and include: (1) number of hours per week spent learning math and science; (2) number of specific sciencerelated topics taught at school; (3) number of science-related subjects in which school provides skills; (4) amount of outside-school-hours tutoring received;14 (5) number of mandatory and voluntary courses in science; and (5) variety of pedagogical techniques used in teaching science. The school questionnaire offers measures of the following school institutional factors: (1) whether the school is public or private; (2) whether it is an all-male or all-female school; (3) percentage of school’s funding from the government, student fees, and benefactors; (4) students’ ability grouping within schools; (5) number of accountability procedures; (6) school academic selectivity, and (7) school grade span. School resources and teacher-related factors, all obtained from the school questionnaire, include: (1) class size; (2) school size; (3) pupil-teacher ratios, (4) teacher education (percentage of teachers with masters degree); (5) teacher certification (percentage of certified teachers); (6) school resources (percentage of school computers linked to web, ratio of computers for instruction to school size, and an index of school resources shortage15); and (7) dummies representing the size of the school community. Child characteristics Two child characteristics are considered in the model: gender and immigrant background. Gender is represented by a 0-1 dummy variable where: “1” represents male and “0” female. The immigrant background is derived from the students’ responses about whether or not they and/or 14
Tutoring is outside of normal school hours, but it can be provided by a school or non-school teacher. This composite index measures the school principal’s perceptions about resources limitations hindering instruction at school. It considers the following 14 school resources: science teachers, math teachers, native language teacher, teachers of other subjects, laboratory technicians, support personnel, science laboratory equipment, instructional materials (e.g., textbooks), computers for instruction, internet connectivity, computer software for instruction, library materials, and audio-visual resources.
15
11
their mother and their father were born in the country of assessment or in another country. It is represented by a 0-1 dummy variable where “1” pertains to native students and “0” is otherwise. Country Fixed Effects Country-specific effects are accounted for by including country dummy variables for each country considered in the analysis, with US as the omitted country. Appendix 1 shows means and standard deviations for all variables. Data Limitations It should be noted that since this data is cross-sectional, we cannot make rigorous causal inferences about the relationships examined here. In the case of the family variables considered here, causality has been established, generally, by studies using more rigorous longitudinal designs (Armor 2003, Duncan and Brooks-Gunn 1997). Since one of our major interests is in explaining country differences, our major caveat would be that any reduction of achievement differences due to school variables would be contingent upon assuming that those school factors are causally related to achievement outcomes. A second caveat concerns the international comparisons of all variables considered here. It is one thing to examine relationships among school, family, and achievement variables with a single country, it is something else to carry out these analyses between countries with very different languages and cultures. We trust that the PISA project has taken these comparability issues into account the best that they can, but there is still the possibility that school or family measures take on somewhat different meanings across two or more countries. This might tend to reduce relationships rather than increase them, or perhaps lead us to interpret them in ways that are not quite consistent across countries.
12
Empirical Model To assess the impact of family SES and school factors on mathematics and science test scores this research estimates the following model: Tij Fi S ij C i Dc ij
(1)
where i indexes individual students and j indexes the school. Tij is the test score (either math or science) of student i in school j, Fi is a vector of variables measuring family SES, Sij is a vector of variables measuring school factors, Ci is a vector of variables accounting for child characteristics, Dc is a vector of dummy variables for countries, and ij is the error term. The constant terms α and the coefficient vectors β, θ, , and are to be estimated. The stratified nature of the PISA data causes two major problems in the estimation of equation (1). The first estimation problem derives from the fact that the error term ij violates the assumption of independence.16 One way of circumvent such estimation problem is, as shown by Moulton (1986), to consider higher-level error components to obtain appropriate standard errors estimates and thereby avoiding spurious results. Therefore, the error term ij of equation (1) is assumed to have a school-specific error component (j) in addition to the student-specific error component (i):
ij = j + i
(2)
Clustering-robust linear regression (CRLR) is used to estimate standard errors clustered at the school level. The CRLR method estimates standard errors assuming that observations might be 16
Three major reasons contribute to the violation of the assumption of independence of the errors. First, the independent variables used in this study are measured at different levels with some of them not varying within classes or schools. Second, students’ performance within the same school may not be completely independent from one another. Finally, the primary sampling unit of the two-stage sampling procedure used in PISA is the school and not the individual student.
13
correlated within schools but are independent across schools. By allowing some correlation within schools, CRLR estimates appropriate standard errors when many observations (students) share the same value on many school-level variables. The second estimation problem results from the fact that different students have distinct sampling probabilities because PISA data is also stratified by country. Therefore, in estimating equation (1), in addition to use CRLR, weighted least squares (WLS) estimation procedure using sampling weights is employed. The WLS procedure ensures that the proportional contribution to the parameter estimates of each stratum in the sample is the same as would have been obtained in a completely random sample (Wooldridge 2001).
Results Figure 1 displays the mean scores of large developed OECD countries as compared to the US before any controls for SES or school factors. The deviations show that, among the ten largest developed OECD countries, most countries rank significantly above US students. Only Italy ranks lower in math and only Italy and Spain rank lower in science, although none of these differences are statistically significant. In addition, the small advantages of Spain in math (5 points) and France in science (6 points) are not statistically significant. There are sizeable cross-country differences in family, school, and child factors by country (shown in Appendix 2). Most important for our purposes, the mean differences between the US and the other large developed OECD countries are quite substantial in many of the variables used to represent family SES and school factors.
14
With regards to family SES, the US shows a rather mixed picture. It has the second highest parental occupational status, a middle-range parental educational level, and middle- to low-range positions in variables measuring family economic means (except for the number of cars where the US ranks third). Australia, Canada, and Germany show relatively high values in all family SES variables, whereas France and the UK tend to have middle-range values in all of them. Italy and Spain tend to have relatively low SES variables. In contrast, Japan and Korea show a less clear picture by performing comparatively well in some variables and bad in others. In terms of school programmatic factors, the variables selected reveal that the US differs substantially from the other countries though in a non-uniform way. In fact, the US combines one of the highest levels of (1) variety pedagogical techniques used in classes, (2) provision of skills in science-related subjects, and (3) science-related topics learned at school, with (4) one of the lowest levels of hours spent learning math per week in regular lessons and (5) number of mandatory courses in science. The number of mandatory courses in science is one of the programmatic factors where the countries differ most from the US, ranging between an average of 1.8 in Korea and 5.4 in Germany. The number of voluntary courses in science (ranging from 0 in Italy to 3.1 in Korea) and the number of hours spent learning science per week in regular classes (ranging from 2.7 in Japan to 4.2 in the UK), are two other variables where the crosscountry differences are particularly significant. Korea distinguishes itself from the group by showing values considerably higher than any other country in three out of the eight programmatic factors considered: (1) hours spent learning math per week in regular lessons, (2) amount of tutoring received, and (3) number of voluntary science courses. Cross-country differences are also substantial in all variables representing school institutional factors. Korea shows a much smaller proportion of government funded schools (47
15
per cent) than any other country, all showing figures ranging from 71 per cent in Japan to 96 per cent in Germany, with the US ranking third with a value of 88 per cent. The same holds for the proportion of public schools, where the low Korean figure (53 per cent) contrasts with values ranging from 67 per cent in Spain to 96 per cent in Canada (92 per cent in the US). The US schools have one of the lowest levels of students’ academic selectivity (1.7), substantially below the Japanese schools which rank first with an average value of 3.6. In contrast, US schools are among the best performers in terms of both school accountability and grouping students according to their abilities. With respect to school resources and teacher-related factors the differences among countries are considerable as well. The US has the largest average school size and intermediate figures in both class size and pupil-teacher ratio. The average class size ranges from 24.7 (UK) and 36.3 (Japan); average school size varies between 668 (Germany) and 1390.1 (US); and pupil-teacher ratio goes from 9.5 (Italy) and 17.3 (Germany). The proportion of certified teachers is relatively similar among countries (from 91.3 per cent to 99.3 per cent) with the US occupying a middlerange position with a value of 94.8 per cent. In contrast, the country-differences in the proportion of teachers with a master degree are pronounced, ranging from 9.6 per cent in the UK to 99.3 per cent in Korea with the US (95.2 per cent) in an intermediate position. The US is among the three best performers in terms of school computers linked to web (ranging from 84.7 per cent in Japan and 97.3 per cent in Canada), ratio of computers for instruction to school size (from .09 in Germany to .27 in the UK), and school resources shortage (from 1.6 in Japan to 2 in Germany). The US (14 per cent) has the largest proportion of schools in communities with less than 3,000 inhabitants, while Korea (85 per cent) and Japan (65 per cent) have the largest proportion of schools in communities with more than 100,000 inhabitants.
16
Given these substantial cross-national differences in family SES and school factors, and in particular the differences between the US and the other nine largest OECD countries, it is reasonable to postulate that these factors might be contributing to cross-national differences in achievement and in the relatively low ranking of the US to the other countries. Table 1 shows the estimation results for equation (1) excluding the school factors (vector Sij), to show the extent to which family SES contributes to explain cross-national achievement disparities and particularly the relatively low performance of the US among the 10 largest developed OECD countries. In line with previous findings, the estimation results show that all the family SES variables are positively and significantly associated with achievement scores, controlling for countryspecific effects and child characteristics such as gender and immigrant background. More importantly, the results also indicate that when differences in family SES and student characteristics are accounted for, the relative ranking of the US among its peers remains unchanged in math and decreases one position in science, ranking ninth (only above Italy) in both subjects. Nonetheless, differences in family SES and student characteristics are responsible for changes in the magnitude of the achievement differences between the US and the other large developed OECD countries. They cause the US achievement gap to shrink in comparison with Canada, Australia, and Germany (by 5 to 10 score points), and to increase with respect of France, Spain, and Japan (by 4 to 11 score points), in both subjects. Taken together, family SES, country-specific factors, and child characteristics account for around one quarter of the cross-national variation in students’ achievement in both subjects. Since the school variables which originated from the school context questionnaire are not available for France, two different full versions of equation (1) are estimated, each defining
17
vector Sij slightly differently. The first model uses just the school variables for which data is available for the 10 countries, which consists only of the school variables for school programmatic factors which come from the student context questionnaire (Table 2). The second model uses all school variables but excludes France from the analysis (Table 3). Table 2 presents the first (limited) model. All but one of the school programmatic variables have a significant effect on student math and science achievement after controlling for family SES and student characteristics, although not necessarily in the expected positive direction. The exception is number of science-related topics learned at school, which is significant for math but not for science. Students who have a greater agreement that school provides skills in sciencerelated subjects, who spend more hours per week studying science and math in regular classes, and who have a greater number of mandatory courses in science tend to perform better in both math and science, controlling for all other variables. In contrast, a greater number of voluntary courses in science, more tutoring, and exposure to a greater variety of pedagogical techniques are associated with lower student achievement in math and science. Once school programmatic factors are accounted for, the impact of family SES variables on students’ achievement reduces by an amount ranging from one tenth (number of books at home) and one third (index of home possessions), both in math and science. Nevertheless, all family SES variables remain positively and significantly associated with achievement scores in both subjects. Being a native student, as opposed to having an immigrant background, does not significantly impact math scores when family SES, school programmatic factors, gender, and country-specific effects are controlled for, but it remains positively and significantly associated with science scores. Boys continue to perform significantly better than girls in both subjects even
18
after these controls. Adding school programmatic factors to the model increases the variance explained to about 40 percent in math and 37 percent in science. It is important to note that the magnitude, significance, and direction of the impacts of family SES, student characteristics, and school programmatic factors on students’ achievement are similar regardless whether France is included or excluded in the analysis. This is important because in the third model France has to be excluded due to missing data. School programmatic factors help explain the relatively low performance of the US in science but not so much in math. The US continues to rank ninth in math when France is included in the analysis, rising to seventh (above Spain and Italy) when France is dropped. For science, however, the US relative position moves up from eighth to seventh, ranking above France, Spain, and Italy. When France is dropped, the US relative position in science achievement remains unchanged. Table 3 shows estimation results for the full model including all school variables, with France excluded. Except for the number of science-related topics learned at school, all the other variables representing school programmatic factors remain statistically significant after controlling for other school factors as well as family SES. The same pattern also holds for the three programmatic variables with significant positive effects and the three with significant negative effects. Among the school institutional factors, only ability grouping and being a public by opposed to a private school (minus Australia) is significantly associated with student achievement in both subjects. Holding everything else constant, schools with more ability grouping tend to have lower scores in both math and science. Being a public versus private school is positively
19
associated with students’ scores in both subjects if Australia, which does not have this variable, is omitted. Among the several variables measuring school resources and teacher-related factors, the resource shortage index, the ratio of computers for instruction to school size, and teacher education (minus Spain) are significant. A greater degree of school resource shortages is associated with lower scores in both math and science, while more instructional computers per student is associated with lower students’ scores in both subjects, holding everything else constant. The proportion of teachers with a master degree is positively associated with achievement scores in both subjects if Spain, which lacks teacher-related variables, is omitted. Interestingly, school resources variables such as student-teacher ratio, class size, and school size do not have significant effects on either math or science scores. When the full array of school factors are taken into account, the magnitude of family SES effects are reduced by more than one third for all variables except parental education. Nevertheless, all family SES variables continue to be significantly and positively associated with both math and science scores. In other words, among the 10 largest developed OECD countries, students from a higher socio-economic background tend to do better in both math and science even when school factors are extensively controlled for in addition to student characteristics and country-specific factors. Finally, family SES and school factors do explain some of the cross-national differences in achievement performance among the 10 largest OECD countries, and in particular they help explain the low levels of the US. Figure 2 shows the country rankings for math before and after controlling for all variables. These factors account for a substantial portion of the US performance gap in relation to five out of the seven countries scoring higher than the US. Japan
20
drops from a 49 point gap to a 14 point gap which is not significant, and Germany drops from a 29 point advantage to -7 which is not significant. Canada drops from 53 to 25 points, Australia drops from 42 to 25 and Korea drops from 73 to 57. SES and school factors do not help to explain the US achievement gap in math with the UK and France, although the gaps are not as large as other countries to begin with. Indeed, it is noteworthy that with the exception of Korea, all the gaps with the US are no more than one-fourth of a standard deviation. For science differences shown in Figure 3, SES and school factors explain a substantial portion of the achievement gap between the US and five of the six countries scoring above it. Japan changes from 42 to 4 points above (not significant), and Germany drops from 27 to minus 14 (not significant). Canada drops from 46 to 16, Australia from 33 to 17, and Korea from 33 to 18. The only exception is the UK who remains about 28 points above with or without controls.
Conclusions and Policy Implications The international comparison of student achievement using PISA 2006 data reveals that US students score much lower than most other developed countries in both mathematics and science. Among the 10 largest developed OECD countries, on average, US students rank above only one country (Italy) in mathematics and two countries (Italy and Spain) in science. However, these rankings based on raw scores do not take into account possible differences in both family SES and school variables. We find that family SES, school programmatic factors, and some other school resource and institutional factors (particularly ability grouping capacity within schools), are all significantly related to math and science achievement across all countries. Overall, all these factors account for 39 percent of achievement variation in mathematics and 35 per cent in science at the student
21
level. This is substantially larger than in previous studies using PISA and TIMSS data, where the proportion of achievement variation accounted for at the student level ranges from 22 to 32 percent in math and from 19 to 26 percent in science (Fuchs and Woessmann 2007, Woessmann 2003). Consistent with previous findings (e.g., Woessmann 2004, Xia 2009), even among countries with comparable and high levels of economic development, family SES factors significantly impact students’ achievement in both math and science. Moreover, family SES is found to strongly and positively impacting students’ achievement before and after controlling for the effect of a vast array of school factors. However, after controlling for school effects, the magnitude of the impact of family SES variables on students’ achievement reduces by more than one third for all variables except parental education. The effects of family SES variables on achievement, while significant, tend to be similar in size among comparable developed countries17 and therefore they do not help explain the relatively low performance of the US among its peers. In fact, the results indicate that when only family SES factors and student characteristics are taken into account, the relative low position of the US among its peers remains unchanged in math and deteriorates one position in science, ranking ninth (only above Italy) in both subjects. In contrast, school differences seem to matter more in explaining differences among the ten countries as well as the relatively low achievement performance of the US. After adding controls for all of the school variables, only five countries are significantly higher than the US in math—and most of the differences have been reduced substantially. One country is significantly lower, and three countries (Japan, Spain, and Germany) are not significantly different from the 17
This inference is consistent with prior findings suggesting that the impact of family background on achievement is strong and remarkably similar in size among the US and Western European Countries (Woessmann 2004).
22
US in Math. In science, the changes are even more marked. There are only four countries significantly higher than the US (and all but one gap is reduced), two countries are significantly lower, and there are no significant differences for three (Japan, France, and Germany). Finally, the evidence suggests that among the school factors considered, all of the school programmatic factors, ability grouping, public versus private school, teacher education, and lack of school resources appear to be the more important school factors in explaining the achievement gap between the US and the other large developed OECD countries. Among the school programmatic factors, it is noteworthy for policy purposes that three have unexpected negative effects – amount of tutoring received, the number of voluntary science courses, and the variety of pedagogical techniques. Also we note the negative effects for ability grouping and the negative effects for instructional computers per student. It is clear that while SES and school factors explain some of the differences among countries and appear to raise the US ranking to some degree, there are other unmeasured factors that may explain country variations. Examples might be length of school year, length of school day, and especially cultural factors that cause variations in student motivation and parental emphasis on schooling. The latter is often mentioned in connection with the very high scores of Korea. Length of school year is also mentioned, but in the case of these countries it does not play a clear role. While high-scoring Korea and Canada both have relatively long school years of 200 days (compared to 180 days in the US), lowest-scoring Italy also has a school year of 200 days.18 There are several potential policy implications for these findings, although more research is needed to confirm some of the unexpected relationships. Among the more interesting are
18
See International Review of Curriculum and Assessment (INCA) website, Comparative Table 15.
23
certain significant school effects, both positive and negative, as well as some of the factors that have no significant effects. For example, on the one hand US policy has generally emphasized hard resources such as smaller classes, more certified teachers, and computers, all of which are expensive. According to this international data, adding to these school resources are unlikely to raise achievement or close the gap between US and its peers. On the other hand, the significantly positive results for hours of studying math and science and number of mandatory science courses, suggest that withinschool curriculum changes – which might be less costly – could hold more promise than reducing class size and adding more computers. Importantly, Appendix 2 shows that the US has lower average hours of math instruction than all but two countries, which happen to be Spain and Italy. The proportion of teachers with masters degrees also has a positive effect, but since the US rate is already 96 percent for 15 year olds, there is not much room for improving this input. Among the more interesting findings are the negative effects for outside-school tutoring, the number of voluntary science courses, and, especially, variety of pedagogical techniques. We are not certain of the meaning of the number of voluntary science courses, but the tutoring finding could indicate a self-selection effect, whereby lower achieving students are the ones choosing to have outside tutoring. Perhaps the most intriguing finding is the negative effect for variety of pedagogical techniques, most of which involve greater student participation in various classroom activities (e.g., asked for their opinions, asked to design their own experiment, etc.). The growth of more student-centered activities rather than the top-down approach of teacher-centered classrooms has become popular in the US, and indeed Appendix 2 shows that the US has the highest score on this variable, while Japan and Korea have the lowest. This might lead to some re-thinking of
24
whether student-centered instruction is a good approach for improving achievement, at least when it comes to the more technical topics of math and science. As a final note, we reiterate our caveats in the Data and Methods section. Namely, since the PISA data are cross-sectional, our findings about school factors explaining country differences are subject to further, more rigorous studies that establish a causal relationship between the school variables in question and the math and science achievement outcomes. Further, some of the interpretations we make, or some of relationships that were not significant, might be affected by cross-cultural differences that cause a variable in one country to take on a somewhat different meanings in other countries.
25
Figure 1 – Mean Achievement Scores Compared to US (no controls) Mathematics Korea Canada Japan Australia Germany France UK Spain USA Italy
Science 73*** 53*** 49*** 42*** 29*** 21** 21**
5 0 ‐14 440
460
480
500
520
540
46***
Canada Japan Korea Australia UK Germany France USA Spain Italy
560
42*** 33*** 33*** 28*** 27*** 6 0 ‐2 ‐13 440
460
480
500
520
Source: Pisa 2006 student-level data. Legend: *p
