Discrepant SAT Critical Reading and Writing Scores ...

This article was downloaded by: [Emily Shaw] On: 08 September 2011, At: 07:59 Publisher: Routledge Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

Educational Assessment Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/heda20

Discrepant SAT Critical Reading and Writing Scores: Implications for College Performance a

a

Emily J. Shaw , Krista D. Mattern & Brian F. Patterson a

a

The College Board

Available online: 08 Sep 2011

To cite this article: Emily J. Shaw, Krista D. Mattern & Brian F. Patterson (2011): Discrepant SAT Critical Reading and Writing Scores: Implications for College Performance, Educational Assessment, 16:3, 145-163 To link to this article: http://dx.doi.org/10.1080/10627197.2011.604241

PLEASE SCROLL DOWN FOR ARTICLE Full terms and conditions of use: http://www.tandfonline.com/page/terms-and-conditions This article may be used for research, teaching and private study purposes. Any substantial or systematic reproduction, re-distribution, re-selling, loan, sub-licensing, systematic supply or distribution in any form to anyone is expressly forbidden. The publisher does not give any warranty express or implied or make any representation that the contents will be complete or accurate or up to date. The accuracy of any instructions, formulae and drug doses should be independently verified with primary sources. The publisher shall not be liable for any loss, actions, claims, proceedings, demand or costs or damages whatsoever or howsoever caused arising directly or indirectly in connection with or arising out of the use of this material.

Educational Assessment, 16:145–163, 2011 Copyright © Taylor & Francis Group, LLC ISSN: 1062-7197 print/1532-6977 online DOI: 10.1080/10627197.2011.604241

Downloaded by [Emily Shaw] at 07:59 08 September 2011

Discrepant SAT Critical Reading and Writing Scores: Implications for College Performance Emily J. Shaw, Krista D. Mattern, and Brian F. Patterson The College Board

Despite the similarities that researchers note between the cognitive processes and knowledge involved in reading and writing, there are students who are much stronger readers than writers and those who are much stronger writers than readers. The addition of the writing section to the SAT provides an opportunity to examine whether certain groups of students are more likely to exhibit stronger performance in reading versus writing and the academic consequences of this discrepant performance. Results of this study, based on hierarchical linear models of student performance, showed that even after controlling for relevant student characteristics and prior academic performance, an SAT critical reading–writing discrepancy had a small effect on 1styear grade point average as well as English course grades in college. Specifically, students who had relatively higher writing scores as compared to their critical reading scores earned higher grades in their 1st year of college as well as in their 1st-year English course(s).

The addition of the writing section to the SAT in March 2005 not only allowed admission officers the opportunity to better understand students’ writing skills but also allowed students to show another aspect of their knowledge, skills, and abilities—those related to writing—that they will need and surely use in their 1st year of college. Although there were a number of proponents of this change, including the president of the University of California system at the time (Atkinson, 2002), numerous critics were skeptical of the new section, claiming that it would only increase test-taking time while adding little value to the meaning of their scores (Baron, 2005; Perelman, 2005). Many thought the new section might be redundant or capture many of the similar cognitive dimensions that were already being examined on the critical reading section. Although reading and writing do rely on many of the same cognitive skills, they are essentially utilized in different ways within these two domains (Kucer, 1987, 2005; Langer & Flihan, 2000; Rosenblatt, 1994; Shanahan & Lomax, 1986; Tierney & Shanahan, 1991). Therefore, it is not surprising that some students have much higher writing than critical reading Correspondence should be sent to Emily J. Shaw, Research & Development, The College Board, 45 Columbus Avenue, New York, NY 10023. E-mail: [email protected]

145


146

SHAW, MATTERN, PATTERSON

scores, and vice versa. Very few studies, however, have effectively analyzed the discrepant reading and writing performance of high school students, despite Stotsky’s (1983) call for such research more than two decades ago. The addition of the writing section to the SAT provides a fitting opportunity to examine this performance discrepancy and to understand the student characteristics and academic consequences associated with much stronger performance in critical reading than writing, and vice versa. A practical understanding of the consequences of discrepant performance in these two domains is particularly useful for admission and enrollment professionals who examine student test scores in conjunction with other student information on an application. Having access to empirical research on the impact of discrepant critical reading and writing performance and its relationship with 1st-year college performance would provide context and deeper understanding when highly discrepant SAT critical reading and writing scores are present during holistic or “whole folder” reviews of applicants (Rigol, 2003, p. 9). This study examines whether certain groups of students are more likely to have higher writing performance as compared to critical reading performance, and vice versa. In addition, this study examines the relationship between discrepant SAT critical reading and writing performance and college outcomes, including 1st-year grade point average (FYGPA) and 1styear English grade point average (FY English GPA). This predictive model includes relevant variables such as academic performance (i.e., SAT total score, high school grade point average [HSGPA]) and demographic variables (i.e., gender, best spoken language, and race/ethnicity) to determine the independent impact of the discrepancy on 1st-year grades. Finally, the interaction between the demographic variables and the critical reading and writing performance discrepancy will be included in the model to determine whether discrepant performance has a different relationship with college outcomes for certain subgroups of students.

REVIEW OF THE LITERATURE It is largely accepted that reading and writing are highly related processes. Although there are clear overlaps in many of the component skills and knowledge bases (Shanahan, 1984, 1987; Stotsky, 1983), there are also differences between reading and writing that are sometimes not as clear (Fitzgerald & Shanahan, 2000; Kucer, 1985; Langer, 1986a, 1986b; Shanahan, 1984, 1987; Stotsky, 1983; Tierney & Shanahan, 1991). Studies primarily examining elementary and middle school students have found that when writing, students tend to be more concerned with bottom-up issues such as syntax, mechanics, and lexical choices than when reading (Gleason, 1995; Kucer, 1985; Langer, 1986b). Students are also more likely to set goals and be more cognizant of the strategies employed while writing versus reading. When reading, however, students are more focused on the content and validation of their understanding of the meaning behind the text. The few studies that have examined discrepant reading and writing performance in depth have looked at this issue among relatively small samples of elementary and middle school students utilizing discrepant performance on local measures to primarily classify students as good readers/good writers, good readers/poor writers, or poor readers/poor writers (e.g., Honeycutt, 2002; Jordan, 1986; Thacker, 1990, 1991). Jordan (1986), for example, looked at the differences between good readers/good writers’ and good readers/poor writers’ composing


DISCREPANT SAT CRITICAL READING AND WRITING

147

processes using think-aloud protocols and used a set of descriptive categories to code reading and writing behaviors. This research showed that good readers/good writers (a) were able to abstract content from a reading passage and write about what they read in their own words, (b) were more aware of the structural features of sentences, and (c) spent much more time planning prior to writing. Good readers/poor writers struggled with each of these aspects. Similarly, based on 90 ninth-grade students divided equally into groups of good readers/good writers, good readers/poor writers, and poor readers/poor writers, Thacker (1990, 1991) studied students’ ability to understand and recognize varying degrees of text organization when reading. He found that good readers/good writers and good readers/poor writers were both skilled at distinguishing between well and poorly organized text. However, good readers/poor writers seemed to lack an awareness of how cohesive ties can bring meaning to disorganized text and would likely benefit from greater instructional focus on cohesive relationships and the effective organization of their own written responses to material. Given that there are students with highly discrepant reading and writing performance and that these students do tend to approach the two related tasks in different ways than students with more consistent reading and writing performance, it would seem useful to study the impact of discrepant reading and writing performance at the postsecondary level. Although discrepant SAT critical reading and writing performance has not yet been studied in relation to postsecondary performance, other analyses related to discrepant SAT performance patterns have been conducted. For example, a recent study by Mattern, Camara, and Kobrin (2007) showed that there are sizeable groups of students considered to have discrepant critical reading and writing scores on the SAT. Mattern et al. standardized students’ critical reading and writing scores across the 2006 College Bound Seniors cohort1 to examine the difference between these scores. Those students with critical reading and writing scores that differed by 1 or more standard units were considered to be discrepant. There were 49,356 students (3.6% of the cohort of test takers) who scored 1 or more standard units higher on writing than critical reading (referred to as better at writing), and there were 50,336 students (3.7% of the cohort of test takers) who scored 1 or more standard units higher on critical reading than writing (referred to as better at critical reading). Mattern et al. (2007) also investigated whether there were any differences between the performance groups with regard to gender and/or racial/ethnic composition. They found that the better at writing group was comprised of almost twice as many female as male students. Conversely, the better at critical reading group had almost twice as many male as female students. With regard to race/ethnicity, the only differences noted were that the percentages of White and American Indian/Alaskan Native students were higher in the better at critical reading group, whereas the percentage of Asian, Asian American, and Pacific Islander students was higher in the better at writing category. The difference in HSGPA among the better at critical reading group, better at writing group, and students who scored similarly on both sections was also investigated. Mattern et al. found that HSGPA was the highest for the better at writing group and lowest for the better at critical reading group, with significant differences (p < .05) among all three groups based on analysis of variance (ANOVA) results. In addition to noting gender and racial/ethnic differences in discrepant critical reading and writing performance on the SAT, in a different study, Shaw (2007) also found that students 1 Comprised

of students with an SAT or SAT Subject Test score that reported to graduate from high school in 2006.


148


that were much stronger in SAT writing than SAT critical reading were significantly more likely (p < .05) to have taken English as a Second Language coursework in high school than students who were much stronger in SAT critical reading than SAT writing. Shaw speculated that perhaps this much stronger writing than reading performance for students with English as a Second Language experience was related to issues of biliteracy, noting that Holm and Dodd (1996) found that students from nonalphabetic written language backgrounds tend to struggle with new or unfamiliar words when attending universities where English is the medium of instruction. There are likely many unfamiliar words to students on the SAT critical reading section, whereas the writing multiple-choice section tends to be more rule based and related to grammar and the SAT essay is student produced, free from most of the constraints placed on a reader by the author of an existing text. These demographic and academic (HSGPA) differences between the discrepant groups signal that there may also be differences in how these students perform in college, or specifically how the magnitude and direction of the critical reading and writing discrepancy might impact 1styear college performance, particularly in English coursework. Despite the importance and value in integrating writing across the disciplines, many content areas require few or no writing assignments, nor do they offer the corresponding writing instruction that would foster success on the writing assignments (Lavelle, 2003). Writing scholars have remarked that training in writing has all too often become the sole responsibility of the freshman English or composition course (Lavelle, 2003; Moore, 2003). Understanding the connection between the critical reading and writing discrepancy with English course performance would be particularly interesting, as both reading and writing activities are both considered central to freshman English coursework (El-Hindi, 1997; Flower et al., 1990). However, Bosley (2008) commented that although there is a large body of literature that documents the value in teaching critical reading and writing reciprocally in the classroom, most college composition courses do not explicitly cover critical reading strategies or effectively integrate reading into the writing lessons and assignments. Further complicating our understanding of undergraduate English coursework and performance, Lavelle (2003) observed that 1st-year composition grades do not always reflect the students’ writing skills (such as analysis, synthesis, transcription, and revision) but includes contaminating factors like attendance, promptness, or public speaking skills. This model-based study allows for the systematic investigation of differences in 1st-year college performance, in addition to descriptive information about discrepant reading and writing performance by different subgroups. Unlike the few previous studies that have focused on describing students with discrepant reading and writing performance, the current research focuses on the academic consequences of discrepant reading and writing performance in college. Moreover, studying discrepant reading and writing performance on the SAT allows for the examination of this issue on a much larger, national scale than has been studied in the past. Also different from previous studies on discrepant SAT reading and writing performance, which developed categorical groups of discrepant performance for analysis (i.e., Mattern et al., 2007; Shaw, 2007), this study uses a continuous discrepancy measure to avoid losing any measurement precision associated with dichotomizing continuous variables (MacCallum, Zhang, Preacher, & Rucker, 2002). More than 1.5 million students take the SAT each year (College Board, 2010), and the test is used in admission decisions at the large majority of 4year colleges and universities, rendering the results of the current study to be both theoretically


149

and practically useful to various stakeholders. In a similar vein, much of the research in this area is based on small samples, thereby potentially limiting the generalizability of the results, whereas the current study includes a sample of more than 140,000 attending a diverse set of 109 postsecondary institutions.

METHOD


Participants The sample for this study is based on the students in the national SAT Validity Study sample (for details, see Kobrin, Patterson, Shaw, Mattern, & Barbuti, 2008). For the national SAT Validity Study, a wide range of 4-year institutions in the United States submitted 1st-year college performance data to the College Board on the first-time, 1st-year students who entered their institutions in the fall of 2006. The final sample in the current study included 140,919 students—all with valid SAT scores, self-reported high school GPAs, and FYGPAs provided by their college/university, from 109 four-year institutions in the United States.

Measures FYGPA. Each participating institution supplied FYGPA values for their 2006 first-time, 1st-year students. The range of FYGPA across institutions was 0.00 to 4.27.2 FY English GPA. Each participating institution supplied grades for all of the courses taken by their 1st-year, first-time students during the 2006–2007 school year. All coursework was coded for the subject area of the course. Those courses coded as English courses taken in the 1st year of college3 were averaged for each student and considered to be the students’ FY English GPA. Of the 140,919 students in the sample, 101,765 took at least one English course; therefore, analyses based on FY English GPA are based on that subset of students. SAT critical reading section. The SAT critical reading section, scored on a scale ranging from 200 to 800, consists of 67 items in two 25-min sections and one 20-min section. The SAT critical reading section measures a student’s ability to read and think carefully based on sentence completions and items related to passages ranging in length from 100 to approximately 850 words and on topics from literary fiction to natural sciences. There are 19 sentence completion items and 48 passage-based reading items, all of which fall into three general content categories: extended reasoning (42–50 items), literal comprehension (4–6 items), and vocabulary in context (12–16 items). 2 Although

a few institutions’ GPA scales ranged from 0.00 to 4.33, most had a maximum of 4.00. and Patterson (2010) examined 1st-year college coursework in English across a national sample of 4-year institutions and found that the vast majority of English courses are composition courses (72%). 3 Shaw

150



SAT writing section. The SAT writing section consists of one 25-min essay, one 25-min multiple-choice section, and one 10-min multiple-choice section with a total of 60 items. The SAT writing section measures a student’s ability to improve sentences, identify sentence errors, improve paragraphs, and write an essay that will assess a student’s ability to think critically and write effectively in response to a prompt adapted from an authentic text, under time constraints similar to those encountered in essay tests in college courses. The essay score scale ranges from 0 to 12 and is based on the ratings of two trained essay readers. The multiple-choice writing section counts for approximately 70%, and the essay counts for approximately 30% of the total raw score, which is used to calculate the 200 to 800 scale score for the section. SAT Questionnaire. The SAT Questionnaire is a survey administered to all students when they register for the SAT, either online or by mail. It consists of 42 questions about the student’s background, high school experiences, and plans for college. Self-reported gender, race/ethnicity, and best language spoken were obtained from the SAT Questionnaire. Design and Procedure This study focused on discrepant critical reading (CR) and writing (W) performance as indexed by the difference in the two SAT section scores (CR-W Discrepancy D SAT CR–SAT W). Unlike previous research, which created categorical groups based on a difference score (i.e., Mattern et al., 2007; Shaw, 2007), the continuous CR-W Discrepancy variable was used in subsequent analyses to avoid any loss of information. The CR-W Discrepancy scores ranged from 320 to 530 with a mean of 6.16 and a standard deviation of 60.58, indicating that, on average, students’ SAT critical reading scores were 6 points higher than their writing scores. The first set of analyses examined whether the magnitude of CR-W Discrepancy differed for specific subgroups, and if so, in what direction. That is, the average CR-W Discrepancy score was computed overall and by gender, race/ethnicity, and best language subgroups. Student’s t tests were conducted to examine whether the average CR-W Discrepancy score was significantly different from zero (p < .01), where a zero value indicates that, on average, students earn the same SAT critical reading and writing score. This information is useful in highlighting the commonalities and differences that characterize students exhibiting discrepant critical reading and writing performance on the SAT and can be particularly helpful in the design and development of specific educational interventions for their weaker area. In addition, this study examined whether a student’s CR-W Discrepancy score is related to subsequent college performance (i.e., FYGPA and FY English GPA), above and beyond traditional measures of academic performance and student characteristics. Such analyses can inform whether discrepant performance has a positive, negative, or insignificant effect on college success. Also, whether the direction of the discrepancy (i.e., higher writing performance vs. higher critical reading performance) mattered in relation to future college performance was examined. Finally, the impact of discrepant reading and writing performance on college performance for different student subgroups, such as English Language Learners, was analyzed. To study these research questions, hierarchical linear modeling (HLM) techniques were employed because of the inherent nested structure (i.e., students within colleges) of the data (Raudenbush & Bryk, 2002). In the first step, a model with student-level demographic characteristics (i.e., gender, race/ethnicity, best spoken language) and academic performance measures

151


(i.e., HSGPA, SAT composite score) was estimated. Next, the CR-W Discrepancy variable was added and the change in model fit was examined using the Akaike Information Criterion (AIC). Finally, interactions among student subgroups and the CR-W Discrepancy were added to the model, and again the change in AIC was computed to examine the change in model fit.

RESULTS


Descriptive Statistics The sample size, mean, and standard deviation for each academic performance measure for the total sample and by gender, racial/ethnic, and best spoken language subgroups are provided in Table 1. In general, female participants earned higher grades in high school and college and had slightly higher SAT writing scores as compared to male participants who had higher SAT math and critical reading scores. With regard to racial/ethnic comparisons, White and Asian students outperformed the other subgroups on all academic indicators. Students whose best spoken language was English earned the highest grades in their 1st-year English courses; however, students whose best spoken language was something other than English had the highest FYGPA. It should be pointed out that not all students took a 1st-year English course, and therefore the sample sizes for that measure are smaller than for the other academic indicators. To determine whether certain students performed more discrepantly on the SAT critical reading and writing sections than other students, the mean CR-W Discrepancy score overall and by student subgroups is provided along with the corresponding t-test results in Table 2. As mentioned previously, the average CR-W Discrepancy score was 6.16 (SD D 60.58), indicating that, on average, students in this sample had critical reading scores that were six points higher than their writing. This is similar to the national results, where the average critical reading

TABLE 1 Descriptive Statistics of Study Variables by Student Characteristics HSGPA Subgroup Overall Gender Female Male Race/Ethnicity American Indian Asian Black Hispanic Other White Best spoken language Another English and another English only

SAT CR

SAT M

SAT W

FYGPA

FY English GPA

n

M

SD

M

SD

M

SD

M

SD

M

SD

n

M

SD

140,919

3.61

0.50

560

95

579

96

553

94

2.97

0.71

101,765

3.12

0.84

75,940 64,979

3.65 3.55

0.48 0.52

556 564

95 95

559 602

93 95

556 550

93 94

3.05 2.89

0.67 0.74

56,064 45,701

3.21 3.01

0.78 0.89

772 13,775 9,944 10,338 4,372 101,718

3.53 3.67 3.40 3.59 3.58 3.62

0.54 0.47 0.55 0.51 0.50 0.49

544 563 508 525 559 568

87 104 88 93 98 92

555 624 505 538 573 585

89 97 87 94 98 92

529 563 499 520 554 561

88 101 87 90 97 91

2.78 3.05 2.64 2.74 2.96 3.02

0.76 0.66 0.73 0.77 0.71 0.69

538 9,426 8,187 7,123 3,217 73,274

2.97 3.15 2.80 2.83 3.08 3.18

0.92 0.79 0.94 0.97 0.83 0.80

1,605 7,033 132,281

3.62 3.62 3.61

0.52 0.49 0.50

464 531 562

99 100 94

606 570 579

114 108 95

480 534 555

102 101 93

3.05 2.90 2.98

0.68 0.72 0.70

1,139 4,959 95,667

3.09 2.97 3.12

0.84 0.89 0.83

Note. HSGPA D high school grade point average; CR D critical reading; M D math; W D writing; FYGPA D 1st-year grade point average.

152


TABLE 2 SAT CR-W Discrepancy by Student Characteristics


Subgroup Overall Gender Female Male Race/Ethnicity American Indian Asian Black Hispanic Other White Best spoken language Another English and another English only

n

M

SD

t

p

140,919

6.16

60.58

38.15