Jun 12, 2003 - The conference call transcribed below was convened in part to respond ... the Peterson studies, in my view, was very clear and unambiguous.
Economic Policy Institute 1660 L STREET, NW • SUITE 1200 • WASHINGTON, DC 20036 • 202/775-8810 • FAX 202/775-0819 www.epinet.org
A CONVERSATION ON SCHOOL VOUCHERS A conference call convened by the Economic Policy Institute June 12, 2003 PARTICIPANTS:
Anne Heald, Economic Policy Institute, moderator Helen Ladd, Duke University Martin Carnoy, Stanford University Alan Krueger, Princeton University Cecilia Rouse, Princeton University Richard Rothstein, Columbia Teachers College John Mooney, Newark Star-Ledger, journalist Sam Dillon, New York Times, journalist Mike Rose, American Educator, journalist
INTRODUCTION The following transcript provides a window into a discussion that took place in a conference call on June 12, 2003. The participants were five nationally known and respected education researchers from Columbia, Duke, Princeton, and Stanford Universities, who spoke with journalists on what policy significance, if any, can be derived from the body of research on school vouchers. Each of these researchers concludes that the best research available does not show school vouchers for poor children to be a promising strategy for attaining meaningful differences in student achievement. As several researchers concluded, on average, there is no significant difference in achievement between the children participating in the voucher programs and those who remain in public schools. The evidence also fails to show that vouchers have a significant competitive effect on public school districts in which parents have a choice of vouchers for their children. Private schools that accept vouchers do not force an increase of quality in public schools despite strong claims to the contrary. And, finally, these researchers collectively describe how a voucher program, such as the one in Milwaukee, can actually have a negative financial impact on public school coffers and effectively subsidize private schools.
–2–
The random assignment studies of the New York City voucher program conducted by Paul Peterson’s team from Harvard University and Mathematica Policy Research are considered by many to be the best-designed studies in the field. The original findings of improved academic achievement for African American students were described as large and significant by the Harvard researchers. Almost immediately, Mathematica urged caution in interpreting these results and followed good practice in scientific investigation by making the data from the New York experiment available to Alan Krueger, an independent researcher from Princeton University. Krueger and his Princeton colleague Pei Zhu reanalyzed the data and included previously excluded students (a large portion of the sample), such as kindergartners and some African Americans. Krueger and Zhu’s analysis yielded a finding that there was no significant difference in student achievement for any ethnic or racial group. The conference call transcribed below was convened in part to respond to Peterson and William Howell’s recent response to Krueger and Zhu’s findings. But more significantly, the researchers addressed the wider body of evidence on vouchers domestically and abroad with respect to student achievement, accountability, competition, and the impact on scarce public resources.
THE CONVERSATION ANNE HEALD: I want to thank you for joining in this call sponsored by the Economic Policy
Institute. We’ve convened this call today because there’s a lot of confusion out there about what effect, if any, vouchers are having on educational achievement. The people you’re going to hear from today are leading experts on this topic, and we’ll walk through what we know about this large landscape. We’ll begin with Helen Ladd, Professor of Public Policy at Duke University, followed by Martin Carnoy, Professor of Education and Economics at Stanford University. Next will be Alan Krueger, Professor of Economics for Public Affairs at Princeton University, and Cecilia Rouse, Professor of Economics and Public Affairs at the Woodrow Wilson School at Princeton University, followed by a wrap-up from Richard Rothstein, Professor at Columbia Teacher’s College, former National Education columnist for the New York Times, and research associate with the Economic Policy Institute. Helen, do you want to begin? HELEN LADD: I’d like to make two points. The first has to do with the lack of public
accountability in the private schools. If you think about the voucher debate, it’s largely about giving low-income families more choice over where their children go to school. The idea is that, with school vouchers, more low-income families will be able to choose private schools for their children. Now, my own view is that giving low-income families more choice is generally desirable. Students have different educational needs, and one size school clearly does not fit all students. But choice for families is not the only or the most important value in designing a K-12 education system. Parental choice needs to be balanced against other goals, such as improving overall student achievement and maintaining a fair system.
–3–
My research in the United States and elsewhere leads me to conclude that these other values are best preserved when publicly funded school choice for parents is limited to the public school system. Given the use of public funding, the public needs to be assured that the schools are publicly accountable—and that is clearly not the case with the private schools. My second point has to do with student achievement. Supporters of school vouchers typically believe that private schools are better than public schools in terms of raising student achievement and that, therefore, it would be unfair to deny low-income families access to private schools. But the belief “The belief that private that private schools, of the type that schools…are more effective than would be accessible to low-income public schools is not consistent students bearing vouchers, are more with the evidence.” – Helen Ladd effective than public schools, is not consistent with the evidence. The best studies of this, I believe, are the ones done by Paul Peterson and his colleagues on voucher experiments in New York City, Washington, D.C., and Dayton, Ohio. These studies are high quality, because they are based on true experiments in that eligible applicants were randomly assigned to receive a voucher or not. These are the types of studies that are routinely used in medical research and that many people believe should be used more frequently in the education field. The main result of the Peterson studies, in my view, was very clear and unambiguous. On average, students who use vouchers to switch to private schools achieve at no higher levels than those who remain in the public schools. So much for the view that the autonomy of private schools makes them more effective. Much of the discussion in the Peterson press conference that was held earlier today is about a subset of the students. Peterson and his co-authors focused their attention on African Americans, but I do not think they would disagree with my reading of the evidence, which is that the effects on achievement are nonexistent for the typical lowincome student who would participate in vouchers. These results are not at all surprising to me. First, the private schools that most lowincome students can go to are not the elite private schools. Second, some private schools are likely to be pretty good, but many are likely to be pretty poor. The basic result from the Peterson studies simply indicates that the typical low-income student who has been involved in any of these voucher experiments in the three cities studied is likely to attend a private school that is no better than the public school he or she otherwise would have attended. I want to build on Helen Ladd’s comments. I agree with Helen that these experiments in the three cities are really some of the best studies available, and the fact that they show these very, very small gains is interesting as an academic exercise. But we have to be very careful in terms of the policy conclusions we draw from them, particularly about the effect on African Americans.
MARTIN CARNOY:
–4–
So they’re interesting research projects, but they involve small numbers of students. We also have to recognize students attend existing private schools with track records. The evidence suggests that, when you scale up, like in Milwaukee or Chile--Chile has 37% of its K-8 students attending private subsidized schools, not including the elite private schools--the effect would be smaller. You have to draw in a lot of new private schools to go into business—just like in Milwaukee—to take advantage of these vouchers. And although we don’t know very much about Milwaukee because the private school students are not tested by the state, we do know something about schools there. They have high turnover rates, just like the public schools. And if I were a betting person, I “All in all, [the voucher policy] is not a would bet that when you do big-hit policy. In fact, it’s probably a nothe kinds of studies that the hit policy, and it’s not free….It’s a nice Peterson group did, the windfall for the private schools, and a gains for the student are not net loss to the public schools.” going to be any different – Martin Carnoy than in the public schools. In Chile, you can even divide [the data] into the kind of private schools, and in the ones that did come in later, the students do worse than students in the public schools. Even when you just correct for socioeconomic class, which is not really correcting totally for the selection process that goes on, students in public schools do better than those in these more commercial private schools that responded to vouchers. The scaling up is a big issue, so whatever gains are shown in New York City, as small as they are, they are even going to be more negligible and perhaps even negative if you try to scale up into a larger system. The second point I want to make is about a completely different argument, which is not really covered in the Peterson study, but it’s definitely being discussed out there, and that is the effect of the competition of private education on kids in public schools. So the idea here is, if you have voucher plans, this is going to put pressure on the public schools to do better, because they’re going to be losing kids to private schools, and therefore, there’s going to be more effort, and the kids in the public schools do better. And there are some studies, particularly by Jay Greene and Carolyn Hoxsby, that claim to show this. But in fact, there are other studies using similar data to those Greene and Hoxsby used that show no effect of competition. And this raises serious issues about whether there’s a competition effect. In Chile, we did a study that was pretty carefully done, and we could not find a competition effect except in one part of Santiago, in one particular area of Santiago, where public school students did better when there were more private schools available nearby. A study by Belfield and Levin that reviews all the studies on competition effects in the United States also suggests that at best the effects are very, very small. So all in all, it’s not a big-hit policy. In fact, it’s probably a no-hit policy, and it’s not free, because it costs something to do a voucher plan. Because over half the kids that
–5–
took vouchers in Milwaukee were already in private schools, or would have gone anyway apparently, Milwaukee actually lost public revenues. The net cost to the school district is negative when supplying pupils with a $5,500 voucher for attending private schools, because, previously, the private schools were doing without the voucher and were charging about $2,000 in tuition. It’s a nice windfall for the private schools, and a net loss to the public schools, and they had to raise real estate taxes in Milwaukee to cover that loss. Let me put the point that Helen Ladd made a little bit differently. The way I read the studies that have been done on these experiments is that there is no overall impact of offering vouchers to minority students as a whole. That includes Hispanic students and African American students.
ALAN KRUEGER:
Now, what has been emphasized from these experiments is that the black students benefited; the fact that the Hispanic students did not benefit or may have had small declines in their scores, hasn’t been given the same attention. So what intrigued me is to try to understand why that’s the case. If you look at the data, the black students and the Hispanic students in New York City were coming from precisely the same schools, so it doesn’t appear to be something about the schools that African American students are attending that is generating the gains that Peterson reported. So that’s what led me to look into these data further. And the more I dug into the data, the more I ended up questioning the conclusion that the African American students did in fact experience higher tests scores as a result of being offered a voucher. First, in Paul Peterson’s previous work, he threw out over 40% of the students because they were missing data on their test scores before the experiment began. With a randomized experiment, you don’t need to do that. The students in the treatment group and control group should be equivalent, on average, because random assignments would be expected to balance all of the characteristics of the students. There’s no reason to suspect that the students offered a voucher would have done any better or worse than those not offered a voucher, because random assignment would have created groups that, on average, were about the same. And this is a point that Paul Peterson has emphasized in his earlier work on Milwaukee. I’ll quote you from a paper by Jay Green and Paul Peterson where they wrote, “analysis of randomized experimental data does not require controls for background characteristics or test scores. Such controls are necessary only when one doubts that the experimental data are truly random.” And as I understand the experiment, there’s no reason to doubt that the data are random. If one does doubt it, then it’s also a problem for the students who have the baseline test scores. But it’s quite clear that Paul Peterson doesn’t doubt it either, because when he studies outcomes such as parental satisfaction, he included students with missing baseline test scores, and that analysis just relies on the random assignment for the identification.
–6–
So what I’ve done, together “What we found was that the effect of with Princeton graduate vouchers for African American students was student Pei Zhu, is to quite weak; indeed, it was very difficult to estimate exactly the same distinguish no impact at all from a small model that he estimated for impact.” – Alan Krueger parental satisfaction, simply comparing the average performance and average test scores of those students offered a voucher and those not offered a voucher, three years after the program began. What we found was that the effect of vouchers for African American students was quite weak; indeed, it was very difficult to distinguish no impact at all from a small impact. So the results are rather borderline when we make this simple change. And I think most researchers, when they analyze experimental data, begin just by comparing the average performance for those offered or not offered the treatment, or in this case a voucher. That’s how the medical research proceeds. And in fact, that’s what was proposed in the proposal that Mathematica and Paul Peterson submitted for funding. The other issue I looked into in some depth was, “How is race defined?”, because that definition seems critical to the result. It doesn’t seem to be the schools that the students are coming from that lead to the claimed gains for African American students, because they’re about the same set of schools. Instead, it looks like it has something to do with a particular pool of students who were picked out and identified as African American students. And what I found there, and you can’t tell this from reading the previously published reports, is that a rather idiosyncratic definition of race was used. The survey did not ask about the race of the student. Instead, what was done was to ask one question concerning the race and ethnicity of the parents. Now this is contrary to what the Office of Management and Budget (OMB) recommends for statistical purposes for collecting data on race. What OMB prefers—and the way the census is done and all other federally supported surveys for self-identified race—is to first ask about ethnicity, and then to recognize that race is separate from ethnicity and ask a separate question on race. So you’ll notice that government survey results will often say, “Hispanics may be of your race.” Likewise, African Americans may be of any ethnicity. But when the New York voucher data were collected, it was impossible to do this, because parents were asked to identify whether they were black/African American (nonHispanic), white (non-Hispanic), Puerto Rican, Dominican, etc., and other. Parents were told to select just one category, contrary to the OMB guidelines. And there are many students in the sample, I’m sure, whose parents checked Dominican for the race/ethnicity, but if they were given an opportunity to identify their race, they would have selected African American. That’s what we find in the census; many of those in New York who indicated that their ethnicity was Dominican also indicated that their race is African American.
–7–
So, what was done, and again you can’t tell this from the previously published work, was that they just assumed that the students had the same race/ethnicity that the mother checked off. That can be restraining, given that fathers have something to do with the racial and ethnic identity of children. So what we did was simply to broaden the sample to include both students whose father identified himself as African American (non-Hispanic) and some other cases where either the father or the mother checked “other” for race (because they thought they were asked to give responses they weren’t comfortable with) and then wrote in African American/Hispanic, or something else that indicated that they were African American or African American combined with a different ethnic group. And when we used this sample, we found that the effect of offering a voucher to African American students was really trivial. When people hear the claims that offering vouchers will narrow the black/white test score gap, I don’t think they have in mind the nonHispanic black/white test score gap, for those who have baseline test scores. I think they have in mind a broader population, more like the one we have considered. So what we tried to do was represent results for the broader sample. And that suggests to me that the results are not very robust, and Mathematica policy research reached the same conclusion. They came to an event that Princeton sponsored at the National Press Club (Paul Peterson was invited, but he declined) and David Meyers from Mathematica presented their own additional follow-up work. What he wrote and concluded is, “at the time of the second follow-up report, Mathematica and policy research cautioned readers about putting too much emphasis on the average impact for African American students in grades 1-4...The new evidence presented by Krueger and Pei Zhu suggests that one must remain cautious when interpreting the findings “My best guess is that offering a voucher for African Americans.” to elementary school students in New Even after reviewing Paul York City probably had no impact on the Peterson’s paper, which I African American student’s test received just yesterday, I achievement.” – Alan Krueger think one ought to remain cautious. The estimates that they present in their paper are from statistical models, which I think is quite inappropriate. Indeed, Mathematica has the same concerns. [Peterson and Howell] present two-stage least squares estimates where they try to make an estimate of the impact of switching to a private school and staying there for all three years; they produce statistical sampling errors, which are calculated inappropriately. This is a technical point, but it is important—they ignore the fact that 40% of the students in the sample have siblings that are also in the sample, so they’re not independent observations. . [Peterson and Howell] just compute OLS standard errors ignoring that, which causes them to exaggerate the precision of their estimates. So, then, after having made these basic statistical mistakes, [Peterson and Howell] maintain that overwhelmingly the estimates they looked at showed positive, significant effects for African American students. I don’t think that’s justified by their analysis. They ignore the most simple estimate, which is simply to compare the mean performance,
–8–
the mean test scores of the students offered a voucher, with the mean scores of those not offered a voucher. [Peterson and Howell] make the further claim that in my study, 30 of the 51 estimates indicate a statistically positive impact for the African American students. Well, of those 30 estimates that they’re pointing to, many of them were simply replicating mistaken estimates that they had presented previously. I don’t necessarily think those estimates are correct. In fact, some of them have serious errors. We discovered in the analysis that they defined some of the variables incorrectly, which Mathematica acknowledged. Actually, in Peterson’s new study, he uses the new corrected data that we pointed out. We also discovered that the sample weights were in error. So I think it’s a mistake to point to the estimates where we purposely used the wrong data to try to replicate their results to show that we were using the data the same way that they had used it, and to say that this supports their previous conclusion. So where this leads me is to the conclusion that the estimates for the African American students are not particularly robust. My best guess is that offering a voucher to elementary school students in New York City probably had no impact on the African American student’s test achievement. Alan Krueger, thank you very much. Now we’ll go to Cecilia Rouse, also from Princeton University.
HEALD:
I’m joining this call largely because Paul Peterson quoted or cited a sentence from a paper I wrote summarizing the literature on school vouchers and African American students and Hispanic students. I don’t have the exact quote, but what I said was that the literature appeared to suggest that Catholic schools generate improved test scores for African American students.
CECILIA ROUSE:
And I wanted to note first that the literature that I was referring to is all based on nonexperimental methodologies, and were state of the art when I wrote the paper. And, indeed, that “The evidence form New York is literature does suggest that there may be gains for not nearly as strong as has African American students in the inner city, not been previously characterized.” for suburban students. – Cecilia Rouse However, that literature has really been trumped by the voucher experiment that Paul Peterson has conducted. And part of the reason I wrote that sentence was that, at the time, it looked like there were positive effects for African American students in the New York City voucher experiment, and, at the time, as well in Washington, D.C. The Dayton results were never very strong. But since then, more recent evidence from Washington, D.C. suggests that maybe there aren’t effects for African American students. The re-analysis by Krueger and Pei Zhu really means that the evidence on whether Catholic schools are more effective for African American students is now in the same boat as all private schools—which is that we really don’t know.
–9–
My second point is that the results are fairly mixed. And my read of the Peterson studies versus the Krueger and Pei Zhu analysis of New York City is that the results basically are not very robust for decisions made by researchers, and we can argue for a long time about what’s the right way to code African American versus Hispanic. I think most people in the experimental world would agree that you don’t need to control for baseline test scores. In fact the Manpower Development Resource Corporation, when they conduct the experiments, they don’t even collect baseline data because it’s not necessary. So, most people would agree that you don’t need to control for baseline results. But clearly what you take away from this is that the evidence from New York is not nearly as strong as has been previously characterized. I’m going to just briefly summarize what the previous speakers have said and highlight the main points. Helen Ladd began by making two points. One was that the Peterson studies, and the whole controversy between Peterson and Krueger about the technicalities of these studies, don’t address the much more fundamental issue of whether private schools are accountable for the use of public money.
RICHARD ROTHSTEIN:
Ladd’s second point was that there’s a widespread belief that the private school students do better than public school students and that private schools are better than public schools. But in fact, the kind of private schools that vouchers access for inner city children are not the kind of schools that the image of private schools is based on. The image is based on typically elite private schools serving affluent middle class children. But none of the voucher proposals propose to allow inner city children to access those kinds of private schools, so the confusion is how the implications of these studies are being interpreted. Ladd also emphasized, as did all the speakers, the fact that the Peterson study, regardless of what technical disputes there may be about African American effects, concluded, and Peterson would agree, that on average, students using vouchers don’t have higher achievement than students who remained in public schools. I would add to what Helen Ladd says that, given that fact, it’s hard to understand what the implications might be of a policy that affected only one ethnic group. Would vouchers only be made available to African Americans, for example, if the Peterson results were correct? Martin Carnoy then spoke about the fact that, even if the Peterson results that African American students did better in private schools when they had vouchers were accurate, these are very, very small experiments, and they don’t take account of the problems and the challenges that would be faced if the vouchers were made more widespread, and new private schools had to be created to accommodate those students using vouchers. The experience is, Carnoy says, that when you scale up these experiments and create large numbers of private schools for voucher recipients, the achievement declines. In Milwaukee for example, although there’s no achievement data from the private schools, there are indications of low achievement from large-scale voucher experiments; for example, very high turnover rates in these schools. In Chile, where Carnoy has also studied the voucher system, when it scales up, students actually do worse in the private schools.
– 10 –
Carnoy also talked about the theory that even if students don’t do better in private schools when they get vouchers, the existence of vouchers forces the public schools to improve; that is what’s called the competition effect. And again, Carnoy finds that in Chile, where there is a large-scale voucher program, there is no competition effect, except in one very small geographic area. Finally, Carnoy made the point that there is a big cost to these voucher programs. Most people think that if you take the money that’s already being spent at public schools and give it to private schools, there’s no additional cost, because the children who are benefiting from money previously in public schools are now doing so in private schools. But in fact, when you look at these actual experiments that are being done, there is no way of implementing the experiments without the children who are already in private schools receiving vouchers. A large share of the money that’s going to these vouchers is going to children who are already in private school or to very young children whose families were going to send them to private school in any event, so it’s a net cost. And I would simply add to what Carnoy said, that this is even a stronger finding in the Cleveland1 voucher studies, that a very, very high “On average, students using proportion of those who vouchers don’t have higher received vouchers were achievement than students who already in private school, so remained in public schools.” there was a large net cost to the – Richard Rothstein public system. Alan Krueger next spoke about the two technical issues that he had investigated in the Peterson studies. He also emphasized that, not only is Helen Ladd correct that the Peterson studies find that there’s no impact, on average, for students who receive vouchers, but that there’s not even an average impact for minority students who receive vouchers. This reinforces the question that was raised before: How would you implement such a policy that benefited some children and harmed others? In any event, Alan Krueger pointed out two technical flaws in the Peterson studies. One was that Peterson threw out 40% of the observations because there were no baseline scores available. This violates the usual methodologies for conducting randomized controlled experiments, because if the experiment is truly random, then it would not need baseline scores, and so there is no justification for throwing out those observations. The second problem is that the Peterson study used a very idiosyncratic definition of race. And what Alan Krueger found was that, when you correct for both of those problems—by including not only African American students whose mothers were black, but also those whose fathers were black, and by including the 40 percent of the observations that Peterson threw out—you find not even a significant effect for African American students who were using vouchers in New York City.
The Cleveland voucher program is publicly funded by the state of Ohio and Indiana University by a group headed by Kim Metcalf. The New York City, Washington, D.C., and Dayton voucher programs and evaluations are privately funded.
1
– 11 –
And finally, Cecilia Rouse added that she had previously written that the literature seems to indicate that African American students do better in Catholic school, but she notes that that was a conclusion that she drew not from the experimental literature, and this new Peterson study certainly trumps the nonexperimental literature that existed before, and she is persuaded by Alan Krueger’s findings about the conclusion that African American students in fact did not do better in New York City voucher schools than in public schools.
Questions and answers As for scaling this up so that many more private or Catholic schools are enrolled into the program per se, one of the things I’ve been talking to Catholic schools about is that a lot of these schools are relying on the notion of vouchers as potentially saving some of them from closing, especially in urban areas where there have been closings of late, including in New Jersey but also in some of the larger cities throughout the country. I think a lot of people might agree that [these schools] do serve some function, whether improving test performance or not. It’s obviously arguable.
JOHN MOONEY:
And I’m wondering, Dr. Carnoy, have you looked at…what kind of impact [voucher programs] have had on existing Catholic schools in these experimental cities, whether this has helped save that sector, which seems to be on a slow road to demise? Do you have thoughts of that as a phenomenon within the Catholic schools that exist now? In Milwaukee it’s not only saving Catholic schools but also saving other religious schools. The tuition was much lower than the voucher is now. It’s not only saving the religious schools, it’s actually subsidizing them, because there’s more than they were subsidized before with just tax breaks.
CARNOY:
LADD:
I don’t have evidence from the United States on this, but it’s interesting sometimes to look at other countries. In the 1970s, New Zealand, which I’ve studied closely, looking at competition and choice, faced a similar problem with many of the Catholic schools about to go out of business. And the solution there, and I realize it’s a different country, because they pay less attention to the separation between church and state than we do here, but the solution there was to incorporate the Catholic schools into the public school system. And the advantage of doing it that way was the Catholic schools get the funding, but with that funding came accountability, so the Catholic schools had to meet all the same requirements as the public schools. I’d just like to add that in Milwaukee there is no accountability requirement for voucher schools, aside from some very minimal financial oversight. I’m not sure whether most religious schools would accept strong accountability requirements as a quid pro quo for the voucher.
CARNOY:
– 12 –
One follow-up and then Richard Rothstein mentioned this, and I may have missed it in what you said on Milwaukee, but are there determined numbers on how many kids using the vouchers in both Milwaukee and Cleveland were existing students and/or would have gone because there were siblings or the like? Are there actually some hard numbers on that, in both Cleveland and Milwaukee?
MOONEY:
Yes, I don’t have the hard numbers in front of me, but the person that did the studies in Cleveland was an Indiana University researcher named Kim Metcalf, and if you want to e-mail us, I can get you the references to his studies. There are hard numbers. They were very, very high.
ROTHSTEIN:
MOONEY:
More than a majority?
ROTHSTEIN:
Yes, yes.
In Milwaukee the best estimate is “In Milwaukee, there is no that, of the 12,000 voucher students in the accountability requirement for 2002-03 school year, 7,000 were either voucher schools, aside from some existing students or students who would very minimal financial oversight … have gone to private school anyway. And (the voucher) is not only saving the the way that that’s calculated is that the religious schools, it’s actually number of private school students in subsidizing them.” – Martin Carnoy Milwaukee rose from 20,000 to 25,000, while public school enrollment rose slightly in the past five years. It’s gone up by a couple of thousand students. So it looks like there are 5,000 net new private school students and the others are not a net gain to private voucher schools.
CARNOY:
ROTHSTEIN: One more thing. In the Cleveland studies, the Indiana researchers led by Kim
Metcalf, one of the things they included was that young children whose siblings were already in private school, private paying students, and they assumed that the younger children who were not yet in school who then took vouchers were children who would have gone to private school even without vouchers. MOONEY:
Which is dependent, right?
ROTHSTEIN:
Yes.
MOONEY:
Okay, thank you.
SAM DILLON: First of all, I’d like to direct my question at Cecilia Rouse. You seem to have
been cited today by default. I mean, you were cited in this press conference and in the report as…I guess they quoted you in order to put their…Peterson quoted you to push his research findings into the mainstream of research findings, I guess is the point. And you’re participating in this conference call, so I sort of see you as somebody that’s being cited or you’re sort of in the middle of this debate between Krueger and Peterson. You have said that, after analyzing these studies, you believe that Peterson’s studies are not robust, and that you sort of side with Krueger. Could you say again concisely why you drew that conclusion?
– 13 –
The way I really read these two studies is not that necessarily one side is right and one side is wrong. I have my own personal beliefs in how I would analyze the data, had I done so. But I think, especially if you read Peterson’s response and Krueger’s response to Peterson, they both have their arguments.
ROUSE:
And reasonable researchers will disagree on which way to go about doing things. Again, I might do things one way or another. But I think the most important thing that I take away is that, in a lot of data, no matter how you push at the data, you keep getting the same results. And in these data, making very reasonable but different decisions, you know, may generate a very different result. And it’s such an important topic. What I took away from it is that, if adding in additional observations makes a huge difference, the results just aren’t very strong. So what I really take away from it is that the results aren’t strong. It’s just not very robust. I think I’m partly responding to the fact that Peterson has been very vocal about how strong the results are, and the evidence was very persuasive before. LADD:
The other interesting thing about all of this is that there are studies from three cities, not just New York City. There’s New York City, Washington, D.C., and Dayton. Not all the Dayton results are in, but the Washington results are interesting on this issue of whether there are positive findings for African Americans. At the end of the second year of the Washington, D.C. voucher experiment it looked like there were large positive findings for African Americans. By the end of the third year, though, the effects had washed out completely. So what we have is no positive effects for African Americans in Washington, D.C., and the possibility—but the contested possibility—that there might be positive effects in New York City. To me, putting those two studies together raises serious questions about whether there are positive effects for African Americans. Helen, the Washington study you’re referring to was done by Howell and Peterson, wasn’t it?
HEALD:
LADD:
It was a group under Peterson’s guidance, and so it was unclear to me exactly who the authors are. The authors of the second-year study were Wolf and Howell, I believe. But these are all part of the Peterson set of studies. And the third-year results for Washington, D.C. and for New York are best captured in the Brookings Institution book by William Howell and Paul Peterson called The Education Gap. At the beginning of Peterson’s presentation, he did a very quick PowerPoint slide showing places that there are important voucher programs underway, and he listed Milwaukee, Cleveland, and Florida. Is that a good representation of the most important voucher experiments or programs?
DILLON:
LADD:
I don’t know why he and other supporters of vouchers in the United States leave out Chile. I realize it’s outside of the United States, but Chile has had almost a textbook voucher program for the last 20 years and that’s the study and country that Martin
– 14 –
Carnoy referred to earlier. He has done a lot of research on that voucher program, as have other people as well. DILLON:
Well that’s good. Milwaukee, Cleveland, Florida, and Chile.
I think the Florida program is a little bit different because, if I understand correctly, the state issues vouchers when schools fail.
CARNOY:
Two years in a row.
LADD:
Two years in a row. And so that’s a slightly different program than either the Cleveland or Milwaukee program, and it’s certainly different from the Chilean program.
CARNOY:
I was going to say that as well, and in fact, in Florida it’s part of the accountability system, and I’m part of a study that’s been following Florida for the past four years. In fact, what you see in Florida is that, because it’s part of the accountability system, there are incentives on the part of the state; they can manipulate who and what schools are considered failing; schools can get out of the program in a different way. Whereas in Cleveland and Milwaukee, it’s basically just based on the income level of the student. So it’s very different, and in fact in Florida, this is the first year that there are really some vouchers being issued.
ROUSE:
And let me add that in Chile, 100% of students who do not attend elite private schools are covered by vouchers. That means any student can get a voucher, and the vouchers are equal in amount whether you attend a public or private school. In Milwaukee, students are eligible for a voucher if their family earns up to 1.75 times the poverty income, and that makes approximately 80% of the students who live in Milwaukee eligible for the vouchers. So it’s almost a universal voucher program in Milwaukee.
CARNOY:
[NOTE: Due to a recording error, a small portion of conversation between Sam Dillon and Richard Rothstein was not picked up in the transcription process. Mr. Dillon reported that Paul Peterson said he had no preconceived notions about vouchers when he began the studies in 1997. But Mr. Rothstein replied by citing op-eds Mr. Peterson had written for the Wall Street Journal in the mid-1990s endorsing vouchers.] I had a question about the New York City voucher schools. I think the samples drop after the first year by about a third.
MIKE ROSE:
KRUEGER: ROSE:
That’s correct.
Any indication of who these kids are, and where they go, from either this research or other research that’s current? I was concerned about that attrition. The attrition in the New York experiment, by the way, was a lot lower than it was in Dayton, Ohio and Washington, D.C., which is one of the reasons why I think the New York study received more attention. And another reason is that it’s the only one where the data has been made available to people outside of the original research team.
KRUEGER:
– 15 –
I looked very closely at the impact of attrition, and from what I can tell, it doesn’t seem to be skewing the results. The students who have dropped out of the sample who couldn’t be tracked from the control group and the treatment group were, on average, about similar in terms of their test achievement. And I would also say tracking students in this population is very difficult. Mathematica made tremendous efforts, invested a great deal of money, trying to track them. And the attrition rate was actually lower the third year than it was in the second year, which is unusual. They were able to find some of them the second year. So, what I took away is that I think one ought to be concerned about attrition. It would be better if there was no loss of the sample, they weren’t dropping out over time. But I don’t think that’s necessarily skewing the results. LADD:
I’d like to add a slightly different angle on this. Alan Krueger has been talking about people dropping out of the sample. It’s also of interest to pay attention to who drops out of the private schools once they are in them. There is a good study by William Howell, one of Paul Peterson’s co-authors, that looks closely at the dynamic changes, the changes over time, in the New York voucher program. He discovers that African Americans are more likely than almost any other group that he looks at to drop out of the private schools that they started out attending as a result of the vouchers. Which actually raises another issue: the estimates that Peterson and Howell have emphasized from the Brookings book are estimates of the effect of attending a private school for three consecutive years, whether some attend one year, and some attend two years.
KRUEGER:
When voucher programs are targeted, students are not going to have a shot at the better schools taking them. If you have a general voucher, the better schools are going to go for the lower-cost student, meaning easier to educate students.
CARNOY:
And no matter how hard you try through lotteries or other means, and the Chilean case is a very good example of that, no matter what kind of rules you put on the game, the private schools figure out ways to get around those rules. The Chilean law said that you could not screen a student into your school if you accepted a voucher, and a study done in 1992, the year before the law changed, showed that 60% of private subsidized schools in Santiago actually tested their students and then interviewed the parents before they admitted a student. So it’s very hard to enforce those kind of laws, and you get all kinds of selection procedures going on. I want to thank some very distinguished participants on this call: Martin Carnoy, Helen Ladd, Alan Krueger, Cecilia Rouse, and Richard Rothstein. I thought this was a very rich and high-caliber discussion. Thank you very much for your time.
HEALD:
– 16 –
SUGGESTED READING Carnoy, Martin. 2001. School Vouchers: Examining the Evidence. Washington, D.C.: Economic Policy Institute. Krueger, Alan, and Pei Zhu. 2002. Another Look at the New York City School Voucher Experiment. Working Paper. Princeton University and NBER. March 2003. Ladd, Helen F. 2002. School vouchers: a critical view. Journal of Economic Perspectives. Vol. 16, No. 4. Metcalf, Kim K., et al. 2003. Evaluation of the Cleveland Scholarship and Tutoring Program, Summary Report, 1998-2001. Bloomington, IN: Indiana Center for Evaluation. Peterson, Paul E., William G. Howell, et al. 2002. The Education Gap: Vouchers and Urban Schools. Washington, D.C.: Brookings Institution Press. Paul E. Peterson and William G. Howell at Harvard University. June 2003. “Efficiency, Bias, and Classification Schemes: Estimating Private-School Impacts on Test Scores in the New York City Voucher Experiment.” Prepared for publication in The American Behavioral Scientist (forthcoming). Rothstein, Richard, Martin Carnoy, and Luis Benveniste. 1999. Can Public Schools Learn From Private Schools? Washington, D.C.: Co-published by the Economic Policy Institute and the Aspen Institute's Nonprofit Sector Research Fund. Rouse, Cecilia. 2000. “School reform in the 21st century: a look of the effect of class size and school vouchers on the academic achievement of minority students.” Princeton University, Working Paper No. 440. United States General Accounting Office. 2001. School Vouchers: Publicly Funded Programs in Cleveland and Milwaukee. Washington, D.C.: Government Printing Office.
