CREDO: Trying Hard to Shape Urban Charter Success from Shoddy Research
In December 2014, Margaret Raymond, founding director of the Center for Research on Education Outcomes, (CREDO), made the following remarks about charter performance at the City Club of Cleveland (Ohio):
I’ve studied competitive markets for much of my career. That’s my academic focus for my work. And it’s [education] the only industry/sector where the market mechanism just doesn’t work. I think it’s not helpful to expect parents to be the agents of quality assurance throughout the state.
As one might expect, these words coming from the CREDO director shocked some in attendance at this pro-charter-funded, Ohio event. As blogger and attendee Stephen Dyer reported:
Considering that the pro-market reform Thomas B. Fordham Foundation paid for this study and Raymond works at the Hoover Institution at Stanford — a free market bastion, I was frankly floored, as were most of the folks at my table.
Thus, it is with raised eyebrow that I learned of a new CREDO study released in spring 2015 and that supposedly demonstrates urban charter school superiority. The study also caught the attention of Baruch College researcher/journalist Andrea Gabor, who informed me of her painstaking invesrigation of the “new” CREDO study. As Gabor noted to me in an email on May 8, 2015:
…After years of researching schools on the ground in charter-heavy districts like New Orleans and New York City, I was skeptical about the [new CREDO] study’s approach and its findings. So, I hired Kaiser Fung, a respected statistician, to help me analyze the study. I then asked CREDO’s director, Macke (Margaret)Raymond, several questions via email; her answers raised more concerns. In the end, Fung and I found several major problems with the study; in the case of its analysis of New Orleans charter schools, the study even violated its own methodology. The problems go well beyond technical quibbles and suggest that any generalizations drawn from the study about the quality of traditional public schools relative to charter schools would be a big mistake.
On April 28, 2015, Gabor wrote a post regarding her findings, New CREDO Study, New Credibility Problems: from New Orleans to Boston, which I feature here, in part:
Last month, Stanford University’s Center for Research on Education Outcomes (CREDO) released a new study on urban charter schools, which purports to show, for the first time, that charters outperform city public schools, at least on standardized-test scores. If true, the study’s findings are a potential bombshell since, thus far, studies have shown no meaningful difference between charter and public schools.
The new study, Urban Charter School Study Report on 41 Regions claims to show that “urban charter schools in the aggregate provide significantly higher levels of annual growth in both math and reading compared to their TPS [public school] peers.”
The years I’ve spent researching schools on the ground in charter-heavy districts like New Orleans and New York City made me skeptical of such an outcome. But because I am not an expert in research methodology, I decided to hire a respected statistician, Kaiser Fung, author of Numbersense and an adjunct professor of statistics at New York University who has no connection to the education-reform movement (and thus no axe to grind), to help me analyze the CREDO study.
After combing through the study and its accompanying technical document, and after exchanging a series of emails with Macke Raymond, Director of CREDO, we found significant problems with the CREDO study. The problems go well beyond technical quibbles and suggest that any generalizations drawn from the study about the quality of traditional public schools relative to charter schools would be a big mistake. In particular, the study does a poor job of explaining the basis on which it includes or excludes charter- and public-school students; an email exchange with Raymond clarified the study’s methodology, but also revealed that it introduced, in many cases, an anti-public-school bias. And, in at least one case—the findings on New Orleans, the first all-charter district in the country—Raymond admits that CREDO violated its own methodology, a fact not disclosed in either the study or its accompanying technical documents.
Let’s begin with a brief description of the study itself. The study analyses data from 22 states, covering just over a million charter-school students, during the 2006/2007-to-2011/2012 school years. It seeks to measure charter-school performance in 41 urban areas against students who attend “feeder” public schools in the same urban areas. The study includes about 80 percent of the charter students in the areas under study (20 percent are excluded from the study because CREDO could not find any matching public-school students.)
The study also relies on a controversial methodology that the researchers used in past CREDO studies and that has been critiqued here and here and here. What’s important to know about the methodology is that it purports to compare each charter student in the study to a “virtual twin,” a composite of as many as seven public-school kids who attend “feeder” schools and who “match” the charter students on both demographics and test scores; the virtual twin is literally an averaged kid. The demographic criteria for creating each virtual twin includes: grade level, ethnicity, gender, Title 1 eligibility, special-education and English-language-learner status.
In this post, I will not revisit the problems referenced above with CREDO’s virtual-twin methodology; rather I will focus on three major problems with this study:
First, the study excludes public schools that do NOT send students to charters, thus introducing a bias against the best urban public schools, especially small public schools that may send few, if any, students to charters. The study implies that the “virtual twins” are drawn from the general population of traditional public schools—specifically that a school is considered to be a feeder if even a single student transferred during the study period. This is not the case. In our email exchange, Raymond explained that to qualify as a “feeder school” a public school must send at least five students to charter schools, a detail not revealed in the study. The study never explains that it uses this stricter, five-student-minimum criteria that public schools must meet to be included in the study. (Nor does the study explain why it didn’t look at all “neighboring” public schools with comparable/charter-like demographics—whether they send kids to charters or not.)
To test my theory, I contacted two of the better Title 1 middle schools in New York City whose demographics I knew would mirror those of local charter schools to see if they meet the criteria that would qualify them as charter “feeder schools.” Global Technology Preparatory, in East Harlem, where most kids are black or Latino, estimates that it has sent only three of its graduating eight graders to charter high schools over the last three years. West Side Collaborative, a ten-year-old school with similar demographics, hasn’t sent a single transfer-student or graduate to a charter school, according to Jeanne Rotunda, the recently retired founding principal. Both schools received an “A” and a “B” on the last two graded report cards from the New York City Department of Education, and are given high marks for quality from parents, students and teachers. Yet, although both GTP and West Side have charter-like demographics and are in an area rich with charter schools, they would not count as feeders and, therefore, their students wouldn’t be included among the virtual twins in the CREDO study.
This also raises several further questions: The study’s geographical filtering mechanism for determining which schools qualify as “feeders” isn’t disclosed, except for some qualitative description in the Technical Appendix. It would have been much more straight forward to rely on simple geographic or district demarcations. New York City, for example, neatly divides its schools into clearly defined neighborhoods, such as East Harlem North and East Harlem South etc., as well as distinct educational districts.
Global Tech and West Side Collaborative also highlight the ways in which CREDO’s matching criteria miss critical differences between public- and charter-school demographics. Urban public school students are often poorer, more likely to attend schools with large number of kids with special needs and English language learners than their charter-school counterparts. They are also likely to have parents who are less engaged, for a variety of reasons, than those in charter schools, which target the most engaged families via everything from lotteries to requiring that parents attend a set number of open houses before they can even enter lotteries. These distinctions are not addressed by the CREDO study.
Second, in the case of New Orleans, the study compares charter students to virtual twins who go to school, not in New Orleans, but anywhere in Louisiana—a clear violation of the study’s feeder-school criteria, and one that isn’t disclosed in the study.
The rest of Gabor’s post is not to be missed, including the results of her communications with Raymond about fudging the student matching criteria for New Orleans schools. To access Gabor’s full post, click here.
It appears that this “new” CREDO study is little more than an attempt to snow the public into believing in a market-driven superiority that Raymond herself has already publicly admitted doubting.
Schneider is a southern Louisiana native, career teacher, trained researcher, and author of the ed reform whistle blower, A Chronicle of Echoes: Who’s Who In the Implosion of American Public Education.
She also has her second book available on pre-order, Common Core Dilemma: Who Owns Our Schools?, due for publication June 12, 2015.