Cowen Institute Erased Its Flawed VAM Report… But I Did Not.
The Cowen Institute at Tulane University has been promoting the New Orleans Charter Miracle since 2007. Cowen Institute has been trying since then to sell the “transformed” post-Katrina education system in New Orleans.
The results are tepid. Still, Cowen tries to sell this New Orleans. Consider this excerpt from Cowen’s history:
[Following Hurricane Katrina] the majority of schools reopened as charter schools, which are publicly-funded and operated by nonprofit organizations or universities, giving New Orleans a greater percentage of students in charter schools than any other district in the United States. Education entrepreneurs and veteran educators from around the country flocked to the city to participate in the greatest public school renaissance in the country. …
…the new model of delivering education to the city’s youth has begun to yield results. Parental involvement, teacher quality, and community engagement have all improved. Between the 2006-07 and the 2007-08 school years, student achievement rose for nearly every school in the city – and across all school types. Overall, the schools collectively saw a 15 percent increase in school performance scores from 2005-2008. Even so, New Orleans still ranks 65th out of 68 school districts in Louisiana, a state which has some of the lowest public school achievement levels in the country. While public schools in New Orleans are still performing at a level far below where they need to be, the improvements they have shown since Hurricane Katrina is very promising. New Orleans, once ranked as one of the worst school districts in the country, has the potential to develop a model for unprecedented innovation in public education. [Emphasis added.]
I’m sorry, but “65th out of 68 school districts” is hardly a “promising,” “innovative” “renaissance.”
Still, Cowen has been producing its reports with a mind to “chronicle the transformation of public education in New Orleans” in order that it might be “utilized by key stakeholders across the New Orleans area, the state, and the entire country.”
Remember, the New Orleans “miracle” was meant to be reproduced, allowing other “choice” districts to become 65th out of 68 in their states. (Tongue in cheek, of course.)
In its efforts to feature New Orleans charter success at “beating the odds,” Cowen took upon itself the task of VAMing Louisiana’s schools. VAM (value added modeling) is a statistical procedure that attempts to predict test score outcomes that (in this case) schools “should” achieve if they are to be considered “on target.”
It is no secret that VAM does not work. To read more about the problems with trying to use VAM to “predict” education “success,” read Dr. Audrey Amrein-Beardsley’s blog, Vamboozled!
For whatever reason– perhaps because VAM is test-driven-reform “vogue”– Cowen tried to use it, anyway.
According to Cowen’s supposed “VAM” report, released on October 1, 2014, a number of the New Orleans charters are doing a stellar job– if one considers “stellar” to mean surpassing so-called “VAM” predictions that Cowen created. Not exactly practical, real-world-translatable “success,” but instead, a theoretical hand-clap of, “Hey, look at us, we’ve improved!”
And yet, before the New Orleans Miracle celebration had time to die down, Cowen embarrassingly removed the VAM report from its website; ever so briefly stated that it was removed due to “flawed methodology” and “inaccurate conclusions,” and added, “The report will not be reissued.”
That must have been some error.
And indeed, it was.
Without getting too technical, what it looks like Cowen is calling VAM is nothing more than three basic multiple linear regression models fitted to state data to produce a “line of best fit” for the cloud of data points of each of the three regression models. Here’s the problem: In fitting a “best” line to a cloud of data points, some points must be above the line and some, below.
Therefore, those charters “exceeding” their prediction are nothing more than an artifact of the analysis.
This is not even VAM.
Cowen featured the report for over a week, during which time some charter sales notables sang about it as evidence that Choice Is Working. As the October 10, 2014, Times-Picayune reports:
Before the retraction, the report had leaped to prominence in some New Orleans education circles, touted by everyone from state Recovery School District Superintendent Patrick Dobard to Leslie Jacobs, a former member of the Louisiana Board of Elementary and Secondary Education and a driving force behind the state’s education reforms of the past 20 years. It also was highlighted via social media by leaders of the charter management organization Collegiate Academies, which runs Carver Prep and Carver Collegiate, and of the nonprofit charter support group New Schools for New Orleans.
(Leslie Jacobs’ celebratory response can be found here.)
Alas, however, the celebration is over. All guests have been asked to leave, and Cowen has removed its report.
But I still have it, cited above.
I think it might make an interesting case study for a number of researchers to whom Louisiana Superintendent John White and his department of education deny access to data.
And, of course, Amrein-Beardsley might like to give it an expert once-over.
It seems such a shame to just erase it… you know, like the pro-“choice” folks are trying to do with traditional public education….
Schneider is also author of the ed reform whistleblower, A Chronicle of Echoes: Who’s Who In the Implosion of American Public Education