In Ravitch’s Defense: Milwaukee Voucher Study Found Wanting

I hold a Ph.D. in applied statistics and research methods from the University of Northern Colorado (2002).  The title of my dissertation is A Monte Carlo Study of the Type I Error and Power Associated with Descriptive Discriminant Analysis as a MANOVA Post Hoc Procedure.

Why am I beginning this post with information about my credentials?

Yesterday, Dr. Patrick Wolf of the University of Arkansas’ Department of Educational Reform (which is heavily funded by an influential Arkansas family bent upon educational reform– guess who?) published a scathing attack on my colleague Diane Ravitch, accusing her of venturing academically “beyond her element,” so to speak, in her commentary of his work. Wolf states that

…commentators such as Ravitch spend their time and energy attacking me as a person because that demonstrates that they don’t have the ability to critique the methodological rigor and quality of my actual research.  [Emphasis added.]

Let me first comment that Ravitch in no way attacked Wolf  “as a person.”  Here is an excerpt from the recent Ravitch post, to which Wolf links:

The “independent evaluator” of the Milwaukee and D.C. voucher programs is Patrick J. Wolf of the University of Arkansas. As we learned during school choice week earlier this year, Wolf is a strong supporter of school choice and he even wrote an editorial saying that his home state of Minnesota needs more school choice because it was in danger of falling behind Arkansas in doing so. How much more independent can an evaluator be? It is perhaps also noteworthy that the University of Arkansas is generously funded by Arkansas’s biggest philanthropy, the Walton Foundation, which pours millions every year into charters and vouchers and anything that has the possibility of undermining public schools. [Emphasis added.]

Whereas she does not personally attack Wolf, Ravich certainly clearly exposes Wolf’s conflict of interest in evaluating a program obviously supported by his funders.

I agree with Ravitch that this conflict of interest is noteworthy for its undeniable potential in “shaping” study reporting and outcomes.

Wolf also appears to be fixated on Ravitch’s holding a doctorate outside of research and statistics.  As one who does hold one of those doctorates in statistics and research, let me underscore that I find Wolf’s arrogance to be an unpalatable professional embarrassment and that I do not share his foolishness in assuming that one must hold a research and stats degree in order to assess the limitations of his work. That said, I want to assure him that I do professionally possess “the ability to critique the methodological rigor and quality” (or lack thereof) of his research as per his imbecilic stipulation that Ravitch does not.  In addition, let me add that I am receiving no questionable funding from any foundation to conduct this examination.

I don’t even shop at Wal Mart.

Now, to consider Wolf’s (et al.) research:

In Wolf’s tasteless post, he alludes to this, his study with other researchers, Student Attainment and the Milwaukee Parental Choice Program: A Final Follow-Up Analysis, which he claims possesses “methodological rigor and quality.”  Ravitch notes (rightly so) that a serious limitation of this study is the attrition rate of students initially receiving vouchers in 2006:

The Wolf evaluations claim an advantage for voucher students in graduation rates. But consider this. In Milwaukee, according to this analysis (see the summary here) of Wolf’s evaluation, 75% of the students who started in a voucher school left before graduation. So of the 25% who persisted, the graduation rate was higher than the Milwaukee public schools. But what about the 75% who dropped out and/or returned to MPS? No one knows.

The summary to which Ravitch alludes is the work of Casey Cobb of the University of Connecticut, published by the National Educational Policy Center. In it, Cobb’s concerns regarding the attrition rate of Wolf’s study are noted:

By 12^th grade, [Cobb] notes, roughly three out of four of the original 801 MPCP 9^th graders were no longer enrolled in a participating private school. The sample attrition “severely clouded” the inferences that could be legitimately drawn about MPCP [Milwaukee Parent Choice Program]’s real impact on graduation rates. [Emphasis added.]

Wolf attacked both Ravitch and NEPC for reporting the 75% attrition rate of MCPC students. The version of the study currently available reports a 56% attrition, which is still high. (This information is not available until the close of the study, an issue I will address shortly.) However, according to NEPC’s rebuttal published today, written by NEPC Director Kevin Welner, a 75% attrition rate was the statistic from the original report by Wolf and his colleagues:

Yesterday, after this was posted, I received an email from one of the EdNext readers, pointing me to Wolf’s critique. I immediately went to page 16 of Wolf’s report. Could we have made such a mistake?!  Actually … we didn’t. Here’s what it said on page 16: “A second caveat is that the majority of students (approximately 75 percent) who were enrolled in 9th grade in MPCP were not enrolled there by the time they reached 12th grade.”

So I followed the link in the Education Next piece and downloaded the same report. Here’s what it says on page 16: “A second caveat is that the majority of students (approximately 56 percent) who were enrolled in 9th grade in MPCP were not enrolled there by the time they reached 12th grade.”

That was certainly odd. Then on third page of the pdf I’d just downloaded, I found the following: “Updated and Corrected March 8, 2012.” It doesn’t say what specifically was updated or corrected, but clearly one change was on page 16.

So here’s the timeline:
1. February 2012: Wolf and his colleagues publishes the SCDP [School Choice Demonstration Project] report, stating that “approximately 75 percent” of the voucher students  enrolled in 9th grade “were not enrolled there by the time they reached 12th grade.” (On February 24th, NEPC sent the report to Prof. Cobb for a review.)

2. March 8, 2012: The SCDP changes that sentence, substituting “56” for “75”.

3. April 19, 2012: NEPC publishes the Cobb review, pointing to (among other things) the 75% figure as evidence of the study’s limitations. Nobody had thought to go back and see whether Wolf or his colleagues had changed important numbers in the SCDP report.

4. April 1, 2013: Wolf attacks Diane Ravitch and NEPC for CORRECTLY quoting Wolf’s own report.

Here is an elephant-in-the-room question: Did Wolf and/or his colleagues knowingly and intentionally alter the original 75% attrition figure to the more acceptable (though still high) 56% in order to make the voucher student sample in the study appear more solid that it actually was?

As a former assistant editor of research for the flagship professional counseling journal, the Journal of Mental Health Counseling, I can tell you that I would not have approved a piece for publication had the researchers drastically and positively altered the study’s attrition rate without also having requested to see the data that support the accuracy of the change.

As I read Wolf’s tacky lashing of Ravitch, I noticed a single comment at the end of the post:

Dr. Patrick, Please hurry and de-identify the data you used in your papers and provide it to independent researchers. I have the ability to critique the methodological rigor and quality of your actual research. I am very very much looking forward to it.

Sincerely,

Julian Vasquez Heilig, Ph.D.

Regarding Dr. Heilig’s request: It is proper research etiquette to make one’s data available to fellow researchers wishing to verify the results of a study. The “de-identifying” piece means that identifying information for study participants be removed for privacy purposes before passing data on to other researchers.

Assuming the data set is released in whole (as it should be), what that data set should tell is the exact nature and degree of the sample attrition for the voucher students.

In short, the data would answer the question as to whether or not the researchers are being honest concerning the voucher student attrition rate. It would also reveal when voucher students are leaving. (For example, are a number of voucher students returning to public school within the first year of receiving vouchers? Within two years?)

Wolf offers no such information as part of his study.

As Ravitch correctly laments, “No one knows.”

No study is better than its data.  Furthermore, no study is better than the integrity and suitability of its sample.  Wolf can puff up all he likes, but his (et al.) sample is the fatal flaw in his (et al.) research. The premise of his (et al.) sampling– the idea that it is sufficient to separate two groups of students based upon their initial enrollments as either voucher or traditional public school attendees– ignore changes of these  students from voucher to traditional public and vice-versa over the next several years– then pretend that outcomes measured four years later are the direct result of a single choice made several years earlier but not adhered to by most participants (voucher or nonvoucher)– yields nothing useful. Nothing.

Here is what Wolf and his fellow researchers offer regarding the sample of voucher students in his study:

The 801 MPCP students are the entire 9th grade cohort of students who we determined to be valid voucher-using students after examining the Wisconsin Department of Public Instruction audited list of voucher recipients based on the 3rd Friday count (September 15, 2006). [Emphasis added.]

One must carefully consider what Wolf et al.are saying here, otherwise one will miss it. “The entire 9th grade cohort” means all students who began as voucher students but who did not necessarily finish. Proper sample reporting should include information regarding the sample attrition, at least a table showing how many voucher students continued as voucher students from year to year. It is arguably misleading to omit clear, detailed information regarding sample attrition in a longitudinal study. And it is not that I believe Wolf and his colleagues do not know this. I am sure they do.

The single statement regarding sample attrition at the end of the study (75% or 56%, depending upon which version of the study one reads), is shoddy research reporting. Wolf et al. follow this statement of attrition by writing,

The results of this paper as a whole should therefore be interpreted as the effect of “exposure” to the MPCP rather than long-term persistence in that sector.

Voucher “exposure”? Are you kidding me? What kind of foolishness is this? It is not as though “exposure” to vouchers leaves some established and undeniable change, as would exposure to asbestos or radium or ebola.  So, if students are choosing to forsake school choice for the traditional public school classroom, one way to excuse this attrition is to say that vouchers are so powerful that initially having a voucher supersedes continuing with the voucher.

“Exposure” to vouchers is a convenient focus for a study with 75% (or 56%) voucher student attrition.

The two principal samples compared in the Wolf et al. study are 1) 801 students who were given vouchers in 9th grade (2006) but most of whom chose not to continue with the vouchers, and 2) 801 students who were not part of the voucher program in 9th grade (2006) and who attended traditional public school.

A further confounding sampling issue is the possibility that those 801 in the traditional-public-school-in-9th-grade-(2006) sample could have opted for a voucher in subsequent years.  Since Wolf does not include detailed information regarding voucher use on the study’s sample, readers simply cannot know the degree to which these two purportedly separate samples are actually similar via voucher use at some point in the high school career.

Perhaps Dr. Heilig will be able to tell us once Wolf releases the data set.

If the two supposedly-mutually-exclusive samples in this study are in serious question– not the actual individuals in each sample but the mutual exclusivity of the school choice experiences of those in the two samples– how is it possible to “compare samples?”  It is not.

To recap:  Students in the “voucher” sample initially received vouchers, but most (56%? 75%?) did not continue using vouchers; students initially not using vouchers but attending traditional public school could have, at a later time, opted for vouchers. Therefore, the two samples are confounded in that both have the potential for some combination of student voucher use and voucher nonuse (i.e., public school preference).

I must say, this convoluted sampling kills any utility that this study might have otherwise offered.

Why not simply compare “voucher completers” (students who accepted vouchers in 9th grade and used them consistently through 12 grade) with “voucher noncompleters” and/or “voucher nonusers”?

(These possible samples noted above qualify as mutually exclusive. For example, one cannot be classed as a voucher completer and also have some degree of voucher nonuse. Or, one cannot be classed as a voucher noncompleter but possess some degree of voucher completion.  This makes for clean sampling, an indispensable condition for research rigor and quality.)

Was the number of voucher completers too low? Was it an embarrassingly low number?

No one knows.

Was there some effort to compare voucher completers to voucher noncompleters, but the idea was perhaps nixed because publicizing the concept of “voucher noncompleters” reflects poorly on those pro-voucher folks who are funding this research?

No one knows.

I had planned to examine specific results of this study by Wolf and his colleagues. However, I know that sampling issues previously discussed render results useless.  I would like to highlight a couple of the points Cobb raises in his review, as I believe the poorly conceived, poorly constructed, and inadequately detailed samples inevitably contributed to his assertions:

Summary statements found in the report’s executive summary and conclusion, while not inaccurate, do invite conclusions about MPCP effects that are likely not warranted by the data presented. For example, the report concludes that the “results here suggest that students who used a voucher to attend private school in 8th or 9th  grade were more likely to graduate high school” (p. 16). The problem is that we do not know exactly where they graduated high school or for how long they were enrolled in a voucher program school. This one caveat alone calls into question the usefulness of nearly the entire study. …

A significant point to note about this report is that there really aren’t many, if any, differences to report here. Even if there were, the research design is not robust enough to inform the reader about the causal effects of a voucher program. [Emphasis added.]

In other words, the researchers keep readers in the dark regarding voucher attrition rates, and crediting graduation rates several years after the initial acceptance of a voucher without requiring voucher school follow-through is a stretch.

My conclusions:  Wolf and his colleagues present nothing in this study to justify disruption of a school system via voucher use.  The one distinguishing factor of voucher use appears to be the high attrition rates– that is, the high rate of student return to the traditional public schools.

Perhaps the “life altering impact” of that initial voucher receipt in 9th grade is the realization that traditional public school aren’t so bad, after all.

I get to write as much since no one is funding me.

Can’t wait until Dr. Heilig is able to verify my summations using that actual Wolf-provided data set.

I’m also looking forward to reading that public apology I am sure Wolf will send to Diane Ravitch since he now knows that she is not angry.

She is simply correct.