50 Schools in 2012-13: A New Orleans Charter School Study of Limited Use
In January 2018, the US Department of Education (USDOE) Institute of Educational Sciences (IES) published a study entitled, “An Exploratory Analysis of Features of New Orleans Charter Schools Associated with Student Achievement Growth.”
The study’s principal author is Patrick Wolf of the Walton-funded Department of Education Reform at the University of Arkansas. (Note: The department web page does not mention the Waltons by name in association with their $10-million underpinning.)
In the Wolf et al. study, the major finding is an assortment of variables associated with higher test scores as captured by school-level value-added-modeling (VAM) scores from the 2012-13 school year for 50 charter schools. From the study’s summary:
Six features emerged from this study as potential indicators of New Orleans charter school effectiveness, particularly for school-level measures of student achievement growth in English language arts. School-level student achievement growth was positively associated with having kindergarten as an entry grade, an extended school year, and more experienced teachers. It was negatively associated with having a higher percentage of teachers with a graduate degree, a higher student–teacher ratio, and more student supports offered.
This exploratory descriptive analysis is limited in important ways. Because it is nonexperimental, the associations identified should not be interpreted as causal. Because the study had to be conducted at the school level and only 50 schools were in the potential indicator analysis sample, the study had a limited ability to identify systematic associations between school organizational, operational, and instructional features and school-level value-added measures of student achievement growth.
A major problem with this Wolf et al. study is that the findings could not be replicated using data from the 2013-14 school year. From the study:
The features of New Orleans charter schools in 2012/13 had no statistically significant associations with student achievement growth in 2013/14. This could be due to the original findings being false discoveries of associations by mere chance, misalignment of 2012/13 measures of the potential indicator variables with the 2013/14 student achievement growth outcome data, or less precision in the 2013/14 value-added measures of student achievement growth. The analysis could not distinguish among these three possible explanations for the nonrobustness of the findings across outcome years.
Specifically, the regression coefficients for the eight cases of associations that were statistically significant for the main analysis on the 2012/13 outcome data were all smaller when the analysis was repeated on the 2013/14 outcome data, consistent with both false discoveries and misalignment as possible explanations for the differences. The standard errors of the regression estimates from the repeat of the analysis on the 2013/14 outcome data were larger than those from the main analysis on the 2012/13 outcome data in every case except for one, consistent with the possible explanation that less precise student achievement growth measures in 2013/14 explain the discrepancy.
Wolf et al. did include a detailed limitations section, which is excerpted below:
The data for this study were limited to the 2012/13 and 2013/14 school years, with the main analysis focused on the 2012/13 school year, when the potential indicator measures and student achievement outcome measures were aligned. It is possible that some of the findings were particular to conditions that students experienced that year and might not hold in other years. A longitudinal study with multiple years of data on student achievement growth outcomes and indicator data aligned with those outcomes would produce more reliable estimates of potential indicators of charter school effectiveness. Such a study was not possible in this case, so readers are cautioned not to draw firm conclusions from these findings. …
The student achievement growth that informed the potential indicator analysis excluded certain important student subgroups. Students in grades K–3 and 9–12 in 2013, students who changed schools in 2013, and students with disabilities who took an alternative assessment in place of the standard state test were not represented in the baseline and outcome tests that generated the school-level student achievement growth calculations. The results of the potential indicator portion of this study apply directly only to the population of stable New Orleans charter school students in a regular education program in grades 4–8 in 2012/13.
The study is limited to the New Orleans open-enrollment charter schools with 2012/13 student achievement growth scores generated from 2011/12 and 2012/13 accountability testing. Schools that did not operate in both 2011/12 and 2012/13, because they were new or because they closed in 2012/13, were excluded from the sample, as were schools that required an academic entrance exam and schools that enrolled too few students in tested grades to report school-level scores without violating privacy laws. These schools could not be included in the analytic sample because their selective nature would have made them too different from the other charter schools to include (in the case of schools with entrance exams) or because there were no data for them (in the case of schools that enrolled too few students to report scores). Thus, the results of this study do not necessary apply to these types of New Orleans charter schools. …
In the final paragraph of the study, Wolf et al. identify what they consider “the most important function” of the findings– namely, that their study has somehow assisted researchers conducting future studies regarding variables to focus on for future research. However, given that the current study fell flat in replicating findings from 2012-13 data to 2013-14 data, it seems that such advice (reproduced below) seems optimistic:
Given the limitations of this exploratory study, practitioners are cautioned to consider the findings as only a first step in identifying potential indicators of school effectiveness. The most important function of these findings is to signal to researchers which features of public charter schools should be evaluated more rigorously. Should follow-up studies confirm that certain school features are causes of public charter effectiveness rather than merely correlates, practitioners would then be well advised to adopt those organizational, operational, or instructional features in their schools.
It is curious that in their study summary, Wolf et al. do not mention the lack of statistically significant association between findings from the 2012-13 school-level VAM data and 2013-14 school-level VAM data. In fact, the 2013-14 analyses are not even mentioned, which arguably misleads readers.
Here’s the full study summary from the published study:
This exploratory study examined the associations of certain organizational, operational, and instructional features (“potential indicators”) of open-enrollment public charter schools in New Orleans, Louisiana, with school-level value-added measures of student achievement growth in the 2012/13 school year. The achievement domains included were English language arts, math, and science.
Of the 85 public schools in the city during the 2012/13 school year, 75 were charter schools, supporting 92 different campuses and enrolling more than 84 percent of public school students in New Orleans. This predominance of charter schools makes New Orleans a particularly promising location for exploring the potential indicators of charter school effectiveness.
Six features emerged from this study as potential indicators of New Orleans charter school effectiveness, particularly for school-level measures of student achievement growth in English language arts. School-level student achievement growth was positively associated with having kindergarten as an entry grade, an extended school year, and more experienced teachers. It was negatively associated with having a higher percentage of teachers with a graduate degree, a higher student–teacher ratio, and more student supports offered.
This exploratory descriptive analysis is limited in important ways. Because it is nonexperimental, the associations identified should not be interpreted as causal. Because the study had to be conducted at the school level and only 50 schools were in the potential indicator analysis sample, the study had a limited ability to identify systematic associations between school organizational, operational, and instructional features and school-level value-added measures of student achievement growth.
Despite these limitations, the exploratory results are a first step in identifying charter school conditions that could be studied through more rigorous research to determine whether they are valid indicators of student success.
And now, a reminder about how the 2012-13 findings were not confirmed using 2013-14 data:
The features of New Orleans charter schools in 2012/13 had no statistically significant associations with student achievement growth in 2013/14. This could be due to the original findings being false discoveries of associations by mere chance, misalignment of 2012/13 measures of the potential indicator variables with the 2013/14 student achievement growth outcome data, or less precision in the 2013/14 value-added measures of student achievement growth.
That noted, the results of the additional analyses using 2013-14 data did make it into the study description on the IES “projects” web page. However, that summary is also misleading in that it reads like only the 3 math and science results could not be confirmed using 2013-14 data (as opposed to all 8 significant findings, 5 of which were associated with ELA VAM):
In the wake of Hurricane Katrina, the number of charter schools in New Orleans has rapidly expanded. During the 2012/13 school year—the period covered by this study, of the 85 public schools in New Orleans, 75 were chartered, enrolling more than 84 percent of all public school students in the city in 92 different school campuses. This study explored organizational, operational, and instructional features of New Orleans charter schools serving grades 3–8 that are potential indicators of student achievement growth in English language arts (ELA), math, and science.
The organizational characteristic of kindergarten provided as an entry grade was associated with higher levels of VAM on the ELA test. The operational characteristic of an extended school year also was associated with higher levels of ELA VAM. The instructional characteristics of a lower percentage of teachers with graduate degrees, more experienced teachers, and a lower student/teacher ratio were associated with higher levels of ELA VAM.
The analysis revealed fewer potential key indicators of charter school effectiveness regarding VAM in math and science. The inclusion of kindergarten as an entry grade was the only school feature that was statistically significant in its association with math VAM; schools with kindergarten were correlated with higher math VAM scores.
Having a lower student/teacher ratio and fewer staff in student support roles were the only school features that were statistically significant in their association with higher science VAM scores.
None of these associations between potential key indicators and math and science VAM scores remained statistically significant when estimated using 2013/14 outcome data, indicating that the results are not robust to such an additional analysis.
Offering kindergarten as an entry grade and having a lower teacher/student ratio were the only potential key indicators with statistically significant associations with more than one VAM outcome. Having kindergarten as an entry grade was positively associated with ELA and math VAM. Having a lower teacher/student ratio was associated with higher ELA and science VAM. [Paragraph breaks added.]
Wolf et al. need to include the results of their 2013-14 data analysis in both the study summary and study description and include it clearly and correctly.
As for the utility of the study outcomes: highly limited. It seems that the results cannot reasonably extend beyond the 50 New Orleans charter schools as those schools were in 2012-13.
__________________________________________________________________________________________
Want to read about the history of charter schools and vouchers?
School Choice: The End of Public Education?
(Click image to enlarge)
Schneider is a southern Louisiana native, career teacher, trained researcher, and author of two other books: A Chronicle of Echoes: Who’s Who In the Implosion of American Public Education and Common Core Dilemma: Who Owns Our Schools?. You should buy these books. They’re great. No, really.
deutsch29: Mercedes Schneider's Blog 
From your description, I wonder that they even bothered to publish the study other than to admit that they learned nothing significant.
publishing a non-useful study which, by its very effort, proves that test-score “accountability” is not about student success but about continuing the game of making money from producing and manipulating statistics.