A Brief Tutorial on Reading “Miracle” Graphs
On August 8, 2013, Louisiana Association of Public Charter Schools (LAPCS) founder and director Caroline Roemer Shirley sent out an email announcement regarding the supposed superiority of the state-run RSD charter schools over non-charters. She even states that her child will attend a charter this year (no specifics provided). Roemer Shirley is the sister of Louisiana Board of Elementary and Secondary Education (BESE) President Chas Roemer, who is helping his sister by promoting charters. (Seems like an ethical issue, but not in Louisiana, where these conflicts of interest are brushed aside.)
The truth about the failure of RSD charters according to the state’s own criteria for such “failure” is no selling point at all. That necessitates some creative graphic constructions to “prove” that “the reforms are working.” Thus, as part of Roemer Shirley’s letter (available here: Orleans Goes Back to School), she includes a graphic designed to manipulate an unsuspecting public into buying the lie that RSD charters are indeed miraculous. Here is the graphic (click on it to make it larger):
A yellow bar represents the elementary and middle schools that would have been in RSD had RSD taken control of these schools prior to Hurricane Katrina (the storm hit August 29, 2005); these are called “proxy” schools. A blue bar represents the RSD-NO (Recovery School District in New Orleans) elementary and middle schools in 2013. The measure is student Louisiana Educational Assessment Program (LEAP and iLEAP) scores for students who were present and took the test.
Roemer Shirley argues that since almost all RSD schools have been converted into charters, this “dramatic improvement in test results, more than a 95% increase,” is proof of charter superiority.
A colleague forwarded Roemer Shirley’s letter and graphic to me in an email, and as I was responding to the manipulations present in the graph, I thought it would make for a fine tutorial on reformer shaping of a miracle message using a graphic.
Here is my email response:
There are so many problems with this graphic, I don’t know where to begin.
I will focus on a couple of points. First, the RSD “proxy” is the schools absorbed by RSD post-Katrina but the data is from these schools pre-Katrina, which means the “study” does not account for the almost-total destruction of the school district. Comparing “before” and “after” is nonsense.
Second, the graph tries to present a great rise in scores. If the student base was different in 2006 (which it was) and these students scored better from the outset, then the years 2006 thru 2013 could manifest no improvement–that is, the scores could have flatlined or even decreased, and one does not get to know as much from the graph.
Third, these supposedly marvelous gains are over an eight-year period— which makes them not so marvelous after all. And that assumes that the gain was steady, which I question in the point above.
Fourth, any Louisiana teacher required to formulate a Student Learning Target (SLT) likely would not have been allowed to set such low goals to prove his/her “effectiveness.”
Finally, the corresponding RSD letter grades are heavily contingent upon iLEAP and LEAP scores, and the 2012 school letter grades for RSD are beyond unimpressive.
So, clear out the smoke and put away the mirrors. And let Roemer Shirley name the exact charter her child supposedly attends. Let’s see if it isn’t one of those cream-skimming schools.
–M [Emphasis added.]
Two additional observations:
In her discussion of RSD charter superiority, Roemer Shirley does not separate RSD charters from RSD direct-run schools. However, the 2012 school performance scores and letter grades have both types of schools as predominately “failing.” This is a comprehensive file of Louisiana school performance scores and letter grades for schools that were issued such (some schools are mysteriously missing from this spreadsheet): 2012SPSbySchool
If Roemer Shirley believes that her charters are superior, why not compare 2013 iLEAP and LEAP scores for RSD elementary and middle school charters to RSD elementary and middle direct-run schools and offer a miracle graph on that?
Reformer avoidance of the obvious means that the obvious is exactly that–obvious.
Also, Roemer Shirley makes no mention of RSD high schools. This allows her to steer clear of any discussion regarding state-run RSD graduation rates and the number play necessary to concoct a “miracle” on that front.
If it is easy for reformers to lie using statistics, it is child’s play for them to lie using graphs. Thus, it is important for the public to become adept at identifying such illusion.
Caroline Roemer Shirley founded LAPCS, and she is clear regarding its purpose:
The Louisiana Association of Public Charter Schools supports, promotes, and advocates for the Louisiana charter school movement….
If it takes lying using graphics to promote her cause, Roemer Shirley is apparently willing to do so. Anything to promote charter success. Her actions remind me of Tony Bennett’s now-famous emails regarding the Indiana charter school Christel House. On learning that the school would likely receive a B, Bennett responded, “That will be problematic because we told Christel… their data made them an A (before the grade was calculated).” And “They (those calculating the school performance scores and letter grades) need to understand that anything less than an A for Christel compromises all of our accountability work….”
The charters just have to shine. Whether or not they really do is apparently irrelevant.