First letter to White/BESE Regarding Scoring Bias
My first letter to White and BESE, upon verifying scoring bias:
November 21, 2012
Mr. White and BESE Board Members:
I have been reading Bulletin 111, Louisiana School, District, and State Accountability System.
It is quite the Frankenstein of measurement—so many changes in instrumentation, weights, and grading scales—all impressively chaotic. In the end, one can use it to produce numbers, as you have done, and attach value labels to those numbers, as you have also done. However, just because you have created a bulletin with the force of law and declared that such is useful in determining school/district performance and growth does not mean that the resulting figures accomplish the purported goal.
The 2012 school performance scores do reveal that you have consistently captured a measurement concept: scoring bias. Using two of the DOE data spread sheets on the 2012 school performance data, I will demonstrate in three calculations how scoring is biased toward high/combination schools and against elementary/middle schools.
For the first calculation, I used the 2012 school performance score data available to the public on the LDOE website. I examined column K, Point Change 2011 to 2012 (this is the point change in baseline scores). Twelve elem/middle and 13 high/combination schools (approx. 2% of all schools) had no data available in column K.
I counted the number of elem/middle schools vs. the number of high/combination schools that showed a point increase of 10+. Twenty-two elem/middle schools showed such an increase; 168 high/combination schools showed the same increase. However, across the state in general, the elem/middle schools outnumber the high/combination schools approx 3 to 1. Therefore, if 168 high/combination schools showed a 10+ point increase, then in the absence of scoring bias, one would expect approx. 168 x 3 = 504 elem/middle schools to yield the same increase. What is evident in the actual 22 vs. the expected 504 is incredible scoring bias toward high schools.
When the criteria is narrowed to the number of schools increasing the SPS 15+ points from 2011 to 2012, only 2 elem/middle schools meet the criteria, and 86 high/combination schools do. If there were no bias, approximately 86 x 3 = 258 elem/middle schools would have met the 15+ criteria. Again, blatant bias.
Increasing the criteria to 20+, the same 2 elem/middle schools meet the gain; 42 high schools do. Without bias, approx. 42 x 3 = 129 elem/middle schools should have also met the criteria. Finally, schools with an increase of 25+ points in SPS from 2011 to 2012: one elem/middle school; 13 high/combination schools. In the absence of bias, 13 x 3 = 39 elem/middle schools should have met the 25+ criteria.
In the presence of scoring bias, you are not measuring performance or growth. You are measuring the flawed Bulletin 111.
For the second calculation, I also used the 2012 school performance score data available to the public on the LDOE website. Once again I examined column K, Point Change 2011 to 2012 (this is the point change in baseline scores) in order to calculate the number of elem/middle vs. high/combination schools with a growth score of lower than –2.5. In Bulletin 111, page 13, a “school in decline” is defined as one “having a declining SPS (more than –2.5 points).” If there were no bias against elem/middle schools, then one could expect approx. three times as many elem/middle schools as high/combination schools fitting the criteria set for “a school in decline” since the number of elem/middle school exceeds the number of high/combination schools by a ratio of 3 to 1. The ratio of elem/middle to high/combination schools scores decreasing by more than 2.5 SPS points is more than 5.5 to 1 (78 elem/middle schools to 14 high/combination schools). Thus, again, here is evidence that the 2012 SPS favor high/combination schools at the expense of elem/middle schools.
For the third calculation, I used the data sent to BESE members prior to the publicizing of the 2012 school performance scores, the data set including the column, Transitional Baseline Scores 2010-11. The idea of “transitioning” the score column directly related to the school letter grade scale is a psychometric blunder. The proper way to adjust tests in differing metrics is to calibrate them one to another, with newly-introduced calibrated to previously-established; if such calibration cannot yield unbiased results, then the proper course of action is to establish separate grading scales, one for elem/middle schools and one for high/combination schools, testing them prior to use to verify that they yield unbiased results. One cannot “transition” high school scores and not do the same for elem/middle scores without introducing score inflation or deflation. Even though columns F, G, and H are “transitioned” to match one another, column I, Growth Target, is not (the values in column I continue to range from 2 to 10, without adjustment, just as before any “transitioning”), and the resulting bias is evident in column J, Meet Growth Target.
I used columns I and J in my calculations for this bias check. Specifically, I focused on the schools with growth targets of 7 to 10 for 2011-12. According to page 15 of Bulletin 111, such schools have “entered into Academic Assistance.”
In this BESE-issued data set, the total number of schools with growth targets ranging from 7 to 10 was 725: 574 elem/middle schools and 151 high/combination schools, for a ratio of 3.8 to 1. If there were no bias toward the elem/middle vs. high/combination schools, then the ratio of approx. 3.8 to 1 would be reflected in the proportion of elem/middle to high/combination schools that met the respective growth goals. Instead, the ratio of elem/middle to high/combination schools having met the growth target was 1.3 to 1 (actual numbers: 105 elem/middle schools vs. 79 high/combination schools). Over 50% of the high schools met the growth goal (79/151 = .52); however, fewer than 20% of the elementary schools met the growth goal (105/574 = .18). Another way to view this situation is that if there were no bias and 79 high/combination schools met the growth goal, then one would see approx. 79 x 3.8 = 300 elem/middle schools also meeting the growth goal.
The 2012 school performance score calculations as set forth in Bulletin 111 are biased against elem/middle schools and biased toward high/combination schools.
Furthermore, in general, the 2012 bias favors school districts with more high/combination schools than those with fewer high/combination schools.
Given that the tenets of Bulletin 111 have not been examined and tested for the impacts upon future score calculations, there is no solid evidence to support the shaky idea that the 2012 school performance scoring bias is a “one-time occurrence.”
Based upon the above examination of the 2012 SPS data, it is not logical to say that the elem/middle schools just didn’t perform as well as the high/combination schools, that the teachers at the elem/middle schools didn’t “try as hard” or “achieve as much” as did the high/combination schools. And it is not decent, ethical, and likely not legal to say, “Oh, well. We’ll fix it next year,” or “We had to calculate scores this way because it’s in the bulletin.” There are jobs, reputations and money tied to these faulty numbers. As the BESE and LDOE, yours is the power to not only correct the gross bias in the 2012 school performance scores, but also to suspend, investigate and reconfigure Bulletin 111 in anticipation of future scoring bias.
Mercedes K. Schneider, Ph.D.
Applied Statistics and Research Methods