Skip to content

VAM-promoting Doug Harris’ Vision for La.’s School Performance Scores

January 7, 2015

Economics professor Douglas Harris is a mysteriously-“Endowed Chair of Public Education” at Tulane University in Louisiana. He has written a book on value-added modeling (VAM), which he believes works in public education. He believes that “a year’s worth” of student learning can be predicted using linear, statistical algorithms and that such a prediction can be certainly and specifically connected to the actions of a single teacher. If a student scores at or above the statistically predicted value, then (let’s go feminine with pronouns here for ease of writing) that teacher is to be rewarded by keeping her job, because she and she alone must be the reason the student’s score rose to the statistically-predicted level. All other variables have been statistically controlled for, and if some variable is too messy to measure and statistically control– like student free will, for example– that variable is simply excluded. If it cannot be measured, it does not fit into VAM, and if it doesn’t fit into VAM, then it does not exist.

VAM is an appealing weapon for those on the hunt to purge the American public school classroom of the “bad teachers” who are hampering America’s ability to become narrowly defined as “globally competitive” via international, standardized test score results.

Really.

Nevertheless, not all who are able to skillfully wield VAM are willing to contribute their talents toward the “bad teacher” hunt.

Here is an excerpt from the American Statistical Association’s (ASA) April 2014 Position Statement on using VAM in education:

Most VAM studies find that teachers account for about 1% to 14% of the variability in test scores, and that the majority of opportunities for quality improvement are found in the system-level conditions. …

Ideally, tests should fully measure student achievement with respect to the curriculum objectives and content standards adopted by the state, in both breadth and depth. In practice, no test meets this stringent standard, and it needs to be recognized that, at best, most VAMs predict only performance on the test and not necessarily long-range learning outcomes. Other student outcomes are predicted only to the extent that they are correlated with test scores. A teacher’s efforts to encourage students’ creativity or
help colleagues improve their instruction, for example, are not explicitly recognized in VAMs. …

Attaching too much importance to a single item of quantitative information is counterproductive—in fact, it can be detrimental to the goal of improving quality. In particular, making changes in response to aspects of quantitative information that are actually random variation can increase the overall variability of the system. …

A decision to use VAMs for teacher evaluations might change the way the tests are viewed and lead to changes in the school environment. For example, more classroom time might be spent on test preparation and on specific content from the test at the exclusion of content that may lead to better long-term learning gains or motivation for students. Certain schools may be hard to staff if there is a perception that it is harder for teachers to achieve good VAM scores when working in them. Overreliance on VAM scores may foster a competitive environment, discouraging collaboration and efforts to improve the educational system as a whole. …

The majority of the variation in test scores is attributable to factors outside of the teacher’s control such as student and family background, poverty, curriculum, and unmeasured influences. …

The VAM scores themselves have large standard errors, even when calculated using several years of data. These large standard errors make rankings unstable, even under the best scenarios for modeling.

With these ASA cautions regarding VAM in mind, let us turn our attention once more to Harris. In January 2015, he released this report to the Louisiana Board of Elementary and Secondary Education (BESE) and a legislatively-established accountability commission.

In his report, Harris offers a number of suggestions that show just how out of touch he is with public school reality– and how much he “values” education through the limited visibility of the economist.

First, Harris suggests schools be “VAMed,” not just teachers:

I recommend adding average teacher value-added to the SPS and assigning 50% of the 150 SPS points to this measure. This would not only do a better job of measuring the performance of the schools, but improve alignment between teacher, school, and district accountability. It would also send a clear message to school and district leaders that they should focus on hiring, retaining, and developing the best teachers.

ASA cautions using VAM? What’s that?

No, no. Let’s up the VAM by making VAM half of a school’s performance score. And it certainly does send a “clear message”: Harris thinks that “best” teachers are those who fulfill their VAM determined, standardized-test-score-delineated destinies.

Several paragraphs later, Harris suggests that instead of 50 percent “growth” (i.e., VAM), why not just make SPS 100 percent “growth”?

I can tell you what would increase for sure under Harris’ VAM-love suggestion for SPS: Ways that administrators would figure out how to game this system– and that includes John White.

Indeed, Harris calls out White on his current, slanted awarding of points to schools for students actually not progressing:

This approach also rewards growth for all students, not, as with the current progress points, just those who are not proficient. The system could be set up to reward schools somewhat more for generating growth among low-performing students, but the present goes much too far, not rewarding growth for proficient students at all.

White’s awarding “bonus points” to schools for students who were not proficient on standardized tests actually helped the Recovery School District (RSD) and made “D and F schools… the best place for nonproficient students,” as my colleague Herb Bassett wrote in this post.

Whereas Harris does call White out for this “bonus points for non-proficiency” farce, keep in mind that Harris’ perspective on “growth” is narrowly defined as increased test scores as governed by VAM.

“Growth” is the test score. The test score is “growth.” Nothing else in the VAM mind constitutes “growth.”

I think Harris assumes that all teachers can be (and should be) VAMed. All teachers are not VAMed. I am not. Instead, I have “student learning targets” (SLTs), whereby I am told a score that my students “will” reach on the standardized test (in my case, this year, it’s the ACT for tenth graders, the PLAN), and I am told what percentage of students will reach this score. However, there is a VAM-like component built in this year: If a student scored higher than the cut score prior to my teaching him/her, and the score appears to go “backward” (not sure how the connection between the ninth-grade test, EXPLORE, and the PLAN has been established), then I will be penalized.

So, my arrangement this year is VAM-like, but teachers of other subjects may not be– or are so ridiculously– like a business teacher’s eval being based on her students’ scores in English. Gotta find a test for all teachers, even if it makes no sense.

Is this “growth”?

I had a 10-pound tumor removed from my abdomen in 2013. Not all “growth” is good.

Harris continues by suggesting that the school letter grading be expanded to include more categories, maybe by adding “plus” and “minus” to the letter grades. He thinks doing so will give schools “incentive to improve.”

What those expanded categories will also show is decline. Not sure how well White would like it if he had to try to hide publicly-nuanced RSD school grade drops– C to C-minus– D to D-minus. Also, a “minus” never “looks” like improvement. So, I think this would be a mixed bag for White. He would like to show how traditional public schools are “minus” schools, but he would not like to show this (especially nationally) for those RSD “miracle” charters.

The next Harris suggestion is my favorite for revealing Harris’ economic mindset as he advises on how to grade schools:

Add students’ college entry and first-year college persistence into the high school and district SPS calculations.

Yep. The endgame is to get students into college. That’s it. And Harris even suggests how to “value” this college entrance that I should push my students to have no matter their own preferences for their own lives:

I recommend a focus only on college entry and first-year (fall-to-spring) persistence because these are most under the control of high schools. …

As a general rule, more points should be given to credentials and other intermediate outcomes that are most closely-associated with the long-term outcomes. The evidence therefore implies four-year institutions should get somewhat more points than two-year institutions and persistence in the same institution should get more points than transfers of any sort. 

He bases the above “on economic research” (of course, of course).

When I think of “college entry” being under the “control” of a high school, my mind immediately goes to Texas charter chain, YES Prep, which gamed the “100 percent college acceptance” system by informing parents and students in its handbook that if a student did not get accepted into a college in the senior year, that student would not be awarded a diploma from YES Prep.

I wonder how long it will be before Harris conducts research on the connection between high-stakes (cough, cough) “accountability” and the proliferation of creative system-gaming among the “accountable.”

Economically-minded Harris suggests assigning point values to certain college acceptance and first-year completion.

This is soo incredibly narrow.

My sister delivered pizzas for years before joining the Air Force. She then attended college and became an electrical engineer. She did not go straight to college. No points for our high school.

My brother graduated at nineteen. He did not like school, and it was a bit of a fight to get him to complete it. He is a commercial fisherman who builds his own boats. This is what he enjoys. No points for his high school.

One of my former students came to visit me last year. He was admitted into a program to train him to be an underwater welder. No points for our high school.

A locksmith recently told me how difficult it is becoming to find young people interested in becoming locksmiths. He said there is a real need for locksmiths and that it is a dying profession. The way to learn is to apprentice with a licensed locksmith. As it happens, one young man had come to him recently in order to take him up on the offer to be apprenticed to become a locksmith. No points for that young man’s high school.

No points, no points.

Harris continues by writing that he does not discount short-term job success or long term career success. He just thinks “we [should] hold people accountable for what they control.”

YES Prep is “in control”….

Harris has other suggestions, and I will leave eager readers to read them. At the conclusion, Harris notes that his goal was to “make the state a national leader.”

Harris neglects to clarify a leader in exactly what, but I’ sure it will be much too “economical” for me to “value.”

Point systems for “grading” the teacher-student (and school-teacher-student) dynamic will always fall short because the complex nature of that dynamic defies quantifying. If test-loving reformers insist upon imposing high-stakes quantification onto schools and teachers, it will backfire, a system begging to be corrupted by those fighting to survive it.

It is not that I cannot be evaluated as a teacher. It’s just that such evaluation is rooted a complex subjectivity that is best understood by those who are familiar with my reality. This should be true of the administrators at one’s school, and I am fortunate to state that it is true in my case.

There are no numbers that sufficiently capture my work with my students. I know this. Yes, I am caught in a system that wants to impose a numeric values on my teaching. My “value” to my students cannot be quantified, nor can my school’s value to my students, no matter what the Harrises of this world might suggest in commissioned reports.

________________________________________________

Schneider is a southern Louisiana native, career teacher, trained researcher, and author of the ed reform whistle blower, A Chronicle of Echoes: Who’s Who In the Implosion of American Public Education

previti chronicle pic

6 Comments
  1. Will Fitzhugh permalink

    Let’s hear it for student free will, even if it doesn’t always help the VAM score. Thanks for reminding us about the “students are human beings” thing that so often gets omitted from the Visions of Pundits.

    Will

    Sent from my iPad

    >

  2. Voices and Places from the past: Pre-Katrina to Post Katrina

    After leaving an Alabama Teacher Corps program involuntarily in 1968 because of the KKK, I landed at Tulane when it still had a department of education. I joined its (TAPES) Teacher Assistant-ship Program in Elementary Schools- a program where I earned my post-BA level teaching credential and went on to teach in the Orleans Public Schools for many years. Those were the days when Tulane still had a Department of Education, (disbanded) because the College of Arts and Sciences (as I was told) didn’t feel the “field of education” was a credible subject for Tulane. Tulane’s Education Dean, Bob Wimpleberg, moved from Tulane to become the Dean of the College of Education at UNO. He and colleagues built through the 80s and 90s one of Americas most credible Urban Education Colleges. UNO’s education departments, (Pre-Katrina) surpassed LSU- big brother at the time, as UNO was still part of the LSU system.

    VAM: You are correct, Mercedes, this use of VAM, by Harris, is a SHAM. Your statistical arguments are most salient. I’d like to offer another argument from an economic point of view. While studying economics as an undergraduate I took a course on “The Theory of Public Finance’ and later utilized the analysis of professor, B. P. Herber (1967) “Modern Public Finance,” in my, 1986, UNO Master’s Thesis: “Education Vouchers: The use of family choice in American Education” (pp. 117-122). My views (then) and my views (now) have evolved considerably – but, the economic theory on private finance found in my thesis remains useful. i.e., Education when thought of as an “economic good” is unique and because of the “exclusion principle” – American School Children CANNOT be “excluded” from participation even if not affordable for a particular family private economic conditions do not exist. Because market conditions for private enterprise do not exist, OUR tax dollars are used for these “privatized” schemes.

    Recently, I read “The Shock Doctrin: The Rise of Disaster Capitalism” by Naomi Klien. I was both disheartened and while not “shocked”- I was saddened to learn just how politically calculating the purveyors of “privatization” have been regarding education policy and other areas of our lives in America. Naomi Klien’s analysis connects Milton Friedman, directly, to the use of “economic shock therapy” to use his term. Shortly before he died at the age of 94 he wrote an opinion piece for the New York Times advocating for the privatization of schools destroyed by Hurricane Katrina. My, what a legacy Dr. Friedman has left us! One that includes the voucher experiment in Chile started under Agusto Pinochet- now gone off the right wing conservative tracks due to recent elections and student protests. Issues of inequity by the system have even been noticed by the International Wall Street Journal- warning of the dire consequences if “fairness” and “equity” disrupt the economy with more public cost in the education system in Chile- of course they call public education initiatives toward equity- socialism.

    Doug Harris’ mis-utilization of VAM, makes him out to be but one more Friedman ideologues masquerading as an academic and given credibility in the Louisiana legislature. He should be honored by Rex at the next Mardi Gras Ball. That would be fitting.
    Monty J. Thornburg, PhD, UNO.

  3. Laura H. Chapman permalink

    Impressive critique of the discredited reasoning of this pontificator who has the kind of vision available to a person trapped in a sewer drain pipe, or economic silo, or choose your metaphor for the world can only be as I see it.

    Your brief case studies within family reveal the stupidity of holding schools accountable for the life choices that students will make. This economics professor disregards the fact that a typical US work has had 11 jobs before the age of 44. That is US Dept. of Labor stat that sticks in my brain, footnote somewhere if really needed.

    My brother was a high school dropout during WWII. At the age of 17 he was recruited by the early version of NASA to build model airplanes for wind-tunnel tests at Langley Field. He completed high school, earned a college degree on the GI bill, ended a long career in mechanical engineering with 19 patents including the mechanism for capturing images from satellites, these now documenting, among much else, the effects of global warming.

    I have dubbed the increasing role of economists in shaping (stupid) educational policies “the econometric turn in education.” This economist is a perfect example with his navel gazing at numbers.

    Here is a more modest an appropriate take on the fixation with “big data.”

    “We are more susceptible than we may think to the “dictatorship of data”—that is, letting the data govern us in ways that may do as much harm as good. The threat is that we will let ourselves be mindlessly bound by the output of our analyses even when we have reasonable grounds for suspecting something is amiss. Or that we will at-tribute a degree of truth to data which it does not deserve. Viktor Mayer-Schönberger & Kenneth Cukier. (2013). Big Data: A Revolution That Will Transform How We Live, Work, and Think. Boston: Houghton Mifflin. p. 166.

  4. Gaye Ingram permalink

    My own experience tells me the teacher is critically important in determining what her students master (note that verb) in her class, both her knowledge of the subject and her pedagogical skills. I think you underestimate that influence.

    What no one in the system wants to do is use a combination ability+mastery to place students in, say, English and mathematics, the core competencies’ areas. And yet research is clear that students of different learning abilities learn not only different things, but also in different ways. The ability to generalize, for instance, is frequently in place and running for the gifted student by first or second grade, whereas it cannot be counted on as a trait of all students until sometime around the fourth grade. The slower learner will acquire it still later. A teacher who has learning impaired, gifted, average, and above-average ability students in one classroom is at a disadvantage from the outset. If we’re going to reform, let’s reform classroom placement and place teachers and students on a fair, reasonable footing. Then assure that the students in each class are presented with serious challenges, work on which no one will always make an A. We are sacrificing our children to a flawed idea of democracy and “equality.” Let’s put learning first as a goal. Then let’s create situations in which teachers’ work may be fairly evaluated.

    And let’s not assume that subject mastery and “imagination” or “creativity” are antonyms. They are complementary, not antithetical to one another. Every serious study of the nature of creativity shows that it grows out of previously acquired knowledge, usually knowledge that has been laid down earlier and is brought together into a “discovery” by a new piece of knowledge. The student who masters the skills and meets the benchmarks in the English standards, will be more imaginative in all his work because he will have skills and knowledge that are fundamental to discoveries in a variety of disciplines.

    One thing that concerns me is that all involved with this issue seem more focused on teacher evaluation than on student learning. If teachers acquire a high degree of mastery of the subjects they teach, their students will thrive.

    On the argument that everyone does not want to go to college or does not need to go to college, I agree. We need a strong vocational education program. But I also believe that every high school student of average or above-average learning ability should meet the requirements for college admission. They should be able to succeed in their senior year at a reasonable level if they are to work in a non-professional vocation. The SAT, ACT,and other measures of achievement and/or potential provide valid reflections of knowledge and reasoning skills that should be expected of a citizen of Louisiana and the United States. And that is what public school education was designed to do.

    Surely statisticians can design measurement tools to assess students’ and teachers’ success at that. Using college success measures for the college bound is also reasonable. But distinctions should be made.

    I have been so disappointed in the negativity that has surrounded the subject of standards and the Common Core and teacher evaluation. How can people so negative expect to succeed? Or, the better question is, how can their students expect to succeed?

    LPB broadcast a discussion of the Common Core recently, and one of the most vocal opponents of the standards in language arts was a former English teacher who warned the group that the Common Core standards grew out of the New Criticism, which she spoke with a certain disdainful horror. That meant, she averred, there could be no courses, say, in English literature, wherein one placed a work of literature in its historical context. How foolish, the New Criticism is but a name given to the technique of learning the meaning of a piece of writing without history or biography. It grew out of a resistance to Victorian instruction, in which the writer’s work of less consequence than his bibliography and biography. Teachers told students what a work meant, and if they encountered something new, students were referred to critical texts to tell them what the work meant. The New Criticism teaches analytical skills that free students from Cliff’s Notes. It opens them to the way a poem or memoir or short story works, so that they will appreciate what they read. And it gives them the skills and background to be discerning judges of language. As consumers, as workers, as citizens who must make civil and political decisions, they should be able to detect errors in reasoning and explain them to others so that a discussion might be had about matters critical to all. The student in American literature should be able first of all to read and through the use of analysis, discern the meaning of a work. After that information about biography enriches that basic understanding. The same is true when it comes to source studies, historical conditions, and other contextual matters. Simple enough if one knows how to read independently, to analyze and arrive at meaning on one’s own—all skills that are taught in the Common Core and the New Criticism. I got the feeling that teacher lacked those skills. Why else would she begrudge the pleasure of personal discovery for her students?

    Having taught those skills, one may use them with any content. And the student who has practiced their use throughout his school years will be better equipped at the end of 12 years of schooling to do anything he wants to do.

    Has anybody thought of offering demonstration lessons for parents? Parents who fear their children will be “brainwashed” (several in the group seemed to fear that) will be pleased to see this new set of benchmarks actually is geared to prevent that event.

    I’d like to see some positive talk in education!

Trackbacks & Pingbacks

  1. Commentary on Harris and Larsen’s New Orleans OneApp Report, What Schools Do Families Want (And Why)? | deutsch29
  2. My Thoughts on Doug Harris’ January 15, 2015, New Orleans Community Meeting | deutsch29

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: