Caveon Test Security: We Don’t Actually Investigate, and We Keep Our Process to Ourselves.

In an effort to improve test security at New Orleans-based Singleton Charter School, operated by Dryades YMCA, the board chair, Darren Mire, plans to employ Caveon Test Security. For the backstory on Singleton and test security issues, see this article and this article by Marta Jewison of The Lens. However, here’s a slice:

The Louisiana Department of Education has formally warned the Dryades YMCA that it violated its rules for charter schools due to problems that led the state to void standardized tests for 165 students at James M. Singleton Charter School last fall.

Most of those tests were voided because students got accommodations meant for those with special-education needs. Some were voided because too many answers had been changed from wrong to right, indicating cheating.

The Y, which operates the school at its facility in Central City, fired Singleton’s school leader and three educators in January. The CEO of the Y resigned, effective mid-March.

The Y has also backed out of its plan to take over the nearby Mahalia Jackson Elementary School facility, which houses a health clinic and a preschool program.

So, Caveon is supposedly going to help ensure test security. However, as Jewison notes, Caveon has a botched history associated with the Atlanta cheating scandal:

In 2016, all Recovery School District schools and the Orleans Parish School Board hired test monitors after Algiers Charter, a group of five schools on the West Bank, reported test scores had dropped at its high school after its CEO hired test monitors.

Mire said Singleton plans to hire Caveon Test Security, the company hired by the school districts.

Caveon is nationally recognized, but its reputation took a hit in 2011 when Georgia investigators said it vastly underrepresented the extent of cheating in Atlanta Public Schools. Thirty-five educators were eventually indicted in the nation’s largest standardized-test cheating scandal.

When I hear of the company, Caveon Test Security, being contracted to investigate testing irregularities, I think of the DC cheating scandal during the time Michelle Rhee was chancellor and how DC hired Caveon to “investigate” (term used loosely). However, it appeared that Caveon did little more than the shielded, specific bidding of its boss, DC Public Schools (DCPS).

In his March 2011 article, Washington Post journalist Bill Turque captures Caveon’s watery job at getting to the bottom of the cheating issue (a goal that Caveon apparently wasn’t hired to achieve):

Versions of the Caveon reports made public Tuesday evening are heavily redacted, concealing even the names of the school principals [at Noyes and other schools Caveon investigated for their 2009 erasures: Birney, Burrville, Stanton and Tyler elementaries, Shaed and Walker Jones education campuses and Sousa Middle School]. They (the reports) consist of brief narrative descriptions of interviews with teachers whose classrooms were “flagged” for high erasure rates. In some instances, teachers who Caveon officials sought to interview were not available. All interviewees denied any improprieties and said all security procedures were followed.

In summaries of their findings for each school, investigators attributed high erasures to factors such as improved test preparation that called for students to look over the answers carefully before finishing. Caveon officials also repeated positive comments they received about improved academic culture at the schools under scrutiny.

“We were told that there has been a culture change at Tyler and the school is now much more orderly and discipline has improved,” investigators John Olson and Dave Couchman reported.

Education journalist John Merrow offers more details in his November 2017 summation, “The DC School Reform Fiasco: A Complete History”:

But let’s dig deeper into the surreal world that Caveon inhabits.  Caveon President Fremer maintains that his firm did not conduct an investigation in the normal sense of the word because his firm does not conduct investigations.  “We use the word ‘investigation’ in our materials because everyone else does,” he said, “but we do analysis, with the goal of process improvement and quality assurance.”  Then he added, “We were not brought in to help DCPS with an analysis of what had happened.”

The contract was for a two-part project: a security audit and questioning of certain people at just eight DCPS schools (even though many more schools had been implicated). But, he emphasized again in our conversation, it was not an investigation because Caveon was hired to “review and collect information.”  He told me,  “I give advice as to where to focus attention. I am not trying to position a client to put people in jail. Instead, we give them enough information about problems to allow them to fix them in the future.”

The security audit, he said, consisted of examining DCPS’ policies and procedures around the testing.  Caveon did not seek to find out if principals and teachers actually followed the rules, and so Caveon apparently did not inform Chancellor Rhee just how easy it would be to cheat on the DC-CAS before, during and after its administration.  Caveon did make some recommendations to improve security–recommendations, he said, that DCPS did not follow.

Part Two of Caveon’s work–the questioning–is even more interesting.  Dr. Fremer told me that DCPS gave him a list of the eight schools it was authorized to go into. DCPS also gave Caveon about 50 questions to ask of teachers, proctors, principals and assistant principals.  He said DCPS indicated that Caveon was not to stray from the list.  Follow-up questions, the essence of a good investigation, were actively discouraged, according to Dr. Fremer.

As for Caveon’s work in Atlanta, it is less than solid, as the September 2011 Atlanta Journal-Constitution (AJC) reports:

“I’m good. I’m fast,” Caveon’s then-President John Fremer told the business and civic leaders who hired the private, for-profit consulting firm in March 2010 to look into cheating in Atlanta schools.

Caveon delivered quickly, but it met spectacular failure. Gov. Sonny Perdue rejected the firm’s findings last year and publicly accused it of seeking to “confine and constrain the damage” from rampant cheating. This July, state investigators who conducted a deep, 10-month examination of Atlanta schools said Caveon vastly underrepresented the extent of test-tampering. …

For years, the small Utah firm has been the go-to contractor for test-fraud detection. But Caveon’s work in Atlanta and D.C. suggests the company is gaining a reputation more for clearing schools than catching cheaters, an Atlanta Journal-Constitution investigation has found.

Caveon officials reject the suggestion they would minimize cheating to please clients. …

The firm has staunchly resisted disclosing the inner workings of its test-data analyses, making it impossible for others in the field to validate or critique the results.

Now, here is something noteworthy in the AJC article concerning Data Recognition Corp (DRC):

[According to Georgia’s state investigators,] “Because of the manner by which Caveon calculated its index, and the contaminated statistical universe it used,” their report said, “many schools for which there was strong statistical evidence of cheating were not flagged.”

The findings prompted criticism from within the testing field, too.

“I think it was a huge mistake what they did in Atlanta,” [testing expert Gregory] Cizek said. “To sort of stake your work and your reputation and your findings on what you know are not the best data to be using I think is not a good idea.”

In contrast, Cizek praised an investigation in Pennsylvania by the Minnesota-based Data Recognition Corp.

DRC provided detailed equations and explanations in its report on dozens of schools it flagged for possible cheating in 2009. Such information allows other researchers to replicate the analysis to test its integrity.

So, let us return to New Orleans’ Singleton Charter School and other New Orleans schools that are planning on using flaky Caveon for its test security:

Louisiana currently has its testing contracts with DRC.

This leads to two pertinent questions:

• Why aren’t New Orleans schools contracting with DRC for test-security-related analyses?
• Why must puffed-up, secretive, non-validated Caveon be utilized instead?

Why, indeed.

__________________________________________________________________________________

I wrote a few books. Here’s one on the history of charter schools and vouchers:

School Choice: The End of Public Education? 

  (Click image to enlarge)

And here are two more: A Chronicle of Echoes: Who’s Who In the Implosion of American Public Education and Common Core Dilemma: Who Owns Our Schools?. Swell stuff.

Don’t care to buy from Amazon? Purchase my books from Powell’s City of Books instead.