FBI Raids Former SAT Exec Manual Alfaro’s Home; Alfaro Posts on LinkedIn Next Day

On August 26, 2016, Reuters reported that the FBI raided the home of former College Board exec Manuel Alfaro, in connection with a leak of over 400 SAT test items to Reuters earlier the same month.

According to Reuters, the FBI “seized computers and other material” from Alfaro’s residence.

Alfaro has been publicly posting his concerns about the redesigned SAT in detail for months on LinkedIn, and he continues to post on LinkedIn the day following the raid, August 27, 2016.

Below is Alfaro’s August 27th post, in full, which he begins with a bold declaration:

 

Shining a Spotlight on the Dark Corners of the College Board: SAT has a Critical Technical Flaw

Manuel Alfaro

August 27, 2016

The FBI raided my home yesterday at the request of the College Board. It appears that Coleman wants to raise the stakes. Challenge accepted!

 Note: Whenever “SAT” is used below, it refers to the “redesigned SAT.”

The SAT test specifications require that, for items in context, 10% are Heavy (more than 60 words), 50% are Medium (40-60 words), and 40% are Light (less than 40 words). On its website, the College Board states that the SAT Practice forms were constructed using the same specifications as the SAT Operational forms. The College Board, however, did not use these specifications to build the SAT forms. And we found out about this after the SAT forms were built.

On January 2015, I received a high priority email from Sherri Miller requesting a report of the item distribution across the three categories (Heavy, Medium, Light)—she needed to provide confirmation, that night if possible, to Cyndie Schmeiser that the SAT forms were built to specifications. After analyzing the form metadata, I let her know that all the forms had about 45% Heavy items. “Wow,” she replied, “We had changed that to 10% heavy in the specs given the timing studies. How did we get to 45% of them being heavy?”

Months earlier, at a meeting with the SAT’s Technical Advisory Committee (TAC), the committee recommended that the SAT specifications be changed to address the timing problem—the formal timing studies conducted by the College Board had shown that the SAT test was speeded (students did not have enough time to complete the test). The other option was to increase testing time to ensure that students had enough time to complete the test. After the meeting with the TAC, Sherri Miller changed the specifications as follows:

Old SAT Specifications (Number of Items)

Heavy 13; Medium/Light 13; Total 26

New SAT Specifications (Number of Items)

Heavy 3; Medium 13; Light 10; Total 26

The new specifications, however, were not used to build the forms. I don’t know where the breakdown in communication occurred, as I was not involved in communications with the team that built the algorithm to select the forms. The only thing I know for sure is that the algorithm used the old SAT specifications to build the forms. Additionally, even if they wanted to use the new specifications, the operational item pool would not have been able to accommodate the new requirements: the item pool was built to support the old specifications, so there were not enough Medium and Light items in the pool to build the forms.

In the paragraphs that follow, I will provide an analysis of the SAT Practice forms to show that the old SAT specifications were used to build the SAT Practice forms and, by extension, the SAT Operational forms. The tables below provide a summary of the findings. Note: form designation (Practice or Operational) was done after the forms were finalized. So all forms have the exact same timing problem.

Actual Number of Items in Practice Forms

Form 1

Heavy 19; Medium 12; Light 0; Total 31

Form 2

Heavy 21; Medium 8; Light 1; Total 30

Form 3

Heavy 22; Medium 6; Light 1; Total 29

Form 4

Heavy 22; Medium 7; Light 2; Total 31

The formal timing studies showed that about 25% for the students were unable to reach 22% of the items on the math test. Further, the test disadvantaged one group of students in particular, with only 37% of the students reaching the last item on the NO Calculator Section and 41% of the students reaching the last item on the Calculator Section. For comparison, 83% of students reached the last item in Reading; 80% of students reached the last item in Writing; and
for Reading and Writing, 73% and 68% of lowest achievers reach the last item, respectively.

Given these timing study results, the SAT is clearly unfair. Which is precisely why Sherri Miller and Jay Happel changed the specifications. The problem was they did not tell the team that wrote the algorithm to select the forms.

I was waiting for the College Board to release the May form to the public before publishing this post so that I could include an analysis of the form in this post. The release of the form has been delayed, according to insiders, to fix some problems. I wonder if one of the fixes the College Board intends to make is to reduce the number of words in the items in order to make the form adhere to specifications.

Please demand that the College Board release UNEDITED copies of the SAT forms students took in May 2016 so that you can verify the timing problems for yourselves.

Alfaro clearly wants to expose the College Board.

I will continue following Alfaro’s story as it unfolds.

 Reuters mails SAT items back to College Board for confirmation.

_________________________________________________________

Released July 2016– Book Three:

School Choice: The End of Public Education? 

  (Click image to enlarge)

Schneider is a southern Louisiana native, career teacher, trained researcher, and author of both A Chronicle of Echoes: Who’s Who In the Implosion of American Public Education and Common Core Dilemma: Who Owns Our Schools?.

Don’t care to buy from Amazon? Purchase my books from Powell’s City of Books instead.