Skip to content

Guest Post: Deb Herbage on American Institutes for Research (AIR)

September 28, 2015

deb herbage  Deb Herbage

Deb Herbage (@DebHerbage) was born and raised in Massachusetts and worked for a Fortune 100 company for 20 years as a business analyst.  She relocated to Trinity, Florida in 2007 with her husband and daughter. Deb is an outspoken advocate and active member of the Opt Out Florida Network and over 50 Opt Out groups across the country.

The following is a guest post by Herbage, who has been investigating American Institutes for Research (AIR).

_______________________________________________________________

Who is Really Benefiting from the State Assessment Contracts?

by Deb Herbage

With its inception in 1946, the American Institutes for Research is world famous for their social science and behavioral and psychological research.  Partnering with the US DOD (Department of Defense), the US DOH (Department of Health), the US COC (Chamber of Commerce), and the US DOE (Department of Education), AIR is a non-profit company based in Washington, DC, and is the recipient of many contracts from the various states for their Common Core “state assessments,” whether it be signing a contract directly with a state Department of Education (like the case in Florida, North Dakota, Delaware, Washington, Oregon, Maine, Michigan and Arizona)or through the Smarter Balanced Assessment Consortium (SBAC) (established in December 2010 for $175,649,539with the funds provided by the Federally funded Race To The Top, RTT).

On September 6, 2012 – AIR announced its partnership with the Smarter Balanced Assessment Consortium to create an online test delivery system for the states.

Here is an excerpt from the SBAC Memorandum of Understanding touting the “services” it is providing to their membership states (after they pay the fees of course) (click on image to enlarge):

SBAC excerpt

SBAC also announced that it chose AIR to partner with the consortium to deliver the initial tests to millions of students in 2013 and 2014.  AIR stated it would make available an “open-source testing platform to enable all states and others to deliver cutting edge tests.”  On December 2, 2014, AIR announced they were teaming up with Microsoft and AssistX to bring students across the United States the most secure online assessment.  AIR has many affiliates under its influence, such as Learning Point Associates,  Association for Education Finance and Policy, College Measures and International Development, Evaluation and Research Program, and many partnerships, such as with Data Recognition Corp. (DRC), the Bill and Melinda Gates Foundation, Council of Chief State School Officers (CCSS) (one of two groups that brought us the Common Core State Standards), Wested, and Google.

When Florida’s Commissioner of Education, Pamela Stewart, signed the contract with AIR (which outsourced the contract to DRC on June 4, 2014), it agreed to $59,933,754 for Year 1, $58,301,988 for Year 2 and $56,954,413 for Year 3 which totals $175,190,155 for the initial 3 year contract. (Click images to enlarge.)

SBAC excerpt 2

SBAC excerpt 3

Should FL choose to renew their contract – the grand total will be $322,799,459.  For that kind of money one would expect a flawless test, and one would expect the FL DOE to hold AIR to the terms of the contract. One would also expect AIR would fulfill its promises and obligations in the contract, such as “must provide empirical evidence of psychometric validity.” AIR stated in numerous places in the contract it signed with the FL DOE that the test was psychometrically valid and reliable.  Here are some excerpts from the executed contract (number 14-652):

AIR’s Assessment Program offers psychometric and statistical services that stand alone in terms of quality and innovation. The integration of psychometrics with statistics and sampling sets AIR apart from the competition.

…State testing programs can undercut the best psychometrics, leading to volatile test results from year to year and inaccurate classification of examinees. AIR combines expertise in sampling and psychometrics; all of our samples are optimized, and our statistics accurately reflect the complexities of the sample designs. (page 165)

In testimony to the Florida Senate on March 4, 2015, Commissioner Stewart stated that the test “was absolutely reviewed and is psychometrically valid and reliable.” This is not a true statement because if it were, then Florida could have saved the taxpayers the $594,000 we paid for the validity study to be conducted.  In fact, Utah’s Department of Education sent a letter dated March 23, 2015, to Senator Simmons, stating they did not have psychometric and reliability data for their SAGE test, another AIR test. AIR used the Utah SAGE questions on the Florida Standards Assessment (FSA).  Why would the FL DOE allow questions from a test that was not psychometrically valid or reliable be used on our test?

Why is this important?  As we know, SBAC is one of the two organizations that make up the Common Core “testing consortia” (with PARCC, Partnership for Assessment of Readiness for College and Careers, being the other one). Whether states signed the contract with AIR or with SBAC (essentially the same thing), the SBAC Memorandum of Understanding (MOU) clearly states that each “member state” is receiving psychometric services (see two excerpts from the SBAC MOU below).

The MOU clearly states SBAC will conduct validity studies as part of the “Assessment and Item Design” when states purchase the “Assessment Package.”  (Note that the two MOU’s from PARCC and SBAC include the similar language; just the state name and number of students tested varies.)

SBAC excerpt 4

SBAC excerpt 5

The SBAC/AIR charge “member states” fees  for services, such as a previously-alluded-to annual membership fee, as well as administrative and support service fees (which include “psychometric” services) and a “projected testing numbers” fee.  They are charging the member states based on number of students tested.  Every member state MUST comply with the confidentiality and the governing board. This “taking the lead” in forming Common Core testing consortia is obviously influenced by the Council of Chief State School Officers (CCSSO).

For not being recognized as a legal “federal entity,” CCSSO sure has a lot of power of persuasion, as does the other Common Core license holder, NGA (National Governors Association).

All the Common Core testing consortia contracts I have looked at all maintain that their tests are “psychometrically valid and reliable.”

(Here is some summary information on state testing contracts and MOUs I have and have examined for Oregon, North Dakota, California, Ohio, West Virginia, Colorado, and Delaware, which have a combined price tag in excess of $150 million, often for three years of testing.)

Based upon all those costs noted in the state contracts and membership MOUs, AIR, SBAC, PARCC, Measured Progress, Data Recognition Corporation are all drawing massive profits.

It doesn’t really matter if a state signed with SBAC or PARCC.  Considering the faulty rollout of the state assessments across the US, after paying the exorbitant taxpayer money for the state assessments, one would think that these tests would be the ultimate display of perfection or at least live up to the bragging in the contracts.  Our kids certainly are not benefiting from these tests– tests they cannot log into, tests that shut down on them, tests that cannot determine academic performance or inform instruction – all the expectations outlined in the state contracts.

And considering we had the hotly-debated “validity study” here in Florida– and due to the fact that SBAC and AIR joined forces (back in 2012) and AIR notes repeatedly they wrote the test for SBAC and also wrote the special adaptive algorithm for the state assessments and according to the below excerpt from the California SBAC MOU– would it be a reasonable assumption to say that states are being charged for psychometric services that are not being provided?

One thing is for sure: Our kids are NOT benefiting from these tests.

But I know a lot of people who are.

 

16 Comments
  1. Comment and Questions for Deb: Thank you for this amazing research!

    AIR is responsible not for the assessments in New York (that fell to Pearson), but they do take our annual testing data and generate the VAM scores now under scrutiny in the Shari Lederman case.

    Do you or Mercedes have any insights into AIR’s NY VAM technical report?

    I suspect we all see how unlikely it is that AIR will ever generate valid and reliable tests for K-12 educational purposes. Is there any reason to believe that their previous research has similar flaws, or was the rush to implementation and huge amounts to taxpayer money too tempting to resist?

    • Deb Herbage permalink

      Hello! Yes – I have information on VAM. AIR wrote the convoluted VAM process for FL. We have paid AIR $6M to date for VAM. This is part of Jeb’s “accountability” plan – holding teachers accountable with the standardized test scores, school ratings (A-F) and holding students accountable. I will be happy to send you the info I have for FL. I do not have specific information for NY but considering it was rolled out to other states under the guise of accountability – it will probably be the same.

  2. Jen in TN permalink

    Tennessee has a brand new test this year called TNReady (because our legislature pulled out of PARCC). The contract for TNReady is through Measurement Inc., but guess where the new test questions were purchased from? Yep, Utah’s SAGE test which is from AIR. What a tangled web of money and deception…

  3. Curious in CT permalink

    Check into this: http://ne-edit.com/?page_id=39 ” The company earned a stellar reputation and began landing contracts from some of the largest educational publishers and testing companies in the country, including Cengage Learning, Pearson, CTB/McGraw-Hill, and the American Institutes for Research (AIR).”

  4. Well done.
    Related: The connection between Lamar Alexander and the “New American Schools Development Corporation” which in “May of 2004, NASDC merged with another company. That company should be familiar to those fighting Common Core. The company NASDC merged with was The American Institutes for Research or AIR for short.”

    http://datechguyblog.com/2015/02/19/down-the-rabbit-hole-with-lamar-alexander/

  5. Laura H. Chapman permalink

    Amazing research and thanks for getting this into circulation. The plundering and malfeasance is THE uACCOUNTABILITY problem that too few are talking about. Following the money and the backscratching and interlocking directorates is not easy.

  6. Andy James permalink

    I don’t presume to understand all of the gobbledegook claims cited in this posting but one thing I’ve noticed: When I’ve challenged the validity/reliability of the SBAC in the presence of its supporters, none have offered any substantive evidence of AIR or anyone else certifying or even attempting to show scientifically researched evidence to counter the claims. Among those who have warned against broadening the “conclusions” attached to the SBAC test scores to include teacher and school evaluations are the American Statistical Association, ETS, the Rand Corporation and the National Academy of Sciences. Why is it that the otherwise competent purveyors of such broadened (mis)use of test results can’t seem to launch a credible rebuttal?

    From: deutsch29 Reply-To: deutsch29 Date: Monday, September 28, 2015 9:12 PM To: Andy James Subject: [New post] Guest Post: Deb Herbage on American Institutes for Research (AIR)

    WordPress.com deutsch29 posted: ” Deb Herbage Deb Herbage (@DebHerbage) was born and raised in Massachusetts and worked for a Fortune 100 company for 20 years as a business analyst. She relocated to Trinity, Florida in 2007 with her husband and daughter. Deb is an outspoken advocate”

  7. Deb, from a commenter banned from my blog:

    “Florida had a Plan B test as they did not sign-up for SBAC. This test had nothing to do with the two consortia. Pesky facts.”

    • Deb Herbage permalink

      Florida had a Plan B alright. They originally chose PARCC and due to growing concerns over “Common Core” Governor Scott dumped PARCC. He then appointed Pam Stewart as commissioner of education and she signed the contract with AIR. AIR partnered with SBAC in 2012. AIR wrote the test for SBAC. AIR wrote the special adaptive algorithm for SBAC. Perhaps the reader should go and click on the links in the article for the facts. They may be “pesky” but they are the facts.

  8. Dr. William Murdick, Emeritus Professor of English permalink

    Such tests can never be valid for certain parts of the Common Core. In English, for example, students are expected to be able to produce a persuasive argument over several pages of writing that draws on readings, bringing in other voices and documenting those sources. That can only be evaluated validly by looking at actual student essays written in and out of class for such an assignment. The Common Core and the tests invented for the Common Core standards are two completely different things and should be evaluated separately. There are certainly some good things in the Common Core that ought to be retained. For example, that writing project I just mentioned is a great preparation for college composition courses. That lengthy project was beginning to become popular in high school English classes just when the testing craze hit. It was eliminated because it couldn’t be measured through standardized tests. The Common Core brings it back, but the testing will probably eliminate it again.

  9. Great post! Thank you.

  10. Deb, you outdid yourself on this one! Proud of you! As you know, I trust AIR as much as a coiled up rattlesnake two feet from my face.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: