Skip to content

Louisiana’s 2017 NAEP Scores Must Not Be Pretty: John White Writes to NCES Prior to 2017 NAEP Release.

April 1, 2018

As reported in the March 30, 2018, Chalkbeat, Louisiana Superintendent John White is concerned about the integrity of soon-to-be released 2017 National Assessment of Educational Progress (NAEP) and how scores could be lower simply because students are struggling with using computers to take the test.

As such, White sent a letter to the National Center for Education Statistics (NCES) detailing his concerns.

john white 5  John White

It is funny to read about John White’s wanting the details regarding NAEP score adjustments; NAEP is not an “in house” job for White, which means that he cannot alter, adjust, manipulate the scores prior to public release, and he cannot delay release.

In short, he is left to appeal to NCES for information to soften the blow for scores he as a state superintendent has likely already seen, given that he “would like to be assured, as soon as possible” that 2017 NAEP isn’t a test of computer skill.

Those must be some shocking numbers White saw.

White’s March 23, 2018, letter to NCES is as follows:

Dr. Peggy C. Carr
Acting Commissioner
National Center for Education Statistics
Potomac Center Plaza
550 12th Street, SW
Room 4061
Washington, DC 20202

March 23, 2018

Dear Dr. Carr,

I am writing in reference to the 2017 administration of the National Assessment of Educational Progress (NAEP) in mathematics and reading, the results of which will be made public April 2018. Specifically, I am writing to urge that additional information related to the results be made available to state chiefs as soon as possible.

Under your leadership, the National Center for Education Statistics (NCES) has both maintained the nation’s high level of trust in NAEP and conducted unprecedented levels of engagement with state education leaders. Your willingness to explain openly the mechanics and results within the Nation’s Report Card has been a welcome approach.

The 2017 NAEP administration marked a significant transition from paper-based testing to computer-based testing. NCES found that, consistent with research on the NAEP (Bennett et al., 2008; Horkay, Bennett, Allen, Kaplan, & Yan, 2006), this shift in the mode of testing contributed to lower performance on NAEP forms among the general U.S. sample population. Using the small sample of paper-based testers, NCES calculated a baseline level of performance and adjusted nationwide scores to maintain the longitudinal NAEP trend. Based on this mode effect adjustment,
NCES has preserved the integrity of its effort to report trends in nationwide math and reading over time.

I understand that NCES may have found disparities in the mode effect on different subgroups of students. However, any disparate effect found was not significant. Thus NCES did not include any difference from one group of students to the next in its calculation of the mode effect. The adjustment NCES made in order to preserve the national trend is the same for every student.

It is my understanding that, though NCES maintained a consistent longitudinal trend at the national level, there remains the possibility that the mode effect in a given state may have been greater than the nationwide mode effect. This could be attributable to a disproportionately large population of a subgroup that experienced a greater mode effect than the national effect. It also could be attributable to the relative capacity of 4th and 8th grade students in a given state to use computers.

As a potential illustration of this point, no Louisiana student in 4th grade or 8th grade had ever been required to take a state assessment via a computer or tablet as of the 2017 NAEP administration. This fact, coupled with a variety of social indicators that may correspond with low levels of technology access or skill, may mean that computer usage or skill among Louisiana students, or students in any state, is not equivalent to computer skills in the national population.

I would like to be assured, as soon as possible, that when NCES reports math and reading results on a state-by-state basis over a two-year interval, the results and trends reported at the state level reflect an evaluation of reading and math skill rather than an evaluation of technology skill. I am therefore writing to request that the following information be made available to state chiefs as soon as possible:

1. The mode effect adjustment applied to each grade and subject nationally

2. The average mean scores for students taking the paper-based test and for students taking the tablet-based test, at the state level and at the national level, in each grade, subject, and subgroup

3. Evidence of the random equivalence of the groups of students taking the paper-based tests and students taking the tablet-based tests, at the state level and at the national level

4. National subgroup performance trends, reported by performance quintile, quartile, or decile.

Having this information as soon as possible will allow state chiefs the comfort to know that NAEP’s state-level reporting possesses the integrity in this transition year that NCES has rightly taken pains to assure exists at the national level.

I thank you for your continued partnership with our state and others, and I look forward to your response

Sincerely,

John C. White
Louisiana State Superintendent of Education

CC:
Hon. James Blew, U.S. Department of Education
Dr. William Burshaw, National Assessment Governing Board
Carissa Miller, Council of Chief State School Officers

References

Horkay, N., Bennett, R.E., Allen, N., Kaplan, B., & Yan, F. (2006). Does it matter if I take mywriting test on computer? An empirical study of mode effect in NAEP. The Journal ofTechnology, Learning, and Assessment (5)2. Retrieved March 23, 2018 from
http://www.jtla.org

Bennett, R.E., Braswell, J., Oranje, A., Sandene, B., Kaplan, B. & Yan, F. (2008). Does it matter if I take my mathematics test on computer? A second empirical study of mode effect in NAEP. The Journal of Technology, Learning, and Assessment (6)9. Retrieved March 23, 2018 from http://www.jtla.org

It seems that White’s letter to NCES was done in concert with a letter from the Council of Chief State School Officers (CCSSO), with the latter letter appearing to be the backup to White’s letter, with White requesting detailed information from NCES and the CCSSO letter being drafted in more general, “help the state chiefs” terms. Both letters are dated March 23, 2018, with both copying each other at the end of each.

In 2015, Louisiana’s NAEP results were mixed, with 4th graders faring better than 8th graders. Here is the Louisiana Department of Education (LDOE) comment to an Associated Press release in the October 28, 2015, nola.com:

The Louisiana Department of Education’s analysis of the figures shows the state moving up in its overall NAEP ranking in fourth-grade reading proficiency from 50th in 2009 to 43rd this year. The state edged up from 48th to 45th in fourth-grade math but is stuck at 48th in eighth-grade reading. For eighth-grade math, the state slipped from 45th in 2009 to 49th this year.

“In Louisiana, the NAEP shows both long-term, steady improvement, of which the state should be proud, and gaps within our state and beyond our borders that should fuel our push for higher expectations,” the state Department of Education said in a news release.

In drafting his time-is-of-the-essence letter to NCES, it seems that White is desperately seeking an angle to soften the media blow regarding some obviously low Louisiana scores on the 2017 NAEP.

The thing about having to explain low test scores is that scores themselves become the focus (and refocus), not the attendant explanation.

With or without explanation, those 2017 NAEP scores are coming.

calendar turning pages

______________________________________________________________________________

Want to read about the history of charter schools and vouchers?

School Choice: The End of Public Education? 

school choice cover  (Click image to enlarge)

Schneider is a southern Louisiana native, career teacher, trained researcher, and author of two other books: A Chronicle of Echoes: Who’s Who In the Implosion of American Public Education and Common Core Dilemma: Who Owns Our Schools?. You should buy these books. They’re great. No, really.

both books

Don’t care to buy from Amazon? Purchase my books from Powell’s City of Books instead.

6 Comments
  1. “This fact, coupled with a variety of social indicators that may correspond with low levels of technology access or skill, may mean that computer usage or skill among Louisiana students, or students in any state, is not equivalent to computer skills in the national population.”

    – “a variety of social indicators” – but NOOOooo said John White for so long as to the effects of poverty on assessment scores!!! “LOUISIANA children are as capable and as smart as children anywhere!” Blah blah blah

  2. “. . . no Louisiana student in 4th grade or 8th grade had ever been required to take a state assessment via a computer or tablet as of the 2017 NAEP administration.” Seriously? Students had a right to demand a paper test? Or those were the only grades that still used paper? I’m truly confused as to what White’s talking about.

    On a related topic, isn’t it ironic that John White is asking for the same level of details from the feds that certain parties have been trying to get from his department for years.

  3. We have been stating this from the very first administration of the CCSS state assessments in the 2014/2015 school year. The state assessments are an indicator of how well a student can manipulate the mouse and use the technology the test is given on. Many kids do not know how to type yet that did NOTHING to stop them from going on- line here in Florida. The state test (FSA) is a psychological test and non-academic in nature. It is also written by AIR/SBAC (American Institutes of Research) who are world-renowned for social and behavioral science. AIR also writes the NAEP test.

Trackbacks & Pingbacks

  1. Mercedes Schneider: John White Is Nervous about NAEP Scores | Diane Ravitch's blog
  2. Louisiana’s 2017 NAEP Does Nothing to Promote White’s Effectiveness as a State Superintendent | deutsch29
  3. NAEP and John White’s Computer-Based-Testing Hypocrisy | deutsch29

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s

%d bloggers like this: