Petrilli and Wright’s Porous Article on Mediocre Test Scores
In November 2015, Fordham Institute’s Michael Petrilli and Brandon Wright published an article entitled, “America’s Mediocre Test Scores,” which is slated to appear in the Winter 2016 issue of Education Next.
The article has a number of limitations worthy of note.
For starters, the article lacks links to references supporting the authors’ assertions. They open by referring to Pasi Sahlberg, Michael Rebell and Jessica Wolff, and Randi Weingarten, and without linking to sources, Petrilli and Wright connect statements these individuals have made regarding the connection between poverty and student performance on standardized tests to a “relative measure of poverty”:
For those educators quoted at the beginning of this essay, the answer is yes. They assert that the U.S. has a sky-high child-poverty rate compared to other developed countries.
To support their claim, they use a measure that assumes all families with less than half the median income in the country are by definition “poor.”
Petrilli and Wright need to provide the links for the source(s) to support their statement above.
Another limitation of Petrilli and Wright’s article is the “two out of three” assumption regarding the influence of poverty upon standardized test scores:
To prove that poverty is the major factor driving America’s meager academic achievement, at least two of the following three claims need to be established:
1. Poverty is related to lower levels of student learning.
2. America’s poor students perform worse than other countries’ poor students.
3. The poverty rate in the United States is substantially higher than the rates in countries with which it is compared.
Petrilli and Wright offer no justification for their “two out of three” condition. It reads as though since the first point above is “obviously in the affirmative” (their words), then Petrilli and Wright are trying to rock-scissors-paper a two-out-of-three to dull that “obviousness.”
The three points above presumably are supposed to align with three questions the authors present as headings in the text of their article, though the last point does not clearly align with the third question as a heading in the article (“poverty” in the statement versus “child poverty” in the question).
As for that first point, Petrilli and Wright offer the question, “Is Poverty Related to Lackluster Learning?” They use graphs of 2013 NAEP grade 8 math data to demonstrate the state and school level, inverse relationship between average scale scores and percentage of low-income students (defined using students’ free and reduced lunch status). Petrilli and Wright note:
Is Poverty Related to Lackluster Learning?
To this first question, the answer is obviously in the affirmative. That’s not to say “poor children can’t learn.” It is to say, rather, that there’s long been a clear connection between families’ socioeconomic status and students’ academic achievement.
Concerning their second question, “Do U.S. Students from Low-Income Families Underperform Their Peers Overseas?”, Petrilli and Wright open with an admission that they are trying to address a “complicated” measurement issue using a simplistic solution, and with more disclaimers:
The next question is whether U.S. students from low-income families are lower-scoring than those in other countries. To explore this question, we’re obliged to wrestle with measurement issues. The problem is complicated because no international data set contains both good measures of family income and good measures of student test-score performance.
The best available information is to be found in the data collected by the Program for International Student Assessment (PISA), which is sponsored by the Organization for Economic Co-Operation and Development (OECD). PISA, for its own analyses, uses an index of economic, social, and cultural status (ESCS) that looks at parent occupation and education, family wealth, home educational resources, and family possessions related to “classical” culture. PISA analysts use the index to stratify each country’s student population into quartiles.
Not everyone will agree with the way the ESCS index is constructed, but the data presented in Figure 2 are nonetheless quite instructive. [Emphasis added.]
Petrilli’s and Wright’s efforts to support their own Question Two are shaky from out of the starting gate. Moreover, even though they acknowledge that “not everyone will agree with the way the ESCS index is constructed,” they offer no details regarding such anticipated disagreement.
In attempting to support their ESCS-derived result, Petrilli and Wright offer this statement about parental education levels, without source or detail, which makes it read like a statement added as last-minute edit:
If we look at a different marker of socioeconomic status, parental education levels, we find a similar pattern. In the U.S., for instance, parents without a high school diploma are much more likely to be in poverty than their better-educated peers, and their children are much more likely than their peers to be low-performing and to drop out of school themselves.
No source for the above statement, but Petrilli and Wright do link to a Hanushek et al. study focused on parent education level. Specifically, Petrilli and Wright state the following:
In a study that examined whether some countries are particularly effective at teaching students from disadvantaged backgrounds, Eric A. Hanushek, Paul E. Peterson, and Ludger Woessmann find little difference in the rank order of countries by the performance of students from families where a parent had a college education and the rank order of countries by the performance of students whose parents had no more than a high school diploma.
However, the above statement is not supported by information in the Hanushek et al. article, which includes no graphic or other statement rank ordering countries “by the performance of students whose parents had no more than a high school diploma.” It does have such graphics for college educated parents and parents with no high school diploma. Furthermore, the term “little difference” is vague and should be more precisely detailed if the term was meant to refer to the rank ordering of countries on the two available graphics.
Finally, in addressing their third question, “Is America’s Child-Poverty High Compared to Rates Elsewhere?”, the authors state that the relative poverty measure (noted previously and defined as the percentage of a state’s population whose household income is below one-half of the state’s median income) is not as good a measure of poverty as is absolute poverty (defined as the percentage of the state living below the federal poverty line). However, Petrilli and Wright admit that their offering concerning absolute poverty is not precise enough to hone in on child poverty– which is the point of their question:
It’s important to note that the absolute poverty rates shown in Figure 3 are for the general population, not for children. It’s possible that absolute child-poverty rates would look quite different. But we have no way of knowing, because the data to calculate those rates across a large number of countries do not currently exist.
If the question concerns child poverty, the answer should address child poverty, which Petrilli and Wright fail to do. (As it is, the original statement at the outset of the article does not mention child poverty, only poverty. Not sure what happened there.)
In the closing of their article, Petrilli and Wright state that poverty cannot be used as an “excuse” to “explain away America’s lackluster academic performance.” They call it “a crutch unfounded in evidence”– as though their porous offering is solid evidence refuting the role of poverty upon standardized test scores.
Too many holes.