Hart, Weingarten, and Power Point Deception

Randi Weingarten really wants to promote the illusion that teachers have bought into the Common Core State Standards (CCSS). She hired Hart Research Associates to conduct a survey of AFT membership regarding perceptions about CCSS, and Hart did so March 27 – 30, 2013. Weingarten has used this survey as a platform to proclaim that “75% of AFT teachers surveyed support the Common Core.” But it is a lie, and I can prove it using a single Power Point slide composed by Hart for AFT and promoted by AFT and comparing that slide to the actual survey question.

I have written about other issues related to the Hart/AFT survey, but the gross manipulation of the survey result I will show in this post trumps all.

Proper Reporting of Survey Results

The National Council on Public Polls (NCPP) was established in 1969 to “set the highest professional standards for public opinion pollsters.” Its membership includes 27 polling organizations, including ABC News, Annenberg Public Policy Center, CBS News, Gallup, Los Angeles Times, and NBC News.

Hart Research Associates is not a member. At least not currently.

NCPP members agree to adhere to certain Principles of Disclosure, which are designated as three levels, the first being what survey reports must include:

Level 1 Disclosure: All reports of survey findings issued for public release by a member organization will include the following information:

• Sponsorship of the survey
• Fieldwork provider (if applicable)
• Dates of interviewing
• Sampling method employed (for example, random-digit dialed telephone sample, list-based telephone sample, area probability sample, probability mail sample, other probability sample, opt-in internet panel, non-probability convenience sample, use of any oversampling)
• Population that was sampled (for example, general population; registered voters; likely voters; or any specific population group defined by gender, race, age, occupation or any other characteristic)
• Size of the sample that serves as the primary basis of the survey report
• Size and description of the subsample, if the survey report relies primarily on less than the total sample
• Margin of sampling error (if a probability sample)
• Survey mode (for example, telephone/interviewer, telephone/automated, mail, internet, fax, e-mail)
• Complete wording and ordering of questions mentioned in or upon which the release is based
• Percentage results of all questions reported

Member organizations reporting results will endeavor to have print and broadcast media include the above items in their news stories.

Member organizations conducting privately commissioned surveys should make clear to their clients that the client has the right to maintain the confidentiality of survey findings. However, in the event the results of a privately commissioned poll are made public by the survey organization the above items should be disclosed.

In the event the results of a privately commissioned poll are made public by the client, the survey organization (a) shall make the information outlined above available to the public upon request and (b) shall have the responsibility to release the information above and other pertinent information necessary to put the client’s release into the proper context if such a release has misrepresented the survey’s findings. [Emphasis added.]

The goal of such stipulations is to ensure unbiased reporting of survey results.  I have bolded the sections in which the Hart/AFT Power Point info I will share has violated such disclosure.

Now, Hart is not currently a member of this organization and is therefore not subject to the hearing procedures outlined at the end of the Principles of Disclosure section.  Even though Hart is no longer a member of NCPP, the guidelines set by NCPP are a standard for assessing unbiased reporting of polling results.

Hart’s non-member status in NPCC does not excuse it from honest, straightforward reporting of survey results. In fact, Peter Hart used to sit on the NCPP board of trustees; his name is on this 1988 letter sent to a non-member polling organization with the following admonition:

As you are not a member of NCPP, obviously you are not bound by our release standards. However, had more of the pertinent information on methodology been released it would have enhanced the credibility of your efforts. … Shortcomings in the release of survey results from any source are of concern to us, as we all suffer when professional standards are breached. [Emphasis added.]

Thus, as a trustee member of NCPP, Peter Hart has been associated with admonishing non-member pollsters of the need for clear, open, above-board release of survey details.

Many of the Principles of Disclosure are common sense. If I conduct a poll and do not clearly and accurately disclose polling questions and actual results fully and according to the responses asked of participants, I do not have the public’s best interest in mind, for I am choosing to withhold clear information. Furthermore, if I reword questions and reduce reported categories, and if I withhold numbers on graphics supposed to represent the result, then I am manipulating the public via limited release of information.

The NCPP Principles of Disclosure include two additional levels. Level Two involves what survey organizations should provide to the public upon request. It is noted in Level Two that “complete wording of questions” is a Level One requirement and becomes a level Two requirement only if the survey has been translated into another language.

Hart released the survey instrument not as part of this original Power Point but upon the request of one of my readers: Hart AFT survey

At a later date, the same reader requested complete response counts. Those counts are yet to be received.

Level Three includes criteria that NCPP member organizations are “strongly encouraged” to meet, including public release of raw data sets (minus participant identifying information) and public release to a website of “complete wording, ordering, and percentage results of all publicly released survey questions… for a minimum of two weeks.”

The goal is… dare I write it… transparency.

The Power Point Slide In Question

Actually, a number of slides in the Hart/AFT survey evidence reporting problems. However, Weingarten’s position in support of CCSS is the hinge for the rest of the survey questions presented in this Power Point. In order to argue that teachers want more CCSS assistance, they first must be shown to support CCSS, period.  Otherwise, the issue needing addressing is not more CCSS assistance but the lack of teacher investment in CCSS.

The teachers in this AFT poll are not invested in CCSS.

The slide in question is page 3 of the Power Point above.

AFT 75%

The first deceptive issue with this slide is its title: Teachers Overwhelmingly Approve of Common Core State Standards. This leads readers to “see” what they are being told to in the slide– not only do “teachers” (not AFT teachers, but teachers in general) “approve,” but by a landslide they approve.  The clearly-intended message: Teachers in general really are for this Common Core. Readers are then told that teachers were asked this question:

 Based on what you know about the Common Core State Standards and the expectations they set for children, do you approve or disapprove of your state’s decision to adopt them?

Yet this was not the survey question.  The actual survey question was as follows:

Q3 Based on what you know about these standards and the expectations they set for children, do you strongly approve, somewhat approve, somewhat disapprove, or strongly disapprove of your state’s decision to adopt the Common Core State Standards?

Strongly approve ……………………………………………………………………….. 1

Somewhat approve …………………………………………………………………….. 2

Somewhat disapprove ………………………………………………………………… 3

Strongly disapprove ……………………………………………………………………. 4

Not sure ………………………………………………………………………………….. 5

Notice that a primary difference between the Power Point question and the actual survey question involves the gradation of the responses. This is key. In the Power Point, Hart/AFT have collapsed categories in order to present the illusion of “overwhelming approval.”  Thus, “strongly approve” and “somewhat approve” have been reduced to “approve,” and “strongly disapprove and somewhat disapprove” have been reduced to “disapprove.”  “Not sure” is not accounted for using a bar.

“Somewhat approve” is not “overwhelming approval.” It is hesitant approval. It is approval with reservation.  But saying that a notable number of teachers “approved with reservation” doesn’t sell the “teachers ‘overwhelmingly’ approve” point-of-view. So let’s manipulate the categories in reporting to support what we want to promote.

Notice also on the page 3 Power Point that only two bars are shown on a three-dimensional drawing of a bar graph.  Three-dimensional bar graphs are misleading; they can be manipulated because there is no straight, obvious “zero” line for readers to measure the graph against. And taking what should be a five-bar graph (one for each of the possible responses to the original survey question) and reducing it to two bars gives the illusion that only two options matter. Such presentation can easily disguise the subtleties of the muliple responses.

Why ask for people to respond using multiple responses then collapse catgories in reporting? Simple: The results didn’t turn out the way the researchers had hoped and require some doctoring to say “what we need them to.”

AFT took Gates money, and Gates supports (and bankrolls) CCSS.

Is this too far-fetched?

Back to bar-graph deception:

Hart chooses a two-bar graph and for each bar uses two different colors, supposedly to represent the multiple possible response categories.  But again, if each category is not clearly presented as its own two-dimensional bar against a clear zero line, the the reader must work to tease out the subtleties in response numbers/percentages.

Notice that the first bar on page 3 is tall and has “75%” at the top. But it is two-toned.  These are the response categories, unlabeled, with actual counts/percentages per category also absent.

This graph is the only one missing any kind of response category labeling.  Based upon subsequent graphs in this presentation, I see that the bottom color in each bar is equivalent to the “strongly approve” (in this slide, dark blue) or “strongly disapprove” group (dark red), and the color on top is the “somewhat approve” (light blue) and “somewhat disapprove” (light red) group.

Notice that the largest area in this graph is light blue– “somewhat approve.”

Most of the respondents were not “sold out” on CCSS. They had reservations.  They “somewhat approve” of CCSS.

This is not “overwhelming approval” for CCSS.

I am sorry that I am unable to report actual numbers at this time. If Hart sends the requested actual counts to my blog reader who requested them I will post the actual survey questions (all of them) with the actual response counts.

As for what we have before us now, on page 3 of this Hart/AFT Power Point, the title is deceptive, the question is misreported, the graph is confusing, and the percentages reported are misleading.

There is also an “aside” on this slide: “79% are very/fairly familiar with CCSS.” Again with the limited reporting and collapsing of categories.

Notice how much more detail is devoted to the reporting on subsequent slides.  They are still weak for their two-bar graphs, but there is less effort to conceal information.  The sale has been made: “75% of AFT teachers surveyed ‘overwhelmingly approve’ of CCSS.” These are now the “bring it home” slides to promote a moratorium on an issue that AFT teachers aren’t sold on but that Hart and Weingarten have shaped reporting to “prove” that they are.

Other Survey Questions

Once I receive the actual results for the survey, I will know how teachers in this survey responded to these two questions:

Q12a Have you read the Common Core standards for mathematics and English Language Arts, or have you not had an opportunity to read the standards?

Have read the standards …………………………………………………………….. 1

Have not had an opportunity to read the standards ………………………… 2

Not sure ………………………………………………………………………………….. 3

Q12b Have you received a hard copy of the Common Core standards, or have you not received a hard copy?

Have received a hard copy ………………………………………………………….. 1

Have not received a hard copy ……………………………………………………. 2

Not sure ………………………………………………………………………………….. 3

It seems that these would be questions asked at the outset of the survey. It does make sense that one would want to check to see if those who say they are or are not familiar with these standards have seen or read them. Instead, Hart/AFT ask and report on this perception question:

Q2 How familiar are you with the Common Core State Standards–very familiar, fairly familiar, just somewhat familiar, or not familiar?

Very familiar ………………………………………………………………………………. 1

Fairly familiar …………………………………………………………………………….. 2

Just somewhat familiar ……………………………………………………………….. 3

Not familiar ……………………………………………………………………………….. 4

Not sure ………………………………………………………………………………….. 5

And even Q3, the focus of the infamous page 3 of the Power Point,  is based upon perception (Based on what you know about these standards…).

But that is for another post.

A Challenge

Hart is a major research company. Weingarten leads a major teachers union.

To them I write:

You have deceived the public with your information manipulation.

You have done what NCPP tries to guard against. You have contributed tarnish to the image of the polling and survey research fields.

Rebut this article, folks.  I can’t wait to read it.