Skip to content

Oregon High Schools: Smarter Balanced, No, Wait, ACT/SAT, No, Wait, Smarter Balanced. But Those Opt-Outs…

July 14, 2018

In spring 2017, the Oregon Department of Education (ODE) unilaterally decided to drop Smarter Balanced assessments for high school beginning in 2018-19 and replace with either ACT or SAT. After stakeholder response, ODE changed its decision.

Oregon school districts want the right to decide whether they change from Smarter Balanced to some other “nationally recognized” test; they communicated as much to ODE in 2016/2017, but in deciding to choose either ACT or SAT for the entire state, the ODE disregarded its stakeholders in the initial decision, which put ODE into backpeddling mode. But backpeddle it did.

The additional $5M – $6M biennial cost might also have had something to do with it.

Below are excerpts from the 76-page report dated June 5, 2018, and submitted to ODE on behalf of stakeholders. Among concerns voiced by stakeholders is the fact that ACT and SAT call the shots on test administration specifics, and any alteration of those specs “would not result in college-reportable scores.”

Overview

The Oregon Department of Education (ODE) has been engaging stakeholders for feedback regarding the development and implementation of our State Plan under the Every Student Succeeds Act (ESSA) since 2016. Oregon’s ESSA Plan, the result of substantial effort and scrutiny by stakeholders, partners, and staff, was approved by the U.S. Department of Education in August 2017.

The ESSA Standards and Assessment Workgroup convened as part of this effort recommended that ODE research the flexibility defined in ESSA around high school assessment options. This flexibility was in fact not pursued, nor was it ever intended to be applied at the student level. Instead of pursuing the flexibility as defined in ESSA, which allowed for local education agencies to request permission from their state education agency to use a nationally-recognized, college entrance examination district-wide, and for the state to establish criteria to evaluate the requests, ODE made a public announcement in April 2017 that it would pursue a nationally-recognized college entrance examination to replace our Smarter Balanced assessment as Oregon’s high school accountability assessment for English language arts and mathematics starting in the 2018-19 school year at the state level. This announcement was followed by a Request for Information (RFI) for potential vendors in May 2017, to which two vendors responded.

An RFI summary was developed by ODE’s Assessment Team in June 2017, but was not publicly shared until January 2018 as part of a State Board of Education docket (though the informational topic wasn’t publicly discussed until March 2018). The Assessment Team began the process of developing a Request for Proposals (RFP) to move the procurement process forward in the fall of 2017. The RFP development allowed ODE to make more accurate cost projections that made it clear that the originally-published timeline to implement a new statewide assessment by 2018-19 was not feasible. ODE informed the field in January of 2018 that it would not be possible to move forward with the RFP without additional stakeholder support, as well as additional budget allocations from the Legislature. If stakeholder support
was present and the Legislature supported funding a switch in Oregon’s high school
accountability assessment to a nationally-recognized college entrance examination, the earliest possible operational administration would occur in 2020-21.

This report provides an overview of Oregon’s discussion surrounding the high school accountability assessment options in five sections, including the initial discussion as part of ESSA in Section 1, the RFI summary in Section 2, additional stakeholder engagement conducted in the winter and spring of 2018 in Section 3, the current state and national perspective on using nationally-recognized college entrance examinations in Section 4, and, finally, a summary and recommendations to the Deputy Superintendent of Public Instruction in Section 5.

This report is submitted by ODE’s Assessment Team, pursuant to the need to provide
documentation of additional stakeholder engagement as well as formal recommendations about next steps for ODE’s summative high school accountability assessment. The report also elaborates the manner in which staff recommendations respond to stakeholder concerns. …

Request for Information Summary

Responding to Stakeholder Feedback

The Standards and Assessment Work Group identified it as a central value that Oregon students deserve an assessment system whose costs in time, energy, and resources are in balance with real benefits to students and educators: timely, usable feedback on learning. In the absence of such benefits, the Work Group urged that we must dramatically reduce the costs in time, energy, and resources of summative assessments for systems accountability. As these benefits increase, more costs may be justified. …

In response to stakeholder feedback received through both the formal stakeholder work groups and the community engagement sessions described in Section 1 of this report, ODE included the following commitment in Section 3 of Oregon’s ESSA State Plan:

Oregon will pursue the flexibility under ESSA to allow districts to use a nationally-recognized assessment in place of the statewide summative assessment. While this process moves forward, ODE will continue implementing Smarter Balanced until another option is available and determined appropriate for local-selection.

ODE will establish a rigorous review process that includes:
• Involvement of key stakeholders
• Alignment to the learning standards
• Reliability and validity
• Comparability across schools and districts
• Accommodation and accessibility supports
• Clear performance targets set at appropriate levels …

May 2017 Request for Information (RFI)

Alignment to State Standards

The evidence a state submits… must demonstrate that a state’s assessment aligned to the full depth, breadth, and complexity of the state’s adopted content standards. The state must be able to provide evidence from a third party evaluation of standards alignment in order to meet federal peer review requirements. The submission of evidence completed by the state education agency or the test vendor is thus inadmissible (as it is clearly subject to confirmation bias).

While both ACT and the College Board asserted in their RFI responses that their assessments align to Oregon’s adopted content standards, neither vendor provided or referenced the availability of third party evidence demonstrating alignment. ACT’s claim of alignment cited that, “there is significant overlap between the Common Core State Standards and the college and career readiness skills that ACT measures. ACT tests are designed to measure student preparedness to achieve their academic and workplace goals” (ACT Response to May 2017 RFI). Similarly, College Board’s based its claim of alignment by stating: “The [SAT] is a profoundly
meaningful assessment that is thoroughly transparent and aligned to critical high school outcomes, best instructional practices, and the Oregon state standards.”
Furthermore, Oregon statute requires that the assessment system shall include criterion-referenced assessments including performance-based assessments, content-based assessments, and other valid methods to measure the academic content standards and to identify students who meet or exceed the standards. (ORS 329.486).

In response to the RFI question regarding whether the vendor’s assessment is criterion-referenced, both vendors responded affirmatively. However, neither vendor’s response indicated an accurate understanding or provided evidence demonstrating that their assessments were in fact criterion-referenced. Criterion-referenced assessments are designed-forward to compare student achievement to levels of mastery, while norm-referenced assessments compare student performance to other students. As an example, 100% mastery is
a desirable outcome for a criterion-referenced, or standards-based, assessment. Norm-based assessments are relative, so whatever achievement or gains are made are relative to the performance of other students. ACT responded that, “the ACT is a nationally normed, criterion-referenced college and career readiness assessment. A criterion-referenced interpretation of ACT scores is obtained through the application of ACT’s College Readiness Benchmarks.

Students, parents, and counselors can use the Benchmarks to determine the academic areas in which students are ready for college course work, and areas in which they may need more work.” Likewise, College Board indicated that, “each assessment in the SAT suite has an associated set of metrics called the college and career readiness benchmarks. The new college and career readiness benchmarks are based on actual student success in entry-level college courses.” …

Accessibility

The Standards and Assessment Work Group expressed a strong value in ensuring that all Oregon’s statewide summative assessments used for accountability purposes must provide all students with access to the same suite of accessibility supports offered through Oregon’s current statewide summative assessment.

Both ACT’s and the College Board’s RFI responses indicate that their accessibility offerings are more limited than Oregon’s current statewide high school summative assessment, which currently includes 54 supports, most of which are available to any student (a small subset of accommodations are available only to students on an IEP or 504 Plan). Moreover, the RFI responses indicate that the vendors apply a more restrictive process for approving the use of offered accessibility supports compared with ODE’s current practices. …

Whereas universal tools and designated supports (which make up the vast majority of Oregon’s current 54 supports) are available to all students with no specific eligibility or documentation criteria in place, both vendors indicated in their RFI responses that the decision to provide a student with accessibility supports are largely offered only to students with disabilities and are subject to vendor approval. Furthermore, both vendors indicated that in many cases, tests administered using state-approved accessibility supports would not result in a college-reportable
score. …

While ACT’s RFI response indicated that they could provide a full Spanish translation at an additional cost beyond their per-student cost, the College Board’s RFI response stated that no such translation was available, with only a word-to-word glossary option approved for English Learners. Neither vendor’s response indicated an option permitting students to respond in Spanish. …

Compliance with ESSA and IDEA Requirements

It is unclear what vendors might be responsive to an eventual Request for Proposals (RFP) if that path is selected. At present, no nationally recognized high school academic assessment has met USED’s Peer Review requirements. Current evidence suggests that ODE is correct to maintain a cautious approach to this potential shift, as misalignment, limitations regarding information about and provision of accessibility supports, and timing/scheduling challenges suggest that compliance… within ESSA is at least questionable. In addition, Individualized Education Program (IEP) teams, 504 teams, and EL planning teams would not have complete information about which accommodations do/do not violate the construct in the eyes of the vendor and thus lead to a non-college-reportable score. …

Equity

Related to accessibility, the Work Group strongly urged that Oregon’s statewide summative assessment must ensure that students who test using these accessibility supports are not penalized in any way and do not have their results treated differently for any reason, and that Oregon’s statewide summative assessment must offer a benefit for each student taking the test without differential treatment. Contrary to this recommendation, both vendors indicated that tests administered with certain accessibility supports may not be college reportable. Furthermore, both vendors indicated that they conduct their approval process on a case-by-case basis, which would make it challenging for IEP teams to make informed decisions about which supports to assign to individual students.

An additional equity concern arises in the context of test preparation resources and tutoring options. These resources are widely available for both the ACT and SAT to students with the financial resources to access those options. Proceeding with procuring such an assessment may therefore have inequitable implications for students without financial resources for multiple opportunities or external preparation options. …

Scheduling and Logistic Considerations

Based on the RFI responses submitted by ACT and the College Board, the testing conditions for these other assessments are more restrictive than Oregon’s current high school summative assessment in terms of the timing for scheduling test administration and the time allowed for students to complete the test(s). Oregon’s current test window allows districts to schedule testing to occur over a five-month test window, with individual test opportunities subject to a 20 – 45 day expiration period, maximizing local flexibility in scheduling and resource allocation and providing individual students the time they need to test at their own pace. By comparison, ACT responded that it would provide an opportunity for ODE to select an initial test date, a make-up test date, and an emergency test date, and the College Board responded that ODE may choose a primary and make up test date, with online test administration occurring over multiple days. In addition, both vendors responded that their tests are timed, meaning that students have a fixed amount of time to complete the test. These challenges could pose increased difficulty for schools to ensure 95 percent participation—a participation requirement established by the USED to ensure that statewide assessment results reflect how schools are
doing at serving all student groups, including those groups that have been historically underserved including Oregon’s students of color, students with disabilities, English Learners, and students experiencing poverty.

Cost

ODE staff’s evaluation based on the RFI responses indicates that the net biennial cost increase of implementing a new test for high school and discontinuing the current test would be approximately $5.8 – 6.8 million per biennium. In addition to the per-student rate identified by the vendors, this figure includes additional implementation costs, such as standard setting, an independent alignment study, and related activities that are necessary to document compliance with best practices in test development and provide documentation for the Title 1 Statewide Assessment Peer Review process. It does not include costs associated with any item development activities that might ensue should the alignment study indicate gaps in coverage
of Oregon’s adopted content standards. …

Implications for Essential Skills Graduation Requirements

Students are required to demonstrate proficiency in the Essential Skills of Reading, Writing, and Mathematics to earn a Regular or Modified Diploma. The State Board of Education has adopted three primary assessment options by which students can demonstrate proficiency in each of the Essential Skills: the statewide summative assessment; other standardized tests; and work samples (local performance assessments scored using the official state scoring guides). The vast majority of students meet their Essential Skills requirement through the statewide summative
assessment. Of the students who graduated in 2015 (in the four-year cohort), 92% of students met the reading requirement through the statewide summative assessment, 67% of students met the writing requirement through the statewide summative assessment, and 81% of students met the mathematics requirement through the statewide summative assessment. Conversely, only 3 – 4% of students met the Essential Skills requirement using another standardized assessment (including the SAT and ACT, among others).

The Essential Skills graduation requirement is a high-stakes policy for students, and the Essential Skills and Local Performance Assessment Manual states that the validity of the assessment results depends on each student having appropriate accessibility supports when needed based on the constructs being measured by the assessment. A student’s ability to access the same accessibility supports received in instruction, to work with the school team to determine which day is optimal for the student, and to take as much time as they need to complete the assessment have all been helpful features of the current statewide assessment in reducing the impact of test anxiety. ACT’s and the College Board’s RFI responses indicate that for their assessment systems some or all of these conditions may be more restrictive.

Adopting a nationally recognized assessment such as ACT or SAT as Oregon’s statewide high school summative assessment would result in all high school students having a required, state-financed opportunity to receive college-reportable scores. However, both ACT and College Board stated in the RFI responses that the use of certain accessibility supports invalidate results….

Additional Stakeholder Engagement

Background

ODE conducted a comprehensive stakeholder feedback process in the winter and spring of 2018 in order to supplement the initial feedback gathered from the ESSA Standards and Assessment Workgroup in 2016 and the Assessment Advisory Committee in 2017. The Standards and Assessment Workgroup made six recommendations for ODE to consider. Recommendation #5 was that “Oregon should support options and explore how to allow districts to use individual
high school student flexibility of the summative assessment. All state-approved, nationally-recognized assessments need to provide comparable data that allows for statewide student performance evaluation. If flexibility is not available at the individual student level, then Oregon should explore how to allow districts to use another state-approved, nationally-recognized assessment in place of the high school statewide summative assessment”…

However, in the April 2017 Education Update from ODE, the Deputy Superintendent announced that “… stakeholders overwhelmingly have urged the state to explore the option of using a nationally recognized college readiness assessment such as the ACT or SAT, in place of Smarter Balanced, as the high school accountability measure.” The article also states that “… we are prepared to replace Smarter Balanced at the high school level with a nationally recognized assessment for the 2018-19 school year.” ODE’s message was not to allow for the flexibility defined in ESSA, or to pursue flexibility at the individual student level, but to replace the Smarter Balanced assessment for all students statewide. …

Though it is not possible to determine what rationale may have been used for this decision, it is clear that allowing districts to use the ACT, SAT, Smarter Balanced assessment, or some other nationally-recognized college entrance examination at the district level would have led to some incredibly challenging, if not insurmountable, measure comparison hurdles. Oregon’s accountability system would be challenged to evaluate the performance of students taking one of the assessments against students taking another. The assessments quite simply are not
measuring the same constructs, nor do they measure academic achievement in the same manner. They are not aligned to Oregon’s content standards in the same manner. ….

High school students & principals, parent advocate groups, and teachers

Over one hundred high school seniors who opted out of the high school assessment as juniors were consulted regarding our high school assessment. The group, as a whole, was not informed about the purpose of the statewide assessment system nor the ways in which ODE uses the data. Only one reported having a substantive discussion with a parent regarding reasons for opting out of the assessment. The students all shared that they would have been much more interested in taking the Smarter Balanced assessment if it could be used for college admissions decisions. They also shared that they would have felt an ethical reason to participate if they had understood that the test results were used to identify students whose education systems were not meeting their needs. Students general felt overwhelmed by testing in junior year, primarily because the testing all occurred at the same time period, in late April/early May, “Junior year, I think all of my classmates will agree, is incredibly stressful. ACT, AP Exams, exams for courses. It’s almost impossible.”

The high school principals consulted generally stated that they recommended the switch to an ACT/SAT whether or not they had personal or professional reservations about making such a recommendation. They felt that their stakeholders wanted the ACT/SAT so they needed to represent their stakeholders’ desires. Principals emphasized the impact of the opt out practices in their locations, noting that students had already met their Essential Skills by taking the PSAT in many cases, and simply did not need the Smarter Balanced assessment results. The one exception was in writing, which is not offered on the PSAT. This likely explains the participation rate differences between ELA and mathematics, at least to some degree, at the high school level.

Parent advocates involved in these discussions were generally against standardized testing altogether, though some recognized the need and utility for such systems for use by state and federal policymakers to protect civil liberties and ensure appropriate use of public funds. The group was most interested in discussing work on formative assessment practices and performance-based assessments.

The teachers and educators who offered official comments were generally opposed to switching to the ACT/SAT, with two exceptions. The majority supported maintaining our current system because it is aligned with our standards and they had worked for years to adjust their curricula and instructional approaches to match the standards; they felt that the Smarter Balanced assessments exhibit strong alignment to Oregon’s adopted content standards and should be maintained. Two teachers who supported the switch identified concerns regarding test scheduling, how long the assessments take, and the lack of engagement from students because they do not have any perceived benefit from participating. One respondent stated
that, “I personally feel like SBAC [sic] should be eliminated and not replaced with any standardized test.” The same teacher stated that the typing required on the Smarter Balanced assessment created educational inequities against underserved populations, as did the allowance for students to take as long as they need to complete the assessments (because they miss instruction while they are completing their tests, though classmates who finish earlier do not). Another respondent identified challenges with test scheduling and noted that students do not need Smarter Balanced to meet Essential Skills in many areas because ODE allows for the
use of many other standardized tests for this purpose, in addition to work samples.

As mentioned, most educators who provided public comment wanted to maintain the Smarter Balanced assessments. They shared statements like the following, “As someone who has been in math education for many years, I feel like we finally have an assessment that measures mathematical thinking AND skills with Smarter Balanced.” Others expressed consternation at the original implementation timeline shared, noting that transitioning to ACT/SAT or some other college entrance measure by 2018-19 was simply not feasible. One member who had been heavily involved in the nationally-recognized college entrance examination discussion as a district test coordinator shared that “There was never a single instance when it was stated,
hinted, implied, or alluded to that Smarter Balanced would be dropped altogether for high schools. When was this decision made and by who [sic] and who were the actual stakeholders that gave input? I have not yet been able to identify a single person who was involved with or consulted about this decision and I have asked everyone I could possibly think of, including members of official ODE advisory committees, who may have had even a slight level of participation.” This submission validated ODE’s decision to conduct additional stakeholder engagement, as the prior process did not appear to have been fully inclusive nor transparent.

Our state stakeholders did not support a switch to a nationally-recognized college entrance examination. …

As far as opting out goes, Oregon is dipping below that federally-mandated 95 percent participation:

…Many stakeholders expressed concerns about statewide assessment opt out rates. The opt out rates are highest at the high school level, where Oregon saw a 94.5% participation rate at the high school level in English language arts and mathematics in 2016-17. The following school districts had high schools with 0% participation rates in mathematics last year: Annex, Bend/La Pine (La Pine Senior High School), Eugene 4J (Twin Rivers Charter School), Lincoln County (Newport High School & Toledo Senior High School), and Redmond (Redmond Proficiency Academy). This level of participation raises several concerns and must be addressed. ODE should consider district opt out practices to ensure that they are consistent with ODE
policy….

According to ESSA, states are expected to “remediate” districts into meeting the 95 percent participation in math and ELA assessments. Whether or not the phrase, “consistent with ODE policy” becomes a euphemism for pressuring Oregon districts to *make* students take math and ELA tests remains to be seen. But forced test participation is the federal game, with states being required to force districts (and to alter state policy in favor of the federal mandate), and districts put in the position to force students.

Here’s how July 11, 2018, Oregon Public Broadcasting reports it:

The state is concerned about students continuing to opt out in large numbers, particularly at the high school level.

Officials said they’ll “renew” efforts to increase participation.

Will it be carrots? Sticks? Stick-wielding carrots?

Stay tuned, Oregon.

carrot-with-stick

________________________________________________________________________________________

Want to read about the history of charter schools and vouchers?

School Choice: The End of Public Education? 

school choice cover  (Click image to enlarge)

Schneider is a southern Louisiana native, career teacher, trained researcher, and author of two other books: A Chronicle of Echoes: Who’s Who In the Implosion of American Public Education and Common Core Dilemma: Who Owns Our Schools?. You should buy these books. They’re great. No, really.

both books

Don’t care to buy from Amazon? Purchase my books from Powell’s City of Books instead.

2 Comments
  1. Threatened Out West permalink

    Off topic, Mercedes, but since you recently wrote about Utah being denied its ESSA plan because Utah state law allows parents to opt out. Because of this, Utah will now be required to count all opt-outs as failing scores: https://www.sltrib.com/news/education/2018/07/13/students-who-opt-out/

Trackbacks & Pingbacks

  1. Oregon High Schools: Smarter Balanced, No, Wait, ACT/SAT, No, Wait, Smarter Balanced. But Those Opt-Outs… — deutsch29 – Nonpartisan Education Group

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s