The College Board and Its Doomed Agilix Connection

On December 12, 2016, Reuters published an extensive article about David Coleman’s botch job in trying to fast-track a “new” SAT that was supposed to be aligned with the Common Core State Standards (CCSS). Based upon a number of College Board internal documents, the Reuters article notes that a number of College Board execs and legal advisors tried to get Coleman to slow down and rethink some components of his “vision”; such is true of the excerpt featured in this post involving online test delivery using the ed tech company, Agilix:

Before Coleman arrived, the College Board had been developing digital testing for more than two years, internal memos show. Offering tests online was complex, veterans warned Coleman. It would require extra research and logistical chores, such as arranging computers for exam-takers. Moreover, creating a digital SAT by March 2015 “is almost certainly not achievable,” [College Board Senior VP Hal] Higginbotham wrote in his note to Coleman, “and definitely not in a responsible manner adhering to good measurement practice.”

Coleman pressed on. In early 2013, he scuttled the College Board’s internal digital project and brought in an outsider to lead the effort: Mark Luetzelschwab, who had helped run an education technology company called Agilix.

That July, Luetzelschwab touted how his former company would help execute the digital undertaking, documents show. Using Agilix software, the College Board would meet its goals “faster, cheaper, and (with) less risk than building” a digital platform internally, he wrote in an email to Jeremy Singer, Coleman’s new chief operating officer.

Coleman’s team worked out a deal with Luetzelschwab’s old firm: a no-bid contract for as much as $30 million, documents show. Other top College Board officials protested.

Three College Board technical specialists warned in detailed memos that Agilix lacked the expertise to deliver the technology it promised. Lawyers didn’t like the deal, either.

General counsel Neil Lane urged Singer to abandon talks with Agilix; Lane viewed the large no-bid contract as a conflict of interest, according to emails. Another lawyer, John Newman, circulated a memo arguing that a contract with Agilix could be seen by the Internal Revenue Service as an illegal diversion of funds by creating a “private benefit” for Agilix. That, Newman believed, could jeopardize the organization’s tax-exempt status.

Coleman stood firm. The College Board signed the deal in late 2013 and provided Agilix with a $3 million upfront payment, according to a document.

Lane, who currently is a special counsel at the College Board, declined to comment. Newman no longer works for the College Board and declined to discuss his memo.

Publicly, Coleman promised to roll out an online version of the test. Four months into the contract, however, Agilix had failed to deliver its first batch of work on the project, a document shows. By mid-April 2014, the College Board notified Agilix that it was in breach of contract. Agilix was fired months later. But it was allowed to keep the $3 million upfront payment, a document shows.

Curt Allen, the CEO of Agilix, declined to comment.

Luetzelschwab, who was fired a few months later, declined to comment on his time at the College Board.

According to Luetzelschwab’s Linkedin bio, he was Senior VP of Digital Solutions at the College Board from March 2013 to September 2014 (19 months). As one might expect, there is no mention in his Linkedin bio of his having been fired. He describes his College Board job duties in successful terms:

Led new digital product development while transforming existing disparate systems to a common technology framework with common design and purpose. Researched new models for scoring student work to reduce costs and increase reliability. Managed a $10-12M budget. Member of senior management team.

Prior to College Board, Luetzelschwab was Senior VP of Product Development and Marketing at Agilix (February 2008 to march 2013– hence the conflict of interest noted in the Reuters excerpt). He describes his Agilix job duties thus:

Transformed company from quiet technology group to dynamic, highly-respected, rapidly-growing education technology organization. Created disruptive business model around learning infrastructure including finding new customers, delivering solutions, and changing the conversation. Increased services revenue 300% in one year while landing over $15M of new long-term license and services business. Envisioned and led creation of a world-class services team that combines academic effectiveness and elegant design with robust technology. Created BrainHoney – an award-winning learning management system used across the globe and Buzz – a radically different personalized learning model being deployed at scale in some of the toughest schools in the country.

And so, even after he left Agilix, Luetzelschwab continued to “find new customers” for Agilix: the College Board. He also assisted Coleman in “disrupting” the College Board’s business model, which, it seems, later disrupted his employment at the College Board.

In searching for details of how exactly Agilix billed itself as the disruptive solution for eager-to-disrupt top-downer Coleman and the College Board that he leads, I found this Agilix presentation, which was ironically first posted on April Fools Day 2013 and reposted April 02, 2013.

Below is the content of the Agilix/College Board presentation slides dated April 02, 2016. I do not know if the College Board agreed to the plan as it is specified in the presentation; however, the content gives a good idea of the “vision” Agilix wanted the College Board to purchase.

The plan is ambitious (intrusive? arrogant? exploitative?). At points it reminds me of the initially-federally-funded PARCC and SBAC testing consortia, which now face their own complications. (SBAC is shopping for a new fiscal agent, and PARCC is seeking a survival makeover.) It also mentions marketing to “large foundations” including Gates, and it plans to collect massive amounts of data on students to, uh, help students and to “expand customer base.”

Teachers also figure into the Agilix-College Board plan as grunt workers producing and validating test items, and the plan also notes wanting to lure “cash strapped” school districts into using its products and services.

There is much more. I’ll leave readers to view the rest for themselves.

Regarding Agilix’s actual slides: I would find it difficult to trust this company from the outset with such a complex plan (and obviously expensive venture) given the numerous typos/inconsistent capitalization and punctuation in the presentation itself. (I transcribed the content of the slides, typos/inconsistencies included.)

Agilix purports to be a data-driven mastermind and is clearly pitching for a contract worth millions, but it cannot clean up a brief slide presentation purposely on display for public viewing. Really?

Proceeding:

It seems that the title of the presentation is “The Big Idea: College Board Innovation” (hard to tell; slides move as part of presentation, and file is untitled).

The presentation is long and revealing. From this point in the post, I let it speak for itself. However, as you read of the marvels of an Agilix-transformed College Board, keep in mind that according to Reuters, Agilix failed to meet its first College Board work deadline and was fired. So much for its “big idea.” Still, even though the College Board under Coleman is floundering, I would not dismiss the ideas below as ones the College Board might not still try to institute at some point.

Here we go:

Slide 1:

The Big Idea

Slide 2:

I want to share a vision and a path for College Board which I believe will completely change the market. I want you to…

Slide 3:

Think Outside the Test (You are here.) [Schneider’s note: “Here” is inside of the “big idea” light bulb.]

Slide 4:

Here’s how we see it…

Traditional film: 7 steps

Polaroid: 5 steps

Digital: 3 steps

How did this happen?

Technical innovations made incremental improvements.

Innovation: Polaroid film (Linear increases over time in performance, market share, and total images.)

Disruptive innovation: Digital (Exponential increases over time in performance, market share, and total images.) Changed the market!

DIGITAL = CHEAPER & BETTER

*Digital displaced film

*Companies, like Kodak, too slow to react, are now finished

Slide 5:

R.P.C. Factor

Replace: AP Courses/ Dual Credit

REPLACE ONE EXPERIENCE WITH ANOTHER

Near Term:

*Student takes dual credit class.

*How do we establish that the rigor, scope and validity meet those of the credit-granting institution?

Long Term:

Provide MOOCs, [Schneider’s note: massive online open courses] open educational resources (OER) and other course providers with established, repeatable validation processes to certify equivalence with credit-granting institution

Prepare: PSAT, SAT / Indicator of Readiness

PREPARE AND ASSESS STUDENT READINESS

Near Term:

*How do we know students are prepared for an opportunity?

*Move to provide online assessments that span formative (and high-frequency & customizable?) to high stakes gateway assessments

Long Term:

*create communities that contribute, improve, and validate assessments, questions, item banks

*Create large datasets for customers’ data warehouses, research projects, etc. to expand customer base

*Create a trust network that connects, contributes, and validates information in an ever-growing pool of data for institutions, students, programs, etc.

Connect: Big Future / Link People and Opportunities

CONNECT STUDENTS WITH OPPORTUNITIES

Near Term:

*use increasingly comprehensive profiles of students to match to college admissions requirements, scholarship requirements, etc., as a service to those institutions (*might a direct-to-student service be another opportunity?)

Long Term:

*place or admit students, based on defined criteria, into institutions that outsource that process to College Board.Create longitudinal data systems that track student progress to identify students likely to succeed beyond admission.

*Admit/place students for the institution with a confidence level for matriculation and eventual graduation. (admit students likely to succeed)

Slide 6:

Not just any solution… Digital Solution

Online Assessment

Near Term:

*Begin with providing the system, not the data

*Create a community of teachers with access to test/item banks (membership-based)

–Teachers use and contribute questions, and moderate/validate (with College Board guidance, or CB identifies mentor/leaders to start?)

–Use for class prep, assessing student progress, adjusting instructional strategy

–Alternative to PNP (what does this mean?) [Schneider’s note: pencil and paper]

–Create a compelling service for cash-strapped ISD [Schneider’s note: independent school districts?]

Long Term:

*Assessments span formative to high stakes, allowing customers/users to:

–Assess with high frequency, validity, reliability

–Customize, use & share formative assessments

–Extract and integrate data from a flexible CB framework into members’ own systems of reporting, response systems, grading, etc.

*Contributor community grows into a Trust Network that is:

–Self-moderating, promoting relevance and quality

–Continually improving and growing data rich

*Build an extensive customer base for subsequent products, services, and data/reporting

Unified Web

Near Term:

*Simplify/Reduce content

–Single point of entry

–Integrate/unify 130 sites

–Simplify navigation for users

Long Term:

*Respond to Changing User

–Build profiles/roles of users according to age/stage (e.g., MS student pre-AP info –> HS student AP, college & scholarship –> college student certification & career readiness)

–Map next and future needs and goals for each profile/role

–Present content relevant for the user as they progress through age/stage/grade

*Unified engagement model

*Focus on Connections to Opportunities

Technology Advancement

Near Term:

*Improve Processes

–Trust Network development

–R-factor work

–Increasingly comprehensive assessment of student preparedness for opportunities (expand data points that measure & indicate student readiness, achievement)

Long Term:

*Reduce Costs Over Time

–AI scoring methods, not manual

–Distributed scoring (scan/route/manage scoring processes)

*New Models/ New Markets

–Even beyond US. Global market, learning center networks

–Customers for this expanded portfolio of services include ISDs, Higher Ed, students, content/course providers, scholarship & other opportunity programs, research entities, for profit educational institutions, employers, certification entities, …

Data

Near Term:

*Improve Processes – Free the Data

*New models/ new markets

–Teacher: “How did my students do on the PSAT, or their formative assessments?” to course correct for student mastery

–Services to institutions, providing data in usable format for their own use in their own systems (e.g., integration into their own data warehouses, for use in IEP’s and RTI efforts,reporting for their grant applications and required data submissions)

Long Term:

* Reduce Costs Over Time

–Community helps develop and validate content

–Network “learns” through relevance and ratings,

–Best content and items in bank are field tested

–Becomes a working pilot/beta group for CB

*New Models/ New Markets

–Sell growing and improving/evolving data sets to research entities

–Attract large customers such as state DoE’s and foundations (Gates, Dell, etc.)

Communications

Near Term:

*Initial branding efforts

*Close articulation with customers: schools, institutions, individuals in key roles that CB needs to understand

–Teachers

–Counselors

–Admissions staff

–Students

–Program leaders

–Certification Entities

–Employers

Long Term:

*Create advisory groups

*Implement pilot groups

*Refine branding efforts

–based on improved understanding of customers

–focused on changing products and services

*Provide educational initiatives to drive awareness of, and participation in, College Board’s changing roles and offerings

–(***This will be key!*** important to do right & sustain)

Our Approach

*Simple

*Mobile

*Accessible

[Audience: executive mgt and non-technical staff]

Reduce score, cost, time; improve user experience with laser focus on core requirements. Ensure all audiences can access from all devices.

*Extended

*Iterative

*Insightful

[Audience: product/program managers development staff, mid-management]

Plan on longer engagements with multiple iterations with planned changes to reflect user input and changing market requirements. Don’t squeeze everything into one product cycle. Use data to drive decisions. Focus on “jobs to be done” not features.

*Secure

*Scalable

*Usable

[audience: IT staff, developers. Executives may just assume we will do these things].

Secure systems that survive rapid growth. Designs that ensure user success with minimal number of clicks

*Automated

*Distributed

*Humanized

[audience is everyone. Let them all know plan is to use technology for what it’s good for: automating manual stuff, creating efficiencies, allowing people to work at higher order thinking, including collaboration.]

Gain efficiencies through automation. Distribute load among the community participants. Humanize interactions to leverage peer-peer communication instead of college board to customer interaction.

“CONSENSUS IS THE ENEMY OF COMPLETION.”

Consensus delays projects, adds cost, time and complexity and disregards the user.

“WHATEVER IT TAKES”

Relentless focus on getting products out the door on time while not sacrificing security, scalability, or usability.

“HOW HARD CAN IT BE?”

We will embark on many projects that may seem impossible. Trust in our experience, ability, and willingness to think outside of the box to achieve a solution.

“CENTRALIZE WITH FLEXIBILITY”

All design, usability, foundation of data, and interconnectedness of data around personas (students, teachers, etc.) in the system will be tied back to a unified framework. Might even steal the XLI motif here.

“FREE THE DATA”

Get into common formats, FERPA compliant, share raw data. Get the data into the hands of the people so they can make their own decisions as quickly as we can.

How hard can it be??

Agilix fired.

Luetzelschwab fired.

College Board staggering clumsily.

That’s how hard.

___________________________________________________________

Released July 2016– Book Three:

School Choice: The End of Public Education? 

  (Click image to enlarge)

Schneider is a southern Louisiana native, career teacher, trained researcher, and author of both A Chronicle of Echoes: Who’s Who In the Implosion of American Public Education and Common Core Dilemma: Who Owns Our Schools?.

Don’t care to buy from Amazon? Purchase my books from Powell’s City of Books instead.