Is the US Department of Ed's Advice to Judge Screen Products by Evidence Helpful? Not Really

The latest advice provided by our US Department of Education advises educators to judge educational technologies by evidence of impact on learning, but there’s one problem, where do we get that so-called “evidence.” From the companies themselves? That’s a horrible idea

“…instructional technologies should be judged by their impact on learning, not simply whether they happen on a screen,” said Assistant Secretary of Elementary and Seconday Education Kirsten Baesler, in her attempt to salvage the status of educational technologies.

The problem is finding such evidence. Sure, you’ll get bombarded by “testimonials” from other districts about how effective a product is, but a pal of an owner of a ed tech company hardly counts as evidence. Besides, testimonials are not reliable and valid evidence. Too much room for corrupting influences in those.

What about any studies the ed tech company provide in their sales pitches? Well, I would not get too excited about that either. According to a 2019 Hechinger Report article, “The Dark Side of Education Research: Widespread Bias” by Jill Barshay, we need to scrutinize any research companies provide as evidence of their product’s effectiveness. Surprise, it is too often biased toward their own products.

Like Big Pharma, who has been guilty of generating their own faulty, self-promoting research for their products, Ed Tech companies and other educational resource companies have been operating from this playbook too.

This 2019 article examined a John Hopkins University study of 30 years of educational research by developers of educational interventions and products. These companies conducted their own research or paid someone to do the research, and most of the time, the results commonly showed greater benefits for students than when the research was independent. “On average, the developer research showed benefits—usually improvements in test scores—that were 70 percent greater than what independent studies found.”

In other words, studies done by or funded by the companies themselves were biased. They always made the product look better than independent studies did.

So, the US Department’s admonition that educators and education leaders “look to the evidence” before banning a screen product will not help at all. Most products, if there is any evidence, it is either testimony from another district, which is highly unreliable, or it is their own insider research which can’t be totally trusted.

Most companies, after all, would not want independent research because you can’t control the outcomes. Who wants to find out their product is garbage?

So what do we do with the US Department of Ed advice? Not much it seems, for independent, reliable research on these products is rare and mostly non-existent.

With school district technology employees taking free training from the likes of Google, Microsoft and others, can you really trust anything they say about the products either?

I might add as well, school technology conferences are also heavily underwritten by technology companies, so…well, not much help there either.

What do we do then as educators to determine whether a screen product is effective at what it does? Without independent research, I would conduct small-scale in-school studies myself before implementation to determine effectiveness. If enough educators did this and shared results, then there could be a pool of independent knowledge about these products. But it is going to require school technology educators become critically independent.

As expected, the US Department of Education’s advice about technology is about useful as a study provided on the safety of cigarettes by tobacco companies.