Can You as an Educational Leader Trust the Evidence of Effectiveness Provided by the Product Developer? Probably Not

Can educators and educational leaders trust the research that companies provide about their products? Should they? Probably not. The tech industry that includes Ed Tech has a major credibility problem in the middle of the calls for screen bans and screen limits. For example, there is growing independent evidence that social media is harming our children. Why else would Facebook settle their massive lawsuit.

I just completed reading Max Fisher’s 2022 book “The Chaos Machine: The Inside Story of How Social Media Rewired Our Minds and Our World” and it clearly describes why we can’t trust research provided by companies on their own products. The book reinforced why I quit Facebook and Twitter sometime ago. But one particular incident in the book illustrates clearly to me why companies just can’t be trusted to provide unbiased, reliable evidence of their own product’s effectiveness or lack thereof.

In the book, Max Fisher points out that Facebook completed an internal study of their technology’s effects when faced with growing suspicion that their site might be making the political divisions in the US worse. Their own research warned internally that the platform was doing exactly what their company’s leaders denied it was doing. 

Their own leaked research said “Our algorithms exploit the human brain’s attraction to divisiveness.” According to Fisher, this was in a 2018 presentation leaked to the Wall Street Journal. This presentation also said that Facebook’s systems were designed in a way that delivered users “more and more divisive content in an effort to gain user attention and increase time on the platform.” 

Later, this research was hidden by Facebook executives and they rejected the recommendations for tweaking the algorithm that would have resulted in less time spent online. They, in other words, deliberately chose to keep users addicted to divisive content that their own machine served up to them.  

This is why companies’ research can’t ever be trusted. Neither can their executives. This is also why educators can’t trust research either done by or paid for by the companies whose products they are considering to purchase.

Companies simply aren’t going to provide educators and educational leaders with reliable, independent evidence that their products are effective. They are also going to hide studies that are less than supporting of their products. I would expect no different, after all that would prevent sales.

Facebook’s own research showed their product was harmful, yet they buried the evidence. They made the deliberate choice to keep their product addictive.

As an educator, I always ask for independent evidence of product effectiveness. Independent being evidence generating by studies not done or paid for in anyway connected to the company. Most often, none exists.

If such evidence does not exist, then   ask the company to allow you to run a study of their product on a trial basis. Use research methods to evaluate the product for what they claim. Once you have that, publish it and make it known. Share it with other individuals.

I can envision a whole network of school technologists who independently evaluate these products and share that evidence independently and not as a testimonial asked for by the company. 

Where the evidence comes from, who paid for it, who conducted the study that generated it, and why the evidence is being provided all matters.

This is what has been lacking in the massive technology acquisition and placement in schools. Now, our parents and common decency requires us to do things differently.

Is the US Department of Ed's Advice to Judge Screen Products by Evidence Helpful? Not Really

The latest advice provided by our US Department of Education advises educators to judge educational technologies by evidence of impact on learning, but there’s one problem, where do we get that so-called “evidence.” From the companies themselves? That’s a horrible idea

“…instructional technologies should be judged by their impact on learning, not simply whether they happen on a screen,” said Assistant Secretary of Elementary and Seconday Education Kirsten Baesler, in her attempt to salvage the status of educational technologies.

The problem is finding such evidence. Sure, you’ll get bombarded by “testimonials” from other districts about how effective a product is, but a pal of an owner of a ed tech company hardly counts as evidence. Besides, testimonials are not reliable and valid evidence. Too much room for corrupting influences in those.

What about any studies the ed tech company provide in their sales pitches? Well, I would not get too excited about that either. According to a 2019 Hechinger Report article, “The Dark Side of Education Research: Widespread Bias” by Jill Barshay, we need to scrutinize any research companies provide as evidence of their product’s effectiveness. Surprise, it is too often biased toward their own products.

Like Big Pharma, who has been guilty of generating their own faulty, self-promoting research for their products, Ed Tech companies and other educational resource companies have been operating from this playbook too.

This 2019 article examined a John Hopkins University study of 30 years of educational research by developers of educational interventions and products. These companies conducted their own research or paid someone to do the research, and most of the time, the results commonly showed greater benefits for students than when the research was independent. “On average, the developer research showed benefits—usually improvements in test scores—that were 70 percent greater than what independent studies found.”

In other words, studies done by or funded by the companies themselves were biased. They always made the product look better than independent studies did.

So, the US Department’s admonition that educators and education leaders “look to the evidence” before banning a screen product will not help at all. Most products, if there is any evidence, it is either testimony from another district, which is highly unreliable, or it is their own insider research which can’t be totally trusted.

Most companies, after all, would not want independent research because you can’t control the outcomes. Who wants to find out their product is garbage?

So what do we do with the US Department of Ed advice? Not much it seems, for independent, reliable research on these products is rare and mostly non-existent.

With school district technology employees taking free training from the likes of Google, Microsoft and others, can you really trust anything they say about the products either?

I might add as well, school technology conferences are also heavily underwritten by technology companies, so…well, not much help there either.

What do we do then as educators to determine whether a screen product is effective at what it does? Without independent research, I would conduct small-scale in-school studies myself before implementation to determine effectiveness. If enough educators did this and shared results, then there could be a pool of independent knowledge about these products. But it is going to require school technology educators become critically independent.

As expected, the US Department of Education’s advice about technology is about useful as a study provided on the safety of cigarettes by tobacco companies.

A Career Caught in the Swirl of Ed Reform: What I’ve Discovered

My entire career as an educator, all 27 years, has been spent in the perpetual swirl of “reform” that enveloped education when I first set foot into the classroom. It was in the cusp of the “Era of Accountability” that I began teaching, and the clarion calls for site-based management, uniform standards, and testing were just beginning to resound. Soon to follow was the Total Quality Classroom movement, multiple intelligence theory, and right-left brain theory, critical thinking teaching, thinking maps, and whole host of other initiatives. There has been no shortage on theory during my career as an educator that’s for sure, and during my entire career, we’ve been reforming education, then reforming our reform in an unending pursuit of a “magical land” where schools succeed, except that there’s been one major problem: we’ve never arrived.

I don’t mean to sound pessimistic in this reflection; in fact, I’m not really that way at all. I am nostalgic in one sense, because there has always been that anticipation of the next great idea that comes around the bend, and the promise that all our educational ills will finally be resolved. Those who’ve promoted this atmosphere of perpetual reform, have, after all, succeeded even if our schools may not really be any better off. It’s those who’ve capitalized on these reforms by promoting products, professional development, computer programs and websites, and new techniques and strategies who have earned a bundle. The promise of their being one single way to resolve the educational puzzle has led many to search high and low, and our market-based approach to these products has not disappointed, at least for those who’ve made the money.

Still, I’ve come to a cold, hard conclusion that is, in fact, very liberating. It is simply this: There is no magical theory out there or discovery that will allow us to suddenly be able educate like we’ve never done before. There is no one best way to teach, and as we already know, there is no one best way to learn. Despite all these infernal emails I get that promise to "raise my students’ ACT scores or SAT scores to exorbitant heights," in the end their promises are more marketing than reality, and in many cases, downright deceptive. Education has become a money making enterprise like everything else, with “experts” arising from all corners of the field with their version of the “final solution to all our education problems.”

My liberating conclusion that all of these are mostly empty promises frees me to view education as the difficult work it is with problems that do not, nor ever will have singular solutions.

Reform has become such a cliche now, every time I hear a politician say the word, I want to flee in panic, or hit him with a rotten tomato. It just won’t happen. Perhaps real “reform” will begin and end with ourselves rather than continuing the fruitless quest for magic. Real reform begins with the liberating thought: “There are no easy answers or solutions to discover about our educational system. There’s only hard work to be done."

Are the Concepts of 'Grit' and 'Mindset' Attempts to Erase Importance of Social Justice & Equity?

While reading an essay entitled "Foucault, Power, and Organizations" by Stewart Clegg, I have begun to write and congeal thoughts about the new embrace by educationalists of the ideas of teaching students about "grit" and "mindsets." More and more books you pick up on educational methods and teaching practices seem to increasingly refere to Carol Dweck's ideas about "mindsets" and their role in success . (I have read her book, Mindset: The New Psychology of Success.) Then there's all this discourse about "grit" as well. Though this concept goes back to Francis Galton, Duckworth, in her book, The Power of Passion and Perseverance, has more recently brought this term to the forefront.
When reading Clegg, he writes: "Bio-power normalizes through discursive formations of psychiatry, medicine, social work and so on. The terms of these ways of constituting the normal become institutionalized and incorporated into everyday life. Our own reflective gaze takes over the disciplining role as we take on the accounts and vocabularies of meaning and motive that are available to us as certain other forms of account are marginalized or simply erased out of currency."
Some questions:
Is the employment of the now "psychological concepts of 'mindset' and 'grit' a means of constituting a new normal using the psychological and educational sciences to marginalize ideas of social injustice and inequity?
Is the employment of these concepts in the educational apparatus a means to erase any thoughts or ideas of inequity and social injustice from our society?

Here's some my of my working thoughts on this matter:
The discourse of "grit" and "growth mindset" could function as a discourse that seeks to install a 'reflexive gaze" into students that asks them to disregard their circumstances in life, their experiences of poverty, misfortune and lives lived in inequity, and "get with the program."
It is an explicit "scientific" manifestation of the "bootstrap mythology" that propagates the idea that "if you work hard, then you will be successful."
It is a reflexive gaze which banishes any thought of inequity in society. It is directed at the souls of students to make them docile and compliant with the educational program.
It conditions individuals to ignore inequity in society and allows those who continue to stack the economic system in their favor to retain their pre-eminence.
It attempts to dispel any resistance to a socially unjust society. Ultimately it is a application of the psychological and educational sciences to the service of disciplining those who question the injustice of society.
I think perhaps before we jump on the "mindset" and "grit" teaching methodology bandwagon, we might want to ask some of these and other critical questions about what they really are doing with our students and our society.