What Happens When Schools and School Districts Use VAMs to Make Decisions about Teachers?

Many school administrators are using value-added measures in making decisions about teachers as if these statistical measures represent the latest, settled and unquestionable science. Those who do this are making a grave error. Despite companies such as SAS, who peddle their EVAAS data systems as the salvation of public education, the science behind VAMs is not settled, and there is even enough doubt about them, that the American Statistical Association issued a strong statement in 2014 against their use in decision-making when it comes to teachers. In that statement, ASA reminds educators that:

VAMs typically measure correlation, not causation---positive or negative---attributed to a teacher may actually be caused by other factors that are not captured in the model. (ASA Statement on VAMs)

Yet, administrators still use VAMs to infer that the teacher causes those scores. SAS, who owns the EVAAS model that North Carolina pays millions of dollars for each year, arrogantly claims that it accounts for all the factors that cause student performance on test scores, even when psychometric experts caution that this isn't possible.

In addition, administrators, who use VAMs to make decisions about teachers, should know better than confuse correlation with causation, but any time they base decisions about teacher status using VAMs, they are automatically assuming that teachers cause test results. If teachers operated in a lab where they controlled all the conditions of learning and the subjects of their learning, then one could perhaps better make this inference.

But there are other concerns about VAMs too. In a recent study by Shen, Simon, and Kelcey (2016), it was found that "using value-added teacher evaluations to inform high-stakes decision-making may not make for a good teacher." Using VAMs to decide the status of a teacher may not have the long-term impact administrators desire. These researchers also recommend that VAMs not be used "to inform disincentive high stakes decisions," which are any decisions regarding the professional status of teachers.

Ultimately, though, I can't help but wonder if those who are sold on using VAMs in administrative decision-making aren't caught up in chasing short-term gains in a measurement that lacks any meaningfulness in the long-term. VAMs aren't settled science. Yet, administrators use that data as if it were. Any decisions made using this data should be balanced with other data.

Shen, Z., Simon, C., & Kelcey, B. (2016). The potential consequence of using value-added models to evaluate teachers. eJournal of Education Policy, Fall 2016.


NOTE: My just completed dissertation was on the practice of using value-added measures to determine teacher effectiveness. My plan is to share over the next several weeks and months my own insights and personal thoughts on this practice. This is the first of may posts I plan to share on this topic. 

Did SAS Provide Gifts to UNC System for Special Future Considerations of its EVAAS System?

It seems, according to an article, software giant, SAS, who already has a multi-million dollar contract with North Carolina Public Schools for its EVAAS value-added software system, provided undisclosed air travel for the UNC Board of Governors and candidate Margaret Spellings during the recent presidential search. (See WRAL's "SAS Provided Undisclosed Air Travel During UNC Presidential Search.") This raises some interesting questions.

Most of us in education are very aware that SAS would stand to gain should North Carolina adopt some of the proposals that would evaluate new teacher programs by tying the performance of these college programs to "student growth measures" on state standardized tests. In light of these "gifts" to the UNC system, SAS would "just happen to be there" with its EVAAS system should North Carolina make this adoption.

Is it not equally interesting that SAS provided air travel to Margaret Spellings, whom most of us in education see as a major advocate for standardized testing and accountability measures? We have her to thank for her support of the failure of the "test and punish" systems of accountability under No Child Left Behind. Naturally, SAS obviously wants someone as UNC president who would push the same kinds of measures at the university level, so why not provide her air travel? I submit that this means that North Carolina sees the adoption of EVAAS value-added measures in the future for all North Carolina university teacher education programs.

Ultimately, the UNC system in situations like this should not accept gifts from corporations who might stand to gain from future university decisions. The UNC system should reimburse SAS for the entire amount of these "gifts" to ensure that it is not subject to being bought by corporations. It is a matter of integrity, which our universities still have and our state politicians do not. It's also very convenient that SAS did not disclose the gifts as they were apparently supposed to do.

Value-Added Measures and 'Consulting Chicken Entrails' for High-Stakes Decision-Making

“Like the magician who consults a chicken's entrails, many organizational decision makers insist that the facts and figures be examined before a policy decision is made, even though the statistics provide unreliable guides as to what is likely to happen in the future.” Gareth Morgan, Images of Organization: The Executive Edition

Could it be that using Value-added data is the equivalent of consulting “chicken-entrails” before making certain high-stakes decisions? With all the voodoo, wizardry, and hidden computations that educators are just supposed to accept on faith from companies crunching the data, value-added data might as well be “chicken entrails” and the “Wizards of VAM” might as well be high-priests or magicians reading those innards and making declarations of effectiveness and fortune telling. The problem, though, is value-added measures are prone to mistakes, despite those who say “it’s best we have.” Such reasoning itself smells of simply accepting its imperfections. One only need hold their nose, and take the medicine.

What President Obama, Arne Duncan, down through our own North Carolina state education leaders do not get is that Value-added measures simply are not transparent. If anyone reads any of the current literature on these statistical models, you immediately see many, many imperfections. There’s certainly enough errors of concern to argue that VAMs have zero place in making high-stakes decisions.

As the “Wizards of VAM” prepare to do their number crunching and “entrails reading” in North Carolina, we await their prognostications and declarations of “are we effective or ineffective?” Let’s hope it doesn’t smell too bad.

Value-Added Measures and Harmful Consequences of Measure & Punish

"The M & P (Measure and Punish) Theory of Change suggests that by holding districts, schools, teachers and students accountable for meeting higher standards, as measured by student performance on high-stakes tests, administrators will supervise America's public schools better, teachers will teach better, and as a result students will learn more, particularly in America's lowest performing schools." Audrey Amrein-Bearsley, Rethinking Value-Added Models in Education

As states and school districts begin to wade deeper into using value-added measures, or VAMs, in high-stakes employment decisions, lawsuits are inevitable. On Wednesday, seven Houston Independent School District teachers and their union filed a lawsuit against the Houston Independent School District (HISD), (See "Seven Teachers and Their Union Are Suing HISD to End Evaluations Tied to Students' Test Scores.") In this case, the teachers and their unions are focusing on the fact that teacher value-added ratings fluctuated immensely from year to year. For example, one of the plaintiffs, Andy Dewey, a social studies teacher, received high ratings in 2012, enough for him to receive a bonus. His results the next year dropped significantly. The lawsuit, which you can read for yourself here (HISD Lawsuit), states, "Mr. Dewey went from being deemed one of the highest performing teachers in HISD to one making 'no detectable difference' for his students." If, as VAM supporters hold to be true, teachers have substantial effect on student scores, how can a teacher get it perfectly correct one year, and get it all wrong the next?

HISD defends the use of value-added in its high-stakes practices, even as organizations such as the American Statistical Association cautions strongly against such use. Contrary to what those who support value-added measures say, even if you set aside the technical and methodological concerns, there is absolutely no evidence that using value-added measures as a part of teacher evaluations has any effect on student learning. There is, however, a great deal of research pointing out that there are potentially harmful, unintended consequences of using standardized tests in any high stakes manner. Those consequences include:

• Increased amounts of time devoted to teaching to the test and test prep activities.
• Administrative decisions made to drop non-tested subjects like art and social studies.
• Decreases in morale among teachers and administrators.
• Administrative decisions to cut time spent in untested subjects to focus on tested subjects.
• Narrowing of the curriculum to only what gets tested.
• Teaching becomes more didactic and teacher-centered rather than student-centered or 21st century oriented.
• Increased levels of frustration for students as they are subjected to more and more standardized tests.
• Teaching shifts to focusing more on "bubble" students or "money" students as I have heard them called. These are the students that have been identified to have the most potential for the greatest amount of growth. The other students receive less instruction and teacher attention as a result.
• Increased student apathy and boredom as a result of the disconnect between content relevancy and what's tested.
• Teachers and administrators shop for students and classes in order to teach students who are more likely to provide them with desired academic growth and test scores.
• Teachers are leaving a profession where they once believed in teaching students content worthwhile, which is rapidly becoming more focused on the raising of test scores.
• Potential teachers are choosing to not become teachers because it is no longer about teaching content they care about; it has become more about playing the game to get high test scores.
• In some schools and districts, teaching has become programmed and scripted and not creative, engaging and self-fulfilling any more.
• Administrators and teachers are held accountable for test scores in an environment where there are so many things not under their control, such as budgets, which violates the Cardinal Rule of Accountability, which states "Hold people accountable for what they control."

The use of high-stakes testing and VAMs are impacting schools and classrooms, but the costs and negative consequences are high. This lawsuit, while it is indicative of some serious methodological concerns about value-added measure, it is also a symptom of a greater issue. Those who still support high-stakes accountability and the use of VAM ignore or minimize any objections to their use. The massive increase in testing and its use for high-stakes personnel decisions under federal and state policy is negatively impacting our schools, classrooms, students, teachers, and our parents. The question becomes, at what point are policymakers going to realize the damage being done to public education?

All this focus on standardized testing is making public education a bizarre world where schools serve soft drinks to students as a test preparation strategy (See "Florida School Stops Giving Students Caffeinated Soda Before Standardized Tests"), and where entire schools hold pep rallies in their gymnasiums to get students "pumped up" for latest tests. Where time-honored subjects have become worthless and what's most trivial and "testable" gets emphasized. Where teachers are forced to focus on "money" students at the expense of other students who have needs too. Does not anyone else see anything morally wrong with this entire picture? To me, it is certainly understandable that when "the test results" are what determines job effectiveness, any educator is understandably going to do what is necessary to increase the measure by which their effectiveness is judged. Still, there are moral boundaries we should be unwilling to cross and ethical principles we just can't violate. Raising test scores is not our highest calling as educators despite what the Measure and Punish crowd think, and "Raising them at any cost" is morally repugnant and gives these tests more dignity and importance than they deserve.

D.C. Teachers Suffer Faulty Evaluations at Hand of Value-Added Measures: Is NC on Same Path?

I have made it known that I am no fan of using value-added measures in teacher evaluations. There's just too much room for error, and there's too many things that can go wrong, from the test to the calculations. Value-added calculations are done in a mysterious black-box and there is too little oversight and protection measures in place to ensure that the data is error-free. As the Washington Post reports here in "Errors Found in D.C. Teacher Evaluations," more than 40 teachers received incorrect teacher evaluations of the year 2012-2013. One teacher was even fired due to miscalculations. That is totally unaccepted and should not every happen.

Many states, including my  own, have adopted the "Value-added measure fad" without piloting or studying it at all, other than listening to the sales pitches and lobbying of companies peddling this methodology. In North Carolina, there is currently no recourse for challenging the scores either. If a teacher suspects their ratings are incorrect, there is no way to independently validate it. But if your goal is to implement corporate reforms measures, any mis-calculations and faulty teacher ratings are acceptable, as long as we implement the reform measure. According to an additional post on the Washington Post Web site, "D.C. Schools Gave 44 Teachers Mistaken Job Evaluations," it was faulty calculations "of the value that D.C. teachers added to student achievement in the last school year resulted in erroneous performance evaluations for 44 teachers, including one who was fired because of a low rating."

This incident illustrates clearly that value-added measures used in teacher evaluations are too error-prone and should be discarded. When education policy gets too caught up in numbers and statistics, people, whether teachers or students don't matter as much to the number-crunchers. The Obama administration should be ashamed of mandating this mistaken education policy too states to begin with. States who have implemented these measures need to immediately discard this statistical fad because it will ultimately do more to harm education than help. North Carolina needs to drop this fad too and begin moving their educational system into the 21st century. Sadly, our state leaders are so blinded by the numbers they just can't let go.