Building a Better Teacher Through VAMs? Not So Fast According to Mark Paige's Book

As a part of my research explorations, I stumbled across a relatively new book published in 2016 about the problems with using value-added measures in teacher evaluations. This book entitled Building a Better Teacher: Understanding Value-Added Models in the Law of Teacher Evaluation is a short and concise read that any administrator who currently encounters the use of value-added data in teacher evaluations should read.

Paige's argument is rather straightforward. Value-added models have statistical flaws and are highly problematic, and should not be used to make high-stakes decisions about educators. Scholars across the board have made clear that are problems with VAMs, enough problems that they should only be used in research and to cautiously draw conclusions about teaching. Later, Paige also provides advice to opponents to using value-added models in teacher education as well. Attempting to challenge the use of value-added models in teacher evaluations through the federal courts may be fruitless. According to Paige:

"At least at the federal level, courts will tolerate an unfair law, so long as it may be constitutional." p. 24

In other words, our courts will allow the use of VAMs in teacher evaluations, even if used unfairly. Instead, Paige encourages action on the legislative side. Educator opponents of VAMs should inform legislators of the many issues with the statistical measures and push for laws that restrict their use. In states with teacher unions, he encourages teachers to use the collective bargaining process to ensure that VAMs are not used unwisely.

Throughout Paige's short read, there are reviews of legal cases that have developed around the use of VAMs to determine teacher effectiveness and lots of information about the negative consequences of this practice.

Here are some key points from chapter 1 of Mark Paige's book Building a Better Teacher: Understanding Value-Added Models in the Law of Teacher Evaluation.

• VAMs are statistical models that attempt to estimate a teacher's contribution to student achievement.
• There are at least (6) different VAMs, each with relative strengths and weaknesses.
• VAMs rely heavily on standardized tests to assess student achievement.
• VAMs have been criticized on a number of grounds as offending various statistical principles that ensure accuracy. Scholars have noted that VAMs are biased and unstable, for example.
• VAMs originated in the field of economics as a means to improve efficiency and productivity.
• The American Statistical Association has cautioned against using VAMs in making causal conclusions between a teacher's instruction and a student's achievement as measured on standardized tests.
• VAMS raise numerous nontechnical issues that are potentially problematic to the health of a school or learning climate. These include the narrowing of curriculum offerings and a negative impact on workforce morale.

Throughout his book, Paige offers numerous key points that should allow one to pause and interrogate the practice of using VAMs to determine teacher effectiveness.

Using VAMs to Determine Teacher Effectiveness: Turning Schools into Test Result Production Factories

"But VAMs have fatal shortcomings. The chief complaint: they are statistically flawed. VAMs are unreliable, producing a wide range of ratings for the same teacher. VAMs do not provide any information about what instructional practices lead to particular results. This complicates efforts to improve teacher quality; many teachers and administrators are left wondering how and why their performance shifted so drastically, yet their teaching methods remained the same." Mark Paige, Building a Better Teacher: Understanding Value-Added Models in the Law of Teacher Evaluation

Mark Paige's book is a quick, simple view regarding the problems with using value-added models as a part of teacher evaluations. As he points out, the statistical flaws are a fatal shortcoming to using them to definitively settle the questions regarding whether a teacher is effective. In his book, he points to two examples of teachers where those ratings fluctuated widely. When you have a teacher who rates "most effective" to "not effective" within a single year, especially when that teacher used the same methods with similar students, there should be a pause of question and interrogation.

Now, the VAM proponents would immediately diagnose the situation thus, "It is rather obvious that the teacher did not meet the needs of students where they are." What is wrong with the logic of this argument? On the surface, arguing that the teacher failed to "differentiate" makes sense. But, if there exists "universal teaching methods and strategies" that foster student learning no matter the context, then what would explain the difference? The real danger of using VAMs in the manner suggested by the logic of "differentiation" invalidates the idea that there are universally, research-based practices to which teachers can turn in improving student outcomes. What's worse, teaching becomes a game of pursuit every single year, where the teacher simply seeks out, not necessarily the best methods for producing learning of value, but instead, becomes, in effective a chaser of test results. Ultimately, the school becomes a place where teachers are simply production workers whose job is to produce acceptable test results, in this case, acceptable VAM results.

The American Statistical Association has made it clear. VAMs do not predict "causation." They predict correlation. To conclude that "what the teacher did" is the sole cause of test results is to ignore a whole world of other possibilities and factors that has a hand in causing those test results. Administrators should be open to the possibility that VAMs do not definitively determine a teacher's effectiveness.

If we continue down the path of using test score results to determine the validity and effectiveness of every practice, every policy, and everything we do in our buildings, we will turn out schools in factories whose sole purpose is produce test scores. I certainly hope we are prepared to accept along with that the life-time consequential results of such decisions.


NOTE: This post is a continued series of posts about the practice of using value-added measures to determine teacher effectiveness based on my recently completed dissertation research. I make no efforts to hide the fact that I think using VAMs to determine the effectiveness of schools, teachers, and educators is poor, misinformed practice. There is enough research out there to indicate that VAMs are flawed, and that there application in evaluation systems have serious consequences.

Arne Duncan's Proposal to Use Test Scores to Measure Teacher-Prep Program Effectiveness

Public schools have suffered under Secretary Arne Duncan's Race to the Top and No Child Left Behind law waivers. Testing, not learning has become the focus. Schools have cut arts programs and non-tested subjects. Enormous amounts of time are spent during the school year getting students ready for the tests. And, since the Obama administration took office, there are many states like North Carolina that administer a record number of state tests, and the use those results as a part of teacher evaluations. It has been this President's education policy that has done more to elevate test scores to even higher levels than under No Child Left Behind. 

Now, Arne Duncan is once again trying to elevate test scores even higher: he wants to use test scores to evaluate the effectiveness of teacher programs too.

Under Arne Duncan's latest effort to hold somebody else accountable for education except himself and politicians, Duncan now wants to create a new, massive bureaucratic procedure to judge the "effectiveness" of teacher preparations programs around the country. This behemoth proposal would bizarrely twist test scores once more in the name of accountability. As I read through this proposed procedure, I simply grow more and more angry at a President and Secretary of Education who simply have no clue as to what their "test-them-if-they-breathe" education agenda has done to schools, students, teachers, classrooms, and the future of the education profession. If you read the fine print of this massive document, you can quickly read between the lines regarding what Arne Duncan is actually proposing.

• Using test scores, most likely value-added measures, to determine the effectiveness of teacher preparation programs that receive federal funding.
• The development of a massive pile of red tape and bureaucratic procedures to make sure teacher preparation programs comply to the dictates of the US Department of Education.
• An enormous overreach of federal power and powergrab by the US Department of Education.

There was a time when I would have defended the existence of the US Department of Education. Now, I am slowly beginning to feel that perhaps the best thing for public schools is for this new Congress to simply dismantle it. Has there been a single good policy or idea that has come down through this department during the Obama Administration?

I think it's perhaps time to write some letters, send emails, and make some phone calls on Duncan's bizarre plan to use test scores in yet another high stakes manner. All US educators and pre-service educators need to take some time and let the President, Secretary Duncan, Congress, and the US Department of Education know their thoughts on this one.  Otherwise, like the Race to the Top, Duncan will claim he has heard only praise for this latest effort to bend the education world to tests.

If you would like to submit your own comment or opinion, you can do so at the address below. The deadline for submitting comments is February 2, 2015. Perhaps enough educators will submit comments that it will take the US Department of Education five years to read them. 

Read and Comment on the Newly Proposed Arne Duncan Teacher Preparation Rule. (BE SURE TO DO SO BY FEBRUARY 2, 2015).

9 Reminders for School Leaders When Reviewing Value-Added Data with Teachers

“A VAM (Value-Added Model) score may provide teachers and administrators with information on their students’ performance and identify areas where improvement is needed, but it does not provide information on how to improve the teaching.” American Statistical Association

Today, I spent a little time looking over the American Statistical Association’s "ASA Statement on Using Value-Added Models for Educational Assessment.” That statement serves as a reminder to school leaders regarding what these models can and cannot do. Here, in North Carolina and in other states, as school leaders begin looking at  No Child Left Behind Waiver-imposed value added rankings on teachers, they would do well to remind themselves of the cautions describe by ASA last April. Here’s some really poignant reminders from that statement:

• “Estimates from VAMs should always be accompanied by measures of precision and a discussion of the assumptions and possible limitations of the model. These limitations are particularly relevant if VAMs are used for high-stakes purposes.”
• “VAMs are generally based on standardized test scores, and do not directly measure potential teacher contributions toward other student outcomes.”
• “VAMs typically measure correlation, not causation: Effects—positive or negative—attributed to a teacher may actually be caused by other factors that are not captured in the model.”
• “Under some conditions, VAM scores and rankings can change substantially when a different model or test is used, and a thorough analysis should be undertaken to evaluate the sensitivity of estimates to different models.”
• “Most VAM studies find that teachers account for about 1% to 14% of the variability in test scores, and that the majority of opportunities for quality improvement are found in the system-level conditions.
• “Ranking teachers by their VAM scores can have unintended consequences that reduce quality.”
• “The measure of student achievement is typically a score on a standardized test, and VAMs are only as good as the data fed into them.”
• “Most VAMs predict only performance on the test and not necessarily long-range learning outcomes.”
• “The VAM scores themselves have large standard errors, even when calculated using several years of data.”

In this season of VAM-viewing, it is vital that informed school leaders remind themselves of the limitations of this data. You can’t take the word of companies promoting these models as “objective” and “fool-proof” measures of teacher quality. After all, they have those multimillion dollar contracts or will lose them if one casts doubt about VAM use. Still, a 21st century school leader needs to have a more balanced view of VAM and its limitations.

Value-added ratings should never be used to inform school leaders about teacher quality. There are just too many problems. In the spirit of reviewing VAM data with teachers, here’s my top ten reminders or cautions about using value-added data in judging teacher quality:

1.  Remember the limitations of the data. Though many states and companies providing VAM data fail to provide extensive explanations and discussion about the limitations of their particular value-added model, be sure those limitations are there. It is common to hide these limitations in statistical lingo and jargon, but as a school leader, you would do well to read the fine print, research for yourself, and understand value-added modeling for yourself. Once you understand the limitations of VAMs you will reluctantly make high stakes decisions based on such data.

2. Remember that VAMs are based on imperfect standardized test scores. No tests directly measure teacher  contributions to student learning. In fact, in many states, tests used in VAMS were never intended to be used in a manner to judge teacher quality. For example, the ACT is commonly used in VAMS to determine teacher quality, but it was not designed for that purpose. As you review your VAM data, keep in mind the imperfect testing system your state has. That should give you pause in thinking that the VAM data really tells you flawlessly anything about a teacher’s quality.

3. Because VAMs measure correlation not causation, remind yourself as you look at a teacher’s VAM data that he or she alone did not cause those scores or that data. There are many, many other things that could have had a hand in those scores. No matter what promises statistics companies or policymakers make, remember that VAMs are as imperfect as the tests, the teacher, the students, and the system. VAM data should not be used to make causal inferences about the quality of teaching.

4. Remember that different VAM models produce different rankings. Even choosing one model over another reflects subjective judgment. For example, some state’s choose VAMs that do not control for other variables such as student demographical background because they feel to do so makes an excuse for lower performance for low-socioeconomic students. That is a subjective value judgment on which VAM to use. Because of this subjective judgment, they aren’t perfectly objective. All VAM models aren't equal.

5. Remind yourself that most VAM studies find that teachers account for about 1 to 14 % of variability in test scores. This means that teachers may not have as much control over test scores as many of those using VAMs to determine teacher quality assume. In a perfect manufacturing system where teachers are responsible for churning out test scores, VAMs make sense. Our schools are far from perfect, and there are many, many things out there impacting scores. Teaching is not a manufacturing process nor will it ever be.

6. Remind yourself that should you use VAMs in a high stakes manner, you may actually decrease the quality of student learning and harm the climate of your school. Turning your school into a place where only test scores matter, where teaching to the test is everybody’s business is a real possibility should you place too much emphasis on VAM data. Schools who obsess about test scores aren't fun places for anybody, teachers or students. Balance views of VAM data as well as test data is important.

7. Remember that all VAM models are only as good as the data fed into them. In practical terms, remember the imperfect nature of all standardized tests as you discuss VAM data. Even though states don’t always acknowledge the limitations of their tests, that doesn’t mean you can’t. Keep the imperfect nature of tests and VAMs in mind always. Perhaps then, you want use data unfairly.

8. Remember that VAMs only predict performance on a single test. They do not tell you thing about the long-range impact of that teacher on student performance.

9. Finally, VAMs can have large standard errors. Without getting entangled in statistical lingo, just let it suffice to say that VAMs themselves are imperfect. Keep that in mind when reviewing the data with teachers.

The improper use of VAM data by school leaders can downright harm education. It can turn schools into places where in-depth learning matters less than test content. It can turn teaching into a scripted process of just covering the content. It can turn schools from places of high engagement, to places where no one really wants to be. School leaders can prevent that by keeping VAM data in proper perspective, as the "ASA Statement on Using Value-Added Models for Educational Assessment" does.

Value-Added Measures and 'Consulting Chicken Entrails' for High-Stakes Decision-Making

“Like the magician who consults a chicken's entrails, many organizational decision makers insist that the facts and figures be examined before a policy decision is made, even though the statistics provide unreliable guides as to what is likely to happen in the future.” Gareth Morgan, Images of Organization: The Executive Edition

Could it be that using Value-added data is the equivalent of consulting “chicken-entrails” before making certain high-stakes decisions? With all the voodoo, wizardry, and hidden computations that educators are just supposed to accept on faith from companies crunching the data, value-added data might as well be “chicken entrails” and the “Wizards of VAM” might as well be high-priests or magicians reading those innards and making declarations of effectiveness and fortune telling. The problem, though, is value-added measures are prone to mistakes, despite those who say “it’s best we have.” Such reasoning itself smells of simply accepting its imperfections. One only need hold their nose, and take the medicine.

What President Obama, Arne Duncan, down through our own North Carolina state education leaders do not get is that Value-added measures simply are not transparent. If anyone reads any of the current literature on these statistical models, you immediately see many, many imperfections. There’s certainly enough errors of concern to argue that VAMs have zero place in making high-stakes decisions.

As the “Wizards of VAM” prepare to do their number crunching and “entrails reading” in North Carolina, we await their prognostications and declarations of “are we effective or ineffective?” Let’s hope it doesn’t smell too bad.

Value-Added Measures and Harmful Consequences of Measure & Punish

"The M & P (Measure and Punish) Theory of Change suggests that by holding districts, schools, teachers and students accountable for meeting higher standards, as measured by student performance on high-stakes tests, administrators will supervise America's public schools better, teachers will teach better, and as a result students will learn more, particularly in America's lowest performing schools." Audrey Amrein-Bearsley, Rethinking Value-Added Models in Education

As states and school districts begin to wade deeper into using value-added measures, or VAMs, in high-stakes employment decisions, lawsuits are inevitable. On Wednesday, seven Houston Independent School District teachers and their union filed a lawsuit against the Houston Independent School District (HISD), (See "Seven Teachers and Their Union Are Suing HISD to End Evaluations Tied to Students' Test Scores.") In this case, the teachers and their unions are focusing on the fact that teacher value-added ratings fluctuated immensely from year to year. For example, one of the plaintiffs, Andy Dewey, a social studies teacher, received high ratings in 2012, enough for him to receive a bonus. His results the next year dropped significantly. The lawsuit, which you can read for yourself here (HISD Lawsuit), states, "Mr. Dewey went from being deemed one of the highest performing teachers in HISD to one making 'no detectable difference' for his students." If, as VAM supporters hold to be true, teachers have substantial effect on student scores, how can a teacher get it perfectly correct one year, and get it all wrong the next?

HISD defends the use of value-added in its high-stakes practices, even as organizations such as the American Statistical Association cautions strongly against such use. Contrary to what those who support value-added measures say, even if you set aside the technical and methodological concerns, there is absolutely no evidence that using value-added measures as a part of teacher evaluations has any effect on student learning. There is, however, a great deal of research pointing out that there are potentially harmful, unintended consequences of using standardized tests in any high stakes manner. Those consequences include:

• Increased amounts of time devoted to teaching to the test and test prep activities.
• Administrative decisions made to drop non-tested subjects like art and social studies.
• Decreases in morale among teachers and administrators.
• Administrative decisions to cut time spent in untested subjects to focus on tested subjects.
• Narrowing of the curriculum to only what gets tested.
• Teaching becomes more didactic and teacher-centered rather than student-centered or 21st century oriented.
• Increased levels of frustration for students as they are subjected to more and more standardized tests.
• Teaching shifts to focusing more on "bubble" students or "money" students as I have heard them called. These are the students that have been identified to have the most potential for the greatest amount of growth. The other students receive less instruction and teacher attention as a result.
• Increased student apathy and boredom as a result of the disconnect between content relevancy and what's tested.
• Teachers and administrators shop for students and classes in order to teach students who are more likely to provide them with desired academic growth and test scores.
• Teachers are leaving a profession where they once believed in teaching students content worthwhile, which is rapidly becoming more focused on the raising of test scores.
• Potential teachers are choosing to not become teachers because it is no longer about teaching content they care about; it has become more about playing the game to get high test scores.
• In some schools and districts, teaching has become programmed and scripted and not creative, engaging and self-fulfilling any more.
• Administrators and teachers are held accountable for test scores in an environment where there are so many things not under their control, such as budgets, which violates the Cardinal Rule of Accountability, which states "Hold people accountable for what they control."

The use of high-stakes testing and VAMs are impacting schools and classrooms, but the costs and negative consequences are high. This lawsuit, while it is indicative of some serious methodological concerns about value-added measure, it is also a symptom of a greater issue. Those who still support high-stakes accountability and the use of VAM ignore or minimize any objections to their use. The massive increase in testing and its use for high-stakes personnel decisions under federal and state policy is negatively impacting our schools, classrooms, students, teachers, and our parents. The question becomes, at what point are policymakers going to realize the damage being done to public education?

All this focus on standardized testing is making public education a bizarre world where schools serve soft drinks to students as a test preparation strategy (See "Florida School Stops Giving Students Caffeinated Soda Before Standardized Tests"), and where entire schools hold pep rallies in their gymnasiums to get students "pumped up" for latest tests. Where time-honored subjects have become worthless and what's most trivial and "testable" gets emphasized. Where teachers are forced to focus on "money" students at the expense of other students who have needs too. Does not anyone else see anything morally wrong with this entire picture? To me, it is certainly understandable that when "the test results" are what determines job effectiveness, any educator is understandably going to do what is necessary to increase the measure by which their effectiveness is judged. Still, there are moral boundaries we should be unwilling to cross and ethical principles we just can't violate. Raising test scores is not our highest calling as educators despite what the Measure and Punish crowd think, and "Raising them at any cost" is morally repugnant and gives these tests more dignity and importance than they deserve.

Let the VAM Lawsuits Begin: Issues and Concerns with Their High-Stakes Use

Lawsuits against states using value-added models in making teaching evaluation decisions has begun in earnest. There are now three lawsuits underway challenging the use of this controversial statistical methodology and the use of test scores to determine teacher effectiveness. This increase in litigation is both an indication of how rapidly states have adopted the practice, and how these same states failed to address so many issues and concerns with the use of VAMs in this manner.

Two lawsuits have now been filed in Tennessee against the use of value-added  assessment, known as TVAAS as a part of teacher evaluation. The first lawsuit was filed against Knox County Schools in Tennessee by the Tennessee Education Association on behalf of an alternative school teacher who was denied a bonus because of her TVAAS ratings. (See “Tennessee Education Association Sues Knox County Schools Over Bonus Plan” ) In this case, the teacher was told she would receive system-wide TVAAS estimates because of her position at an alternative school, but 10 of her students were used anyway in her TVAAS score, resulting in a lower rating and no bonus. This lawsuit contests the arbitrariness of TVAAS estimates that use only a small number of teacher’s students to determine overall effectiveness.

In the second lawsuit, filed also against Knox County Schools, but also against Tennessee Governor Bill Haslam, state Commissioner of Education Kevin Huffman and the Knox County Board of Education, an eighth grade science teacher claims he was also denied a bonus unfairly after his TVAAS value-added rating was based on only 22 of his 142 students. (See “TEA Files Second Lawsuit Against KCS, Adds Haslam and Huffman as Defendents” ) Again, the lawsuit points to the arbitrariness of the TVAAS ratings.

A third lawsuit has been filed in Rochester, New York by the Rochester Teachers Association alleging that officials in that state “failed to adequately account for the effects of severe poverty, and as a result, unfairly penalized Rochester teachers on their Annual Professional Performance Review” or yearly teacher evaluations. (See “State Failed to Account for Poverty in Evaluations”). While it appears that this Rochester suit is disputing the use of growth score models not value-added, it also challenges the whole assumption and recent fad being pushed by politicians and policymakers of using test scores to evaluate teachers.

North Carolina jumped on the value-added bandwagon in response to US Department of Education coercion, and now the state uses its TVAAS version called EVAAS, or Educator Value Added Assessment System as part of teacher and principal evaluations. Fortunately, no districts have had to make high stakes decisions using the disputed measures so the lawsuit floodgate hasn't opened in our state yet, but I am sure once EVAAS is used to make decisions about employment, the lawsuits will begin. When those lawsuits begin, the American Statistical Association has perhaps outlined some areas of contention about the use of VAMs in educator evaluations in their ASA Statement on Using Value-Added Models for Educational Assessment. Here’s some points made by their position statement that clearly outlines the questions about the use of VAMs in teacher evaluations, a highly questionable statistical methodology.

• “VAMs (Value-added models) are complex statistical models, and high-level statistical expertise is needed to develop the models and interpret their results.” States choosing to use these models are trusting third-party vendors to develop them, provide the ratings, and they are expecting educators to effectively interpret those results. Obviously, there’s so much that can go wrong with the interpretation of VAM results, the ASA is warning that there is a need of people who have the expertise to interpret those results. I wonder how many of these states who have implemented these models have spent time and money training teachers and administrators to interpret these results, other than subjecting educators to one-time webinars or "sit-n-gets"?
• “Estimates of VAM should always be accompanied by measures of precision and a discussion of the assumptions and possible limitations of the model. THESE LIMITATIONS ARE PARTICULARLY RELEVANT IF VAMS ARE USED FOR HIGH STAKES PURPOSES (Emphasis Mine).” I can’t speak for other states, but in North Carolina there has been little to no disclosure or discussion about the limitations of value-added data. There’s been more public relations, advertising, and promotion of the methodology as a new way of evaluating educators. They even have SAS promoting the methodology for them.The Obama administration has done this as well. The attitude in North Carolina seems to be, “We’re gonna evaluate teachers this way, so deal with it.” There needs to be discussion and disclosure about SAS’s EVAAS model and the whole process of using tests to evaluate teachers in North Carolina. Sadly, that’s missing. I can bet it’s the same in other states too.
• “VAMs are generally based on standardized test scores, and do not directly measure potential teacher contributions toward other student outcomes.” In other words, VAMs only tell you how students do on standardized tests. They can’t tell you all the other many, many ways teachers contribute to students’ lives. The main underlying assumption with using VAMs in teacher evaluations is that only test scores matter, regardless of what supporting policymakers say. While its true that the North Carolina Evaluation model does include other standards, how long will it take administrators and policymakers to ignore those standards and zero in on test scores because they are seen as the most important? The adage, "What gets tested, gets taught!" is true and "What get's emphasized the most through media and promotion, matters the most" is also equally true. When standard 6 or 8 is the only standard on the educator evaluation where an educator is "In Need of Improvement" then you can bet test scores suddenly matter more than anything else.
• “VAMs typically measure correlation, not causation: Effects---positive or negative---attributed to a teacher may actually be caused by other factors that are not captured in the model.” There are certainly many, many things----poverty, lack of breakfast, runny noses---that can contribute to a student’s test score, yet there’s a belief that a teacher directly causes a test score to happen, especially by those pushing VAMs in teacher evaluations. The biggest assumption by those promoting VAMs in teacher evaluations is that the teacher's sole job or part of their job is the production of test scores. In reality, teaching is so much more complex than that, and those reducing it to a test score have probably not spent much time teaching themselves.
• “Most VAM studies find that teachers account for about 1% to 14% of the variability in test scores, and that the majority of the opportunities for quality improvement are found in system-level conditions.” Yet in most states, educational improvement falls almost entirely on the backs of educators in the schools in the form of VAM-Powered Teacher Evaluations. There's little effort to improve the system. There’s no effort to improve classroom working conditions, provide professional development funding/resources, adequate material/resource funding. Instead of looking at how the system prevents excellence and innovation with its top-down mandates and many other ineffective measures, many states, including North Carolina and the Obama administration place accountability entirely and squarely on the backs of educators in the classrooms and schools. If the education system is broken, you don't focus on parts, you improve the whole.
• “Ranking teachers by their VAM scores can have unintended consequences that reduce quality.” If all learning that is important can be reduced to a one-time administered-bubble-sheet test, then all is well for VAM and the ranking of teachers. But every educator knows that tests measure only a minuscule portion of important learning. Many important learning experiences can't even be measured by tests. But, if you elevate tests in a high stakes manner, then those results become the most important outcome of the school and the classroom. The end result is teaching to the test and test-prep where the test becomes the curriculum. Getting high test scores becomes the goal of teaching. If that’s the goal of teaching, who would want to be teacher? Elevating test scores through VAM only will escalate the exit of teachers from the profession and discourage others from entering it. because there's nothing fulfilling about improving student test scores. We didn't become educators to raise test scores; we became educators because we wanted to teach kids.
• “The measure of student achievement is typically a score on a standardized test, and VAMs are only as good as the data fed into them.” Ultimately, VAMs are only as good as the tests administered to provide the data that feeds the model. If tests don’t adequately measure the content, or if they are not standardized or otherwise of high quality, then the VAM estimates are equally of dubious quality. When states try to scramble to create tests on the fly and do not develop quality tests, then the VAM estimates are of dubious quality too. North Carolina scrambled to create multiple tests in many high school, middle and elementary subjects just to have data to feed their EVAAS model. Yet, those tests and the process of their creation and field testing, even how they’re administered makes them questionable candidates for serious VAM use. VAMs require high-quality data to provide high-quality estimates. The idea that "any-old-test-will-do" is an anathema to VAMs which require quality test data.

The American Statistical Association position statement on using value-added models in educational assessment makes some supporting statements about their use too. They can be effectively used as part of the data teachers use to adjust classroom teaching. But when a state does not return those scores until October or later, its impossible to use that data to inform teaching, three months into the school year. Also, just getting a rating does little to inform teaching. Testing provides an opportunity for policymakers to provide teachers with valuable data to improve teaching. Sadly, the current data provided is too little and too late.

As the VAM-fed teacher evaluation fad and craze continues and grows, it is important for all educators to inform themselves about the controversial statistical practice. It is not a methodology without issues despite what the Obama administration and state education leaders say. Being knowledgeable about it means understanding its limitations as well as how to properly interpret and use such data. Don't wait for states and the federal government to provide that information: They are too busy promoting its use. The points made in the American Statistical Association’s Statement on Using Value-Added Models for Educational Assessment are excellent points of entry for learning more.