What Happens When Schools and School Districts Use VAMs to Make Decisions about Teachers?

Many school administrators are using value-added measures in making decisions about teachers as if these statistical measures represent the latest, settled and unquestionable science. Those who do this are making a grave error. Despite companies such as SAS, who peddle their EVAAS data systems as the salvation of public education, the science behind VAMs is not settled, and there is even enough doubt about them, that the American Statistical Association issued a strong statement in 2014 against their use in decision-making when it comes to teachers. In that statement, ASA reminds educators that:

VAMs typically measure correlation, not causation---positive or negative---attributed to a teacher may actually be caused by other factors that are not captured in the model. (ASA Statement on VAMs)

Yet, administrators still use VAMs to infer that the teacher causes those scores. SAS, who owns the EVAAS model that North Carolina pays millions of dollars for each year, arrogantly claims that it accounts for all the factors that cause student performance on test scores, even when psychometric experts caution that this isn't possible.

In addition, administrators, who use VAMs to make decisions about teachers, should know better than confuse correlation with causation, but any time they base decisions about teacher status using VAMs, they are automatically assuming that teachers cause test results. If teachers operated in a lab where they controlled all the conditions of learning and the subjects of their learning, then one could perhaps better make this inference.

But there are other concerns about VAMs too. In a recent study by Shen, Simon, and Kelcey (2016), it was found that "using value-added teacher evaluations to inform high-stakes decision-making may not make for a good teacher." Using VAMs to decide the status of a teacher may not have the long-term impact administrators desire. These researchers also recommend that VAMs not be used "to inform disincentive high stakes decisions," which are any decisions regarding the professional status of teachers.

Ultimately, though, I can't help but wonder if those who are sold on using VAMs in administrative decision-making aren't caught up in chasing short-term gains in a measurement that lacks any meaningfulness in the long-term. VAMs aren't settled science. Yet, administrators use that data as if it were. Any decisions made using this data should be balanced with other data.

Shen, Z., Simon, C., & Kelcey, B. (2016). The potential consequence of using value-added models to evaluate teachers. eJournal of Education Policy, Fall 2016.


NOTE: My just completed dissertation was on the practice of using value-added measures to determine teacher effectiveness. My plan is to share over the next several weeks and months my own insights and personal thoughts on this practice. This is the first of may posts I plan to share on this topic. 

Teacher Evaluations That Effectively Impact Learning and the Classroom

“States have implemented teacher evaluations in their race to avoid the impossible demands of NCLB.” W. James Popham, Evaluating America’s Teachers Mission Possible?

Getting teacher evaluation right should be a priority, and in my opinion, a higher priority than any other educational initiative on the table right now. How we evaluate teachers will impact more of what happens in our classrooms than any other reform measure we implement. For example, if we make test scores a large part of this evaluation, then we can expect much of what happens instructionally is going to directed toward improving test scores. If we make technology a priority, we can expect to see more teachers engaging in its use in the classroom. And, if getting students working collaboratively is emphasized, then expect to see teaming as a big part of classroom practice. What get's evaluated is what gets done, period! As Popham points out, "A teacher-appraisal system that inclines teachers to make good instructional decisions is likely to do just that. Conversely, a state teacher-appraisal that points teachers in unsound instructional directions, will, unfortunately, also just do that." What we evaluate is going to be what we get instructionally.

With the importance of what to include in our teacher evaluations in mind, getting teacher evaluation right should not be a hurried process to satisfy federal education policy, though that is what has happened in many states. In order to get waivers from the impossible and ludicrous demands of federal No Child Left Behind legislation, states have been hurriedly putting together teacher evaluations that include using test scores in some manner. This express route to using test scores to determine teacher quality should be frightening to any educator and our parents because of the likelihood that how students do on tests will become the center of what we do in our classrooms. In a word, our schools, every single one of them, become massive "test-prep centers." Let's just hope those evaluations encourage sound instructional decisions and not unsound decisions. Otherwise, we could end up with an education system much more worse off than what we have.

What then are some major mistakes states could make in this rush to implement federally mandated teacher evaluations? In his book Evaluating America's Teachers Mission Possible? W. James Popham describes what he calls "Four Teacher Evaluation Implementation Mistakes" that is, perhaps, a good start for critiquing these state teacher evaluation schemes.

Mistake 1: Using Inappropriate Evidence of a Teacher's Quality

According to Popham, implementation mistake one is simply using "poorly chosen evidence" to determine teacher effectiveness. Evidence is poorly chosen when it does not accurately or in a valid manner tell us anything about a teacher's quality. For example, an observation by a poorly or untrained classroom observer may not provide us with appropriate evidence to determine a teacher's quality. Also, using state achievement tests may also be poorly chosen evidence as well. As Popham clearly points out, "There is almost zero evidence that these state accountability tests yield data permitting valid inferences about a teacher's instructional quality." So even standardized test scores could be inappropriate evidence if one can't make a valid inference about teacher quality from those tests. With this mistake in mind, it is vital that administrators and teachers carefully scrutinize what states choose as evidence in their teacher evaluations.

Mistake 2: Improperly Weighting Evidence of a Teacher's Quality

Improperly weighting evidence involves assigning improper weights to the various forms of evidence used in teacher evaluations. The different sources of evidence most often used include: 1) student test performances, 2) administrator ratings of teachers' skills, 3) classroom observations, and 4) parent or student ratings of teachers. As Popham points out, "a weighting mistake occurs when a given source of evidence is given either far greater, or far lesser, evaluative importance than it should be given." In other words, this error occurs when states place too much or not enough emphasis on one source of evidence. For example, some states weigh test scores as 50 percent in their evaluation schemes. Such heavy weighing of test scores is improper, if those tests do not really allow one to make any valid inferences about teacher quality. Properly weighting the evidence means doing so in a reasonable and fair manner. It is important for teachers and administrators to also scrutinize the weighting scheme states apply to teacher evaluations as well.

Mistake 3: Failing to Adjust Evaluative Weights of Evidence for a Particular Teacher's Instructional Setting

As Popham points out, "To evaluate teachers as though they were operating in identical instructional settings is naive." Using a "cookie-cutter" evaluation system that fails to take into account the unique qualities of that teacher's instructional setting forces standardization when classrooms are far from being standardized. Doing so will not make them standardized either. When a teacher evaluation system weights all evidence without taking into account a teacher's instructional setting, you are going to force that teacher to do the "hoop-jumping" dance, just to satisfy the evaluation. Also, classrooms are as diverse as the teachers teaching them. Even in the unreal world of 20th century, factory designed instructional settings were not standardized, though some nostalgically think so. To evaluate teachers as if they were all teaching under the same conditions shows pure ignorance of what goes on in classrooms. They are highly, complex and diverse environments, and to think one can evaluate them the same fails to take into account this diversity. Teachers and administrators would do well to scrutinize the evaluative weights states place on various forms of evidence and demand that those weighting systems accommodate the diversity of instructional settings that exist too.

Mistake 4: Confusing the Roles of Formative and Summative Teacher Evaluation

As Popham points out, "Formatively, we want to improve teachers' prowess so they can do their most effective job in helping students learn. Summatively, we want to identify the exceptional teachers who should be rewarded as well as those teachers who, if they cannot be helped, should be relieved of their teaching responsibilities." Mistake four involves combining these two functions of teacher evaluation. As Popham points out, these two functions of teacher evaluation can conflict with each other, making neither effective. In addition, isn't it really unrealistic to expect a teacher to open up and be candid and reflective about their performance with the person who holds their future employment in his or her hands? Perhaps it us time for teachers and administrators to advocate for separating the formative and summative functions of teacher evaluation so that both work more effectively and can do their jobs of improving teaching.

Getting teacher evaluation right should be a high priority, because how teachers are evaluated is going to directly impact how instruction is carried out in the classroom. Unfortunately, states, in my opinion, including my own state North Carolina, have rushed to try to satisfy federal mandates for that evaluation system, and I fear the negative impact on education and the teaching profession in North Carolina, and around the country, is going to be felt for years.

Data-Driven Decision Making Usually Means “Test-Score Driven Decision-Making"

I received another one of those “Data-Driven Decision-Making Workshop” email advertisements the other day. I usually receive several of those during the course of a week. It’s as if someone out there has drawn the conclusion that I do not make decisions based on data, so I am in need of more training in this area.

So what is it that most of these workshops offer? They should be entitled, “Test-Score Driven Decision-Making” because that’s what most of them are really about. They are about using test scores to drive all school improvement activities. They're most often about not moving a muscle until you've consulted the numbers. Those who push making decisions based on test scores alone, elevate them to "sacred" status because it is somehow seen as "objective" and above reproach. Their true "objectivity" is certainly always debatable, but test scores are clearly not all the data we have at our disposal as school leaders. I really can’t help but resent the fact that these individuals actually think the decisions I make as an administrator aren't data-driven. I submit that every decision I make as principal is data-driven, otherwise I would be leading a school by the old proverbial “seat of my pants.” But perhaps the data I use isn't "sacred" enough to be considered data by the data-purists.

So what do I mean when I say my decision-making is always data-driven? It means that every single time I make a decision, I use data. When I am staring at each and every decision-point, I always begin the process by gathering data. For example, if I notice a student is failing multiple subjects, I have used data (grades) to determine that a student is in trouble. I might then call that student in my office and ask questions to gather additional data. "Why are you struggling in Algebra I?" I might ask. Or, "Is there something that’s keeping you from being successful in biology?" The answers to these questions are data. I would add that this kind of data allows me to make much better inferences regarding a student’s classroom learning and what to do about it than the test scores alone would do.

The problem with the term “data-driven decision-making" is that, most of the time, people most often always mean “just test scores.” They wall off a whole world of data as irrelevant because it isn't “sufficiently objective.” It’s almost as if speaking to a teacher to get their impression of a student’s performance is somehow tainted because it isn't numerical or something. Don’t get me wrong. I do see test scores as useful. They can tell us a bit of what we need to know about a student’s learning. What they do not tell us is the whole picture. For example, a student who repeatedly gets poor test scores in reading could allow us to draw any number of inferences about the reasons for their performance. Those might be:

• They have had bad teachers in former years.
• The school they attend is more or less a war zone rather than a school.
• They didn't have breakfast. In fact, perhaps they are homeless.
• They didn't get any sleep the night before the test.
• They hate the subject being tested.

Any number of inferences based on those scores may be valid, but I can't immediately jump to the conclusion that his score or the class's test scores are poor because the teacher is a bad teacher. It’s only when I investigate further that I can make any of these inferences valid. When we check with the parents and teachers, we might discover that the real problem is that the child can’t see, and is in need of glasses.

Making decisions entirely based on test scores alone is educational malpractice; there's simply no other way to state it. Data-Driven decision making can consider test scores, but it should also include that qualitative stuff that drives data purists nuts. But then, those people don't live in a real world anyway.

The point of today’s post isn't to create excuses for bad test scores. The point is to make clear that “data-driven decision-making” should never, ever be just about test scores. Those pushing “promotion-based-on-test-score” schemes for students” and “teacher-evaluations-based-on-test score” schemes would do well to remember that our best decisions about the welfare of the kids, and teachers, in our schools are not based on the latest end of grade tests or latest SAT scores. The best decisions we make might glance at these scores, but are based on data that is much closer to what the student needs are. Our best decisions are made when we take considerable time to investigate and gather much more than numbers on a spreadsheet. 

So, in order to keep from having to read any more these "data-driven workshop emails, I have set up a Gmail filter to dump them into my trash folder immediately. The whole idea that any one in education could make decisions without data is just plain ludicrous.

Call for Skepticism and Caution When Using Test Scores in Teacher Evaluations

“We need to be careful that the tests we use in a properly designed teacher-appraisal system do, in fact, contribute to a valid (that is accurate) inference about a teacher’s quality.” W. James Popham, Evaluating America’s Teachers: Mission Possible?

North Carolina took the plunge this year and started using test scores as part of teacher and principal evaluations. The state has even invented a "new" kind of test, called a "Measure of Student Learning" in order to make sure there is plenty of test data to go around. What is particularly telling is how "carefully" the state crafted the term "Measures of Student Learning." It's as if somehow, not calling it a test, makes it not a test. State level educational logic never ceases to amaze me. Of course, the state then started calling these "Measures of Student Learning" something else. They started calling them "Common Exams." Notice again, the careful use of the word "exam" rather than "test." It's almost as if you don't call it a test, it isn't a test, but apparently state level policymakers haven't heard the old saw about a rose still being a rose even if it has another name.

Besides North Carolina's struggle with what to call their newly implemented tests, there's still the question of what the unintended consequences of having thousands of teachers "teaching to the test" is going to do for students in our state. Ultimately, being able brag that your students "Have the best scores in the world" is most likely what politicians and state level education officials are after. That's why they see salvation through test scores as the means to the "Educational Promised Land." Ultimately, there's a flawed logic driving this whole accountability and testing movement: it's the whole idea that learning can be entirely reduced to bubble sheet answer sheets and taken in a single sitting. And, that teachers can't be trusted to tell when a student has demonstrated that they have learned or not.

In my years as an educator, I have been amazed how trusting and accepting educators in North Carolina are when it comes to the latest policy flowing down from on high. It's as if they accept that those at the state level know more than they do, or somehow have access to magical information they do not have. So, when they implement something like the use of test scores in evaluations, many educators accept that the powers that be at the state level know what they are doing, so they trust them. Given the history of reform ideas and educational policy that travels down from on high, this "trust" is highly misplaced. I like to think that state level education officials mean well, but what often has happened during my career, these ideas when implemented locally have sometimes been a disaster and have been sometimes downright bad for kids. Instead of being so trusting, I submit that all educators in the schools and districts need to become skeptics and ask tough questions of our state-level, and federal level policymakers. We should never accept the "trust me, this will work" answer.

It is in this spirit of skepticism, I turn to Popham's book, Evaluating America's Teachers: Mission Possible? and our state's venture into making high stakes testing even more high stakes. In spite of what our state-level policymakers say,  I am not fully satisfied that North Carolina's tests are adequate measures of educator effectiveness, and  a healthy skepticism is still in order. This whole push to add test scores to teacher and principal evaluations has been a rush from the start. Depending on when you asked questions, how the tests were to be implemented has changed multiple times throughout the last two years. Never mind the fact that not a single teacher in North Carolina even saw the test before they were implemented. In their rush to have "test data" it's as if our state level policymakers think "any old data will do." They have failed to take the time to establish whether any of these tests really tell us anything about teaching quality.

In light of our state's push into "higher stakes testing", I think Popham reminds us of some important key issues and ideas about tests and teacher evaluations that state politicians and policymakers seem to forget.

• “Tests are not valid or invalid. Instead, it is a test-based inference whose validity is at issue.” In other words, it isn't the test that’s valid or invalid, it is the inferences drawn from those tests that have these qualities. It boils down to whether you can actually make an inference based on the test or not. The question is whether North Carolina's tests, which have been implemented haphazardly and a thrown-together-manner, actually tell us anything at all about the quality of teaching in our classrooms. Can I honestly say Teacher A is a good teacher because she added "this much" value to her students' Measures of Student Learning? Seems to me that it puts a great deal of faith in a single test.
• “Tests allow us to make inferences about a test taker. This inference, depending on the appropriateness of the test as a support for the inference being made, may be valid or invalid.” As Popham points out, the inference we make about the learner may be valid or invalid depending on the “appropriateness of the test” in its role to support the inference being made. As we know, the word validity is the extent to which that inference, or conclusion, is well-founded or corresponds to the real world. This boils down to whether the inference we draw about a student is valid or not. For example, should we infer, based on a student’s test scores that he is not proficient in the subject, we must be satisfied that the test we are using is the “appropriate measure,” and we must also make sure the conclusion we draw considers all real world facts. Ignoring a student’s socio-economic status, or even whether he experienced  a death in the family, can make our inference about the student’s proficiency invalid. Then there's the whole issue about making an inference about a teacher or principal's effectiveness using this same test. Has North Carolina sufficiently established the appropriateness of their Measures of Student Learning, End of Grade Tests, End of Course Tests, as instruments that allow for making inferences about teacher and principal quality? I'm not sure they have. Another question, do these Measures of Student learning allow us to make valid inferences about teacher quality? I'm not convinced they do.

As North Carolina moves forward with a teacher and principal appraisal instrument that uses test scores to determine effectiveness, all educators need to educate themselves and scrupulously ask questions of policymakers at all levels.

As Popham suggests, “If heavy importance is being given to students’ performances on state tests for which there is no evidence supporting such an evaluative usage, then teachers (I would add principals too) might wish to engage in further study of this issue so that, armed with pertinent arguments, they can attempt to persuade educational decision makers that more appropriate evidence should be sought.” In other words, all educators, administrators, and teachers need to study how North Carolina or any state is using test scores to determine educator effectiveness.

Administrators owe it to their teachers, and themselves, to understand that some of these tests were never designed to determine educator effectiveness, so that data needs to be viewed with skepticism. And, I would add that the manner in which these Measures of Student Learning were developed and are administered may not allow them to draw valid inferences about teacher quality. Test scores in North Carolina currently are only 1/6th of the teacher evaluation, and effective administrators are going to keep this in mind and not let the allure of numbers numb them to the other 5 standards.