Chait all wet on high-stakes testing

Jonathan Chait has been a critic of the perverse incentives that distort our healthcare delivery system.  But he is willfully oblivious to the perverse incentives created by overemphasis on high-stakes testing in public school systems. 

Here's his reaction to the Atlanta cheating scandal, in which former Superintendent Beverly Hall and 34 other 34 teachers, principals and administrators were indicted on allegations that they “conspired to either cheat, conceal cheating or retaliate against whistle-blowers in an effort to bolster C.R.C.T. scores for the benefit of financial rewards associated with high test scores”:

[Eugene Robinson's] factual premise — that connecting teacher and principal incentives to student achievement leads to more cheating — is probably true. Is this a reason to get rid of incentives? No, it isn’t.

Incentivizing any field increases the impetus to cheat. Suppose journalism worked the way teaching traditionally had. You get hired at a newspaper, and your advancement and pay are dictated almost entirely by your years on the job, with almost no chance of either becoming a star or of getting fired for incompetence. Then imagine journalists changed that and instituted the current system, where you can get really successful if your bosses like you or be fired if they don’t. You could look around and see scandal after scandal — phone hacking! Jayson Blair! NBC’s exploding truck! Janet Cooke! Stephen Glass! — that could plausibly be attributed to this frightening new world in which journalists had an incentive to cheat in order to get ahead.

The incentives Chait is subject to as a journalist are not comparable to those now burdening and in too many cases corrupting school districts.  Even if, say, 40% of Chait's compensation were based on the traffic he drives to New York -- or worse, they degree of admiration his writing inspired in reader comments -- they would not be comparable, because the statistical feedback would stem directly from what he produced, rather than from the myriad factors that shape students' learning, which have proven devilishly difficult to tease out.

More broadly, the degree to which incentives induce corrupt, dangerous, short-sighted, or otherwise nonproductive and immoral behavior varies widely from field to field and from design to design (though they probably are generally worse than we assume). The issue is not whether performance incentives are always and everywhere wrong, but what is the effect of the incentives incorporating student test results to evaluate teacher performance currently being implemented in states and school districts across the nation? In the highest profile cases, i.e. those districts purporting to post "superman"-style results, there's good reason to believe that the effects mirror those alleged in Atlanta (as summarized by Dana Goldstein):

The Atlanta indictment alleges a top-down conspiracy, and Hall may or may not have explicitly directed her underlings to erase students’ wrong answers. But she fostered a no-excuses culture obsessed with test-score gains; fired principals and teachers who questioned this agenda; and, according to the grand jury report, concealed allegations, from 2006 on, that one of her favored principals, Christopher Waller, was leading an institutionalized cheating effort at Parks Middle School. In fact, even after a district investigator found evidence of cheating at Parks, Waller was rewarded with at least $17,500 in performance bonuses.

Michael Winerip, reporting in the Times, adds detail:

[A teacher who cooperated with prosecutors] said teachers were under constant pressure from principals who feared they would be fired if they did not meet the testing targets set by the superintendent. 

Dr. Hall was known to rule by fear. She gave principals three years to meet their testing goals. Few did; in her decade as superintendent, she replaced 90 percent of the principals. 

Teachers and principals whose students had high test scores received tenure and thousands of dollars in performance bonuses. Otherwise, as one teacher explained, it was “low score out the door.”

Hall, who received $500,000 herself in performance bonuses and was lionized by Education Secretary Arne Duncan, was typical of the current regime's superstars:

Her focus on test scores made her a favorite of the national education reform movement, nearly as prominent as the schools chancellors Joel I. Klein of New York City and Michelle Rhee of Washington. Like them, she was a fearsome presence who would accept no excuses when it came to educating poor children. She held yearly rallies at the Georgia Dome, rewarding principals and teachers from schools with high test scores by seating them up front, close to her, while low scorers were shunted aside to the bleachers. 

 The evidence is strong that the problem is pervasive. Goldstein summarizes:

Is Atlanta an isolated case? The extent of the top-down malfeasance under Beverly Hall may be unprecedented, but as I report in this Slate piece, there is reason to believe that policies tying adult incentives to children’s test scores have resulted in a nationwide uptick in cheating. An investigation by the Atlanta Journal Constitution found 196 school districts across the country with suspicious test score gains similar to the ones demonstrated in Atlanta, which statisticians said had only a one in 1 billion likelihood of being legitimate. A 2011 study by USA Today of test scores from just six states found 1,610 instances in which gains were as likely to be authentic as you are likely to buy a winning Powerball ticket. Absent independent, local investigations of suspected wrongdoing—which are rarely conducted—we simply cannot know the full extent of the cheating, which makes it difficult to assess whether the United States ought to continue down the road of tying teacher and administrator pay and job security to kids’ standardized test scores.

While the incentives to show test score improvements are massive, the incentives to detect cheating are miniscule, comparable to the bankers' incentives to find evidence of bad loans  in mortgage securitizations: a moral compass and a regard for long-term as opposed to short-term interests. Test-score alterations, moreover, like explicit, deliberate fraud in all environments shaped by skewed incentives, is only the tip of the spear. The climate of fear alleged in the Atlanta system under Beverly Hall generates a range of softer cheating  methods, such as strategically placing weaker students next to stronger ones, or encouraging weak students to stay home on test day, and still more pervasive ill effects, such as relentless teaching to the test.

Minimizing the problem leads Chait to rather nonsensically minimize the importance of addressing it:

There’s a useful debate to be had over how to design the criteria for measuring effective teachers. You can minimize cheating through effective oversight – a few Atlanta-style prosecutions will make most cheaters think twice. Granted, if your top priority in designing the system is to minimize cheating, then yes, you should avoid anything that incentivizes more effective teaching, or punishes bad teaching.

But minimizing cheating is a terrible top priority. The top priority should be teaching students better.

You can't teach students better if your measure of how well you're teaching them generates corruption, or if it distorts and inhibits what you teach them and it what way.

That's not to say that there should be no standardized testing, or that results should not be carefully incorporated as an element in teacher evaluation.  There's good evidence, though, that there's something rotten in the way testing and teacher evaluation are now being done in many U.S. school systems at the federal government's prompting. Chait ignores that evidence.