Another silver bullet for election forecasting

My wife's hospital floor (she's a nurse midwife) is staffed by doctors and nurses from all over the world. Politics comes up quite a bit, at least in election season. One Nigerian nurse, whenever my wife expressed anxiety about Obama, would always respond, "Cindy, don't worry. God will take care of him."

As election day hurtled toward us, this nurse told Cindy, "At 10 o'clock on Tuesday, you will be drinking champagne."  Not strictly true, since at 10 that night Cindy was attending a birth (with a woman who pumped her fist and shouted "Obama!" between contractions). And the forecast was of course off by about 75 minutes.  But about as good as Nate Silver, no?

As for me, this "God will take care of him" mantra was relayed to me repeatedly. I did envy this lady her faith.

The real final score: Obama 11, Romney 1

Nate Silver does have a way with data visualization. His analysis of the Democrats' current electoral college edge includes a electoral scorecard that lists the 50 states in order of the margin of victory, beginning with Obama's wins and moving on to Romney's. The close states in the middle reveal a striking fact: Romney won only one state by a margin of less than eight points. Obama won eleven.

Forget for a moment the demographic contest -- one candidate's large advantage among whites vs. the other's larger advantage among all ethnic minorities.  Forget, too, the strategic plusses and minuses of pursuing independents vs. turning out your base. Forget national popular vote margins. The simple fact is that Romney won only one state that any Republican would not have won.  A dozen states were competitive, and Obama won eleven of them -- by margins that were increasing rapidly at the end, and exceeded the final polls. He outperformed on every front --turnout, targeted advertising, and ultimately, the debates. He just kicked Romney's ass across the political field.

Update: it occurs to me that Silver comes to a very different conclusion: that even if Romney had won the national popular vote by two percentage points, Obama still would have won the electoral college. That is, Obama's advantage was structural, and would be shared by any Democrat at present. That assumes, I believe, a proportionality between Obama's margins in the swing states and the popular vote totals. But most of the direct competing was done in the swing states.  The tipping point, in Silver's reckoning, was Colorado: that is, Romney would have had to win Colorado and every other state that Obama won by a lesser margin than Colorado (Virginia, Ohio, Florida) to win the election.  And Obama won Colorado by 4.7 percentage points -- quite a large margin for Romney to have overcome. Does that mean that Obama's advantage was structural, i.e., that no competent Republican could have overcome it this year?  I don't know.  The margin there, and in all the truly competitive swing states, seemed much smaller just a few days before the election.

Conservative debate watchers have a point, but...

On one hand, I am pissed that Republican spin on the last debate is getting some credence in the media -- Politico and ABC both retailed the hypothesis that the debate might strengthen Romney's position, since he came across as credible an unscary -- and the electorate doesn't much care about foreign policy.

On the other hand, I think that the conservative analysis digested by the Dish -- by Douthat, Levin, Lowry and Ponnuru -- has some validity. Lowry summarizes their collective take well:

I think Romney executed what must have been his strategy nearly flawlessly: reassure people that he’s not a bomb-thrower; project strength but not bellicosity; go out of his way to say how many Obama policies he agrees with to create a sense of his reasonableness; focus on the big picture of a world that seems out of control; get it back to the economy as much as possible; and communicate a real passion for the future.

In fact, my own reaction to Romney's performance, taken by itself, was somewhat similar.  I disagree, though, with this from Levin, and in that disagreement lies hope:

The Silver bullet for election prediction may need lengthening

Nate Silver argues that job growth during a presidential election year is the best single indicator of whether a president will be reelected.  After a series of calculations, he pegs Obama's"magic number" for job growth in 2012 at 150,000 jobs gained per month.

I have neurotically tracked Obama's trajectory -- approval ratings and national unemployment numbers -- against Reagan's for 2-3 years, and my intuition is in sync with Silver's conclusions (and partly shaped by his prior analyses, no doubt). Far be it from me with my sixth grade math skills to quarrel with the intricacies of his model. But there's one basic assumption that I would quibble with: that the relevant periods are necessarily defined by the calendar year.

Silver argues that the actual year of election matters a lot, and the three years prior, not so much. When conditions are bad early in a president's term, he may even get extra credit for a late-term surge. More specifically, Silver finds

that job growth during the third year of a president’s term has a positive effect on his re-election odds, while the coefficients attached to the first two years are negative.

But none of these results are statistically significant or particularly close to it; only job growth during the fourth year of a president’s term has a clear effect.

True, perhaps, as far as it goes. But why break the term into years? Why assume that voters' relevant perceptions track the movement of the earth around the sun?

As Silver indicated, the ur-case for unemployment figures tracking closely with election results is lucky Ronald Reagan, who in Silver's model benefited from the 2012 equivalent of 487,000 jobs created in 1984.  Yet my own personal break point for comparing Obama's home-stretch performance to Reagan's has been July in the year prior to election -- because July 1983 is the month in which measured employment surged most dramatically for Reagan, from 10.1% to 9.4%. While that particular break point may be arbitrary, the broader point is that the unemployment rate shrank most dramatically for Reagan in the second half of 1983, from 10.1% in June to 8.0% in January 1984. It fell further, to 7.4% in October 1984.  But morning in America shone brightest at dawn, in late 1983.

Obama's uncertain trigger finger, cont.

It is perhaps a measure of Obama's poor negotiating track record that as I read both Republican and Democratic assertions that Democrats have all the leverage in the fight over the payroll tax cut and unemployment benefits extension, I worry that Obama will find a way to cave.

Nate Silver, who asserts that Obama's recent poll bounce is likely due more to economic upticks than to his more confrontational stance over the last two and a half months, provides the rationale for another 'hostage negotiation':

Nate Silver's death knell for Obama's jobs measures

One nugget in Nate Silver's exhaustive statistical analysis of the degree to which each of  a broad range of economic variables have affected presidential elections since 1948 makes it pretty clear why Republicans are going to block almost every job-producing measure put forward by President Obama. It's this (charts omitted):

Another poorly performing variable is the unemployment rate. It has had essentially no relationship to election results at all.

However, while the unemployment rate had told us very little, the rate of change in the jobs market has been fairly meaningful. Here, for instance, is a comparison of election results to the rate of payroll jobs growth — the variable you often see highlighted when the government releases its jobs report on the first Friday of each month.

Question for political scientists

Political scientists who descend to the blogosphere are at pains to make the rest of us understand that, to paraphrase Yogi Berra, 95 percent of electoral combat is half structural -- that is, national election results are driven mainly by the state of the economy, at least in peacetime.  Candidates' skills and political strategy matter only at the margins -- though in a close election, the margin can be decisive. 

This structural view escapes determinism only to the extent that a) an incumbent can, in fact, affect economic conditions, including via ultimately destructive short-term jolts such as Nixon's imposition of wage and price controls; or b) economic conditions are mixed enough, or other factors such as war are salient enough, to put an election up for grabs (as in, for example, the recession we didn't know we had entered in late 2000, or Americans' unease with the course of the Iraq War in 2004).

Of course conditions are often mixed and murky. Nonetheless, I'd like to test the determinism of political scientists struggling to educate journalists and the rest of us, such as Brendan Nyhan, Jonathan Bernstein, John Sides & friends -- and of those who taken their data to heart, such as Jonathan Chait, Ezra Klein, and Matthew Yglesias. Oh, and one who defies category, master of probability Nate Silver.

So here's the challenge: using an economic measure of your choice, such as growth or shrinkage of personal income, or GDP, or unemployment (often dismissed as a lagging indicator), is there a threshold below which you would be prepared to say that Obama cannot be reelected?  Hedge it how you will: exclude military or environmental emergency or disaster, or terrorist attack,  or Republican nomination of a nutcase...hell, make it "Obama can't beat Romney if..." if you like.  And let's not make this too easy, as in Depression-level GDP shrinkage or unemployment.  Hewing as close to our current bad-normal conditions as possible, what's the can't-win economic marker for Obama?

UPDATE 11/3: Nate Silver created for himself a more sophisticated version of this challenge, gaming out various 2012 scenarios while averaging out incumbent's approval rating, GDP growth (or lack thereof), and challenger's ideological rating (moderate to extreme). Given Obama's current approval rating, with a candidate in the historical middle of the ideology scale -- Romney -- Silver rates the challenger's odds at 83% if there's 0% GDP growth over the next year, and at 40% if there's 40% growth.  That indicates almost a tossup given perhaps the likeliest scenario: 2-3% growth.  I must say, I find Silver's 3-factor model for prediction intuitively satisfying, finely calibrated -- check it out. 

Unemployment and the incumbent president: look at the last 16 months

Binyamin Appelbaum recently warned in the New York Times:

No American president since Franklin Delano Roosevelt has won a second term in office when the unemployment rate on Election Day topped 7.2 percent. 

In response, political scientist Seth Masket plotted out presidential elections since 1948 and concluded, "the unemployment rate does not predict presidential elections at all." Carlisle Rainey dug deeper into the data and found that there does seem to be some relationship when incumbent presidents are up for re-election. Analyzing a massive series of data subsets, Rainey concludes more broadly, "the data are consistent with both very small and moderately large effects."  Nate Silver, similarly, dives deep and comes up agnostic, concluding that the unemployment rate does reflect (if not drive) the incumbent's fortunes,but that the relationship between the unemployment rate and electoral fortunes is too entwined with a host of other factors to map out with any confidence.

Silver also looks at the direction unemployment moves over an incumbent's term. Here I think he is getting warm. But as he looks over the whole four years, he finds little correlation.

I have for some time been (nervously) tracking Obama's political fortunes against Reagan's, since it's often pointed out that both inherited a vicious recession; at Obama's low moments I've taken some comfort in the fact that Reagan's approval rating reached a nadir of 35% in January 1983, shortly after the unemployment rate peaked at 10.8%.  The impression I've soaked up is that the unemployment rate really matters in the 16-odd months before a president stands for re-election, when the electorate starts to tune in and measure him against the opposing party's candidates.  For Reagan, I've noted, a decisive turn came in July 1983, when the unemployment rate dipped to 9.4% from 10.1% the prior month. From there he had the wind at his back to October 1984, when the rate was 7.4%. Hence Reagan's landslide.  He asked us to stay the course, and the course started running downhill run when it counted.

The anguish of the data collectors

Jim Manzi views the dispute between poll analyst Nate Silver and pollster John Zogby in economic terms:

Silver intelligently combines multiple polls to make more accurate predictions than are usually achieved by any one individual pollster. On one hand, the math of this is irresistible – in the real world, voting models often work. On the other hand, it would be pretty uncomfortable for a pollster to combine his own results with various competitive poll results to achieve equivalent accuracy (or at least to do so transparently). So, the pollsters do all the tedious work to collect and analyze the data, and then Nate Silver comes along and creates all this value with it in a way that is hard for the pollsters to duplicate. You can see why this situation might upset the pollsters.

In every industry that combines data collection with analysis, there is an endless battle between the data collectors and the analysts. The data collectors bear the hard costs – people, office space, telecommunications, travel budgets, etc. – that are required for interviewing people, visiting stores, and so forth. Their nightmare world is to become commodity data collectors paid for their costs plus a small margin set by competitive bidding. Their typical defenses are to attempt : (1) to build proprietary methods for collecting superior data, or equivalent data at much lower cost, and (2) to integrate the analysis and the data into a single product, and forbid by contract the paying client from using this for other purposes. The analysts, on the other hand, want to have an open market in commoditized data and compete on analytical capability.

The pattern Manzi outlines is at work in the investment world as well.  On one level, conventional equity research, like polling data, is now subject to aggregation and analysis; services like Investars and Reuters' Starmine offer average ratings for equities as well as ranking analysts by the performance of past ratings on given stocks and sectors. Traditional fundamental research has become commoditized to a degree, as the research itself, like the material company information of which it's composed, can no longer be provided selectively to favored clients.  More broadly, active fund management itself is giving way to indexing, in large part through ETFs. Suzanne Duncan of the IBM Institute for Business Value has forecast,on the basis of a study polling financial executives and their clients, that within twenty years, 85--90% of assets under management will be invested in passive instruments such as index funds. (Not coincidentally, the study also found that only 10% of hedge funds actually produce any alpha, i.e. earn their exorbitant fees. The same is doubtless true of managed mutual funds.)

What Ailes Democracy in America

Nate Silver, delving into the cross tabs of the NYT/CBS Tea Party poll and noting that 59% of partiers have a positive impression of Glenn Beck (compared with 18% of the population as a whole), draws this conclusion:

Academic studies have found that FOX News in fact has quite a bit of ability to persuade voters and reformulate public opinion. That Beck (whose program debuted one day before Barack Obama's inauguration) has become one of its prominent voices may help to explain why the tea-parties have become so influential, and why public opinion appears to have shifted so dramatically within the past 18 months. FOX may, or may not, have a role in directly promoting the tea-parties. But its opinion programming has become the water-cooler around which the tea-partiers gather.

Perhaps there will always be a Father Coughlin (or a passel of them) to channel the rage of the paranoid and the ignorant, particularly in times of 10% unemployment, but I still find this disturbing.  Fox, like the Palins and Bachmanns it empowers, is a danger to democracy, a relentless amplifier of lies. And while I guess the U.S. had ample capacity to home-grow toxic political media, it still strikes me that Rupert Murdoch has managed to degrade political discourse on three continents (I assume he's done it in Australia as well as the UK and US). 

Two directions on tax reform

Demolishing AEI president Arthur Brooks's claim in today's WSJ that Americans do not favor progressive taxation, Jonathan Chait cites a public opinion snippet that raises a fundamental question about tax reform (my emphasis):

The Quinnipiac University poll found that 60 percent of Americans among both major political parties think raising income taxes on households making more than $250,000 should be a main tenet of the government's efforts to tame the deficit. More than 70 percent, including a majority of Republicans, say those making more than $1 million should pay more.

The question is: leaving aside the precise optimal level for the top marginal income tax rate, why is the highest bracket fixed at a mere $250k in family income?  Matt Yglesias and Nate Silver batted this question around in March '09. Silver:

 What the discussion over the top marginal tax rate ignores, however (and what Ygelsias picks up upon) is that this rate has been assessed at very different thresholds of income. In 1940, for example, the top marginal tax rate was 81.1 percent -- but this rate only kicked in once you made $5,000,000 or more in income, which is equivalent to about $75,000,000 in today's dollars.

Bracketing up: HCR, income redistribution and tax reform

David Leonhardt  points out today that the health care bill represents a significant check to the growth of income inequality over the past generation. This point about its tax impact triggered a memory:

A big chunk of the money to pay for the bill comes from lifting payroll taxes on households making more than $250,000. On average, the annual tax bill for households making more than $1 million a year will rise by $46,000 in 2013, according to the Tax Policy Center, a Washington research group. Another major piece of financing would cut Medicare subsidies for private insurers, ultimately affecting their executives and shareholders.

The memory was this: lefty blogger speculation at the dawn of the Obama presidency as to whether Obama would seek to reengineer tax brackets. Nate Silver:

What the discussion over the top marginal tax rate ignores, however (and what Ygelsias picks up upon) is that this rate has been assessed at very different thresholds of income. In 1940, for example, the top marginal tax rate was 81.1 percent -- but this rate only kicked in once you made $5,000,000 or more in income, which is equivalent to about $75,000,000 in today's dollars.

But today, the threshold where the top tax bracket kicks in isn't $75 million, or $5 million, or even $1 million ... it's a mere $357,700. The progressivity of the tax code stops there....

Reza Pahlavi seconds Nate Silver

Nate Silver's blog tagline, "Politics done right," amounts to an admonition: "do the math." To a verbally inclined baby boomer who felt free to blow off math in mid-eleventh grade, 538 has offered a sobering demonstration that while the truth may set you free, the math very often reveals the truth.

On Monday, Silver took a dive into Iranian pre-election polling data, which according to pollsters Ken Ballen and Patrick Doherty of Terror Free Tomorrow showed that Ahmadinejad was indeed likely to win the "real" vote count. Focusing on the large number of respondents who would not reveal or had not made a choice, paired with wide margins in favor of reformist policy positions, Silver found evidence of multi-level voter intimidation. His conclusion:

The point that few commentators are realizing -- Al Giordano is an exception -- is that this story really isn't about the way that the votes were counted. It's about whether Iran is capable at this point of having an election in which the democratic will of its electorate is properly reflected. If Ahmadinejad hired a bunch of thugs to hold every Iranian at gunpoint while they were casting their ballots, it would not have been difficult for him to get 63 percent of the vote -- indeed, he'd probably have wound up with very close to 100 percent. This would be an election -- and there would be no need at all to tamper with the results. But it wouldn't be an expression of democracy. We need to separate out those two concepts. Ahmadinejad, as far as we know, did not go so far as to hold anyone at gunpoint. But the tentacles of fear in Iran run deep.

Are good math skills are proxy for experience? Exiled Reza Pahlavi, the 48-year-old son of the last Shah of Iran, needs no numbers crunching to assess the regime that deposed his father. Here's what he told the WSJ:

"We have a major paradigm change right now," he said. "It's a mockery to call any of these theatrics an election process. However, given the limits the Iranian people have, that was the only way to show the whole world they are against the established regime. Now they have gone to questioning the very legitimacy of the regime, and it's unfolding before our very eyes."

Of course, Pahlavi is hardly an unbiased observer. Nonetheless, the congruence with Silver's analysis is striking.

I should add, too, that it doesn't require advanced math skills to contrast the poll numbers for the candidates with the poll numbers for policy positions. But it's the habit of mind that counts. And often, Silver's mathematical analysis goes far deeper.

How will Obama raise tax revenue?

Robert Reich hypothesizes that Ben Bernanke is helping the Obama administration prepare the ground for new taxes on the wealthy:

But Bernanke also wants to deliver a message to Congress, a message the White House doesn't want to deliver because it's politically awkward: Congress will have to raise taxes on the wealthy in order to finance universal health care and reduce looming budget deficits. Such tax increases won't slow down the economy because the wealthy don't spend that much anyway (that's what it means to be wealthy -- you've already got most of what you need), but may be necessary, at least to ward off inflation fears.

What sort of higher taxes on the wealthy? Bernanke didn't say, of course, but the White House has already floated limits on deductions and seems willing to consider taxing employee-provided health benefits for employees over a certain income. And maybe lifting the cap on Social Security payroll taxes, at least for workers earning over $250,000 a year.

More radical ideas batted around by Matt Yglesias and Nate Silver a couple of months ago: create new tax brackets far above the current top level of $357k -- or make tax brackets "infinitesimal," i.e. ratcheting up proportionately for every extra dollar earned. As Silver pointed out, the current top bracket threshold is anomalously low compared to those of prior eras.

Bracketing up: will Obama soak the superrich?

Given Americans' immersion in low tax ideology, how can Obama fund his ambitious plans long-term? Matthew Yglesias floats an idea that occurred to me years ago:

some day I should write again about the idea of making tax brackets infinitesimal so that there is no “top bracket.” This would have been unworkable 100 years ago, but with computers there’s no reason we can’t do it.

In other words, there should be an algorithm for perfectly progressive tax rates that ratchet up infinitesimally for every dollar earned, rather than bumping up abruptly at fixed thresholds.

More prosaically, Yglesias floats the idea of adding new marginal tax brackets above the current top level ($357k). While he minimizes the potential for raising significant revenue this way, Nate Silver starts the math and finds otherwise:

What the discussion over the top marginal tax rate ignores, however (and what Ygelsias picks up upon) is that this rate has been assessed at very different thresholds of income. In 1940, for example, the top marginal tax rate was 81.1 percent -- but this rate only kicked in once you made $5,000,000 or more in income, which is equivalent to about $75,000,000 in today's dollars.

But today, the threshold where the top tax bracket kicks in isn't $75 million, or $5 million, or even $1 million ... it's a mere $357,700. The progressivity of the tax code stops there....

The question, of course, is why there isn't a millionaires tax bracket now ... or even a multi-millionaires tax bracket. I haven't run the numbers, but I'm guessing that if you established a new tax bracket at, say, 40.5 percent, that started at incomes of $1,000,000 or more, this would bring in as much revenue to the government as restoring the $250K tax bracket (which is really $360K now given indexing to inflation) to 39.6 percent, as it was under Clinton.

Brushing this subject without quite hitting on it, meanwhile, Clive Crook gives the rationale for creating new upper-level brackets -- though he himself favors the regressive but broader-based VAT:

Not everybody would regard two-earner households with an income of $250,000 a year as rich; and many of the taxpayers in question have seen their retirement savings, college funds and housing equity destroyed. The scandal of widening inequality that still animates the Democrats' thinking is a story about the top fraction of one per cent of the income distribution, not the top end of the middle class.

I wonder if Yglesias hasn't in fact stumbled on Obama's as-yet-veiled long range tax plans. Obama claims that he is going to tackle the long-term sustainability of the Federal budget; he's meanwhile planning large increases in Federal spending while promising not to raise taxes on anyone earning under $250k; and a core commitment in his campaign was to roll back the galloping rise in income inequality that Crook alludes to. There is a lot of wiggle room to raise taxes on the superrich while staying well below past U.S. norms. It's probably either that or a VAT. Or both...

Silver bullet in the War on (electoral) Terror?

In this corner we have Nate Silver, data-crunching young technocrat with a dizzying array of charts and an astounding track record, coolly assuring us that John McCain's chances of winning the election stand at 3.8%, that state poll data remains strong for Obama, and that any national poll tightening is as yet insignificant. And in this corner, nailbiting TNR vet Noam Scheiber:

I'm not popping anything just yet, except maybe some sleeping pills to get me through the night... In the last few days, pretty much every tracking poll I trust (WaPo, Gallup, Rasmussen) and several I either don't trust (that would be you, Zogby) or don't have much of an opinion about (Kos, Investor's Business Daily) has shifted toward McCain, in some cases sharply. ...

As of this writing, Obama's lead in the national tracking polls looks to be around five points (I get 5.5 when I average all six of the trackers I mentioned, along with the Hotline and Battleground trackers, which haven't changed much in the last few days). If that drops two-to-three points, as it easily could in a week, I don't think it's crazy to think McCain will have a shot at winning Pennsylvania, Virginia, and/or Colorado. Unlikely, yes, but not crazy.

And then again today, ironically citing the more sanguine John Judis:

Judis makes a great point in his otherwise overly-optimistic post on the polls:

I mention the Bradley effect because I think, too, that McCain and Sarah Palin's attack against Obama for advocating "spreading the wealth" and for "socialism" and for pronouncing the civil rights revolution a "tragedy" because it didn't deal with the distribution of wealth is aimed ultimately at white working class undecided voters who would construe "spreading the wealth" as giving their money to blacks. It's the latest version of Reagan's "welfare queen" argument from 1980. It if it works, it won't be because most white Americans actually oppose a progressive income tax, but because they fear that Obama will inordinately favor blacks over them.

Agreed.

Worse, though I have no evidence for this (nothing new there), I worry that these insinuations are reinforced in the minds of working-class whites by the millions of African-Americans lining up early to vote for Obama. How sad for the country if McCain ends up jujitsuing something (i.e., record turnout) that should be a source of pride.

Silver is brilliant and has all the confidence of a young black box hedge fund manager. Scheiber, however, has been around the block a few times. Silver writes sanguinely about McCain's need to shave a half-point off Obama's cumulative poll lead per day and the historical unlikelihood of doing so. Scheiber speaks to that gut feeling that polls can shift a lot more suddenly than that.

I'm off to barackobama.com to contribute one last c-note.

UPDATE: Make that $120. Those clever devils have moved the defaults to base 6. I consent to be upsold!

P.S. Behind the cool Silver, there's the even cooler (and equally data-based) Obama...