In Response To Nate Silver

Someone in the comments was kind enough to point out that Nate Silver has penned a little ditty over at FiveThirtyEight.com entitled “Bad Math and the Bradley Effect”, which purports to be a response to my earlier article speculating on what the scenario is for McCain to win. Ideally I’d spend my Sunday early afternoon doing something more productive, but since he’s basically challenging my intellectual honesty by accusing me of cherry-picking polls, I suppose that it deserves a response.

Silver’s biggest complaint with my methodology seems to be that I define my dataset as something less than every election that was held during the primary season. Of course, one of the biggest challenges of any type of statistical analysis is defining your dataset. I did not go into my reasoning for my dataset selections, since (a) I was trying to explain the scenario (which I considered unlikely) for McCain winning, not explore under strict adherence to political science principles whether a given effect occurred or not and (b) it is a blog post, not an academic paper. But since it has been brought up, the reasoning was as follows.

I excluded caucus states, such as Iowa and Nevada. I think anyone with a basic understanding of the dynamic of caucuses would understand my decision to do this, and would know that it had nothing to do with whether or not the results in those states fit any particular hypothesis. They were excluded because they are relatively low turnout affairs (Iowa being something of an exception to this) with Byzantine voting rules where it is almost impossible to know the true first-choice preferences of everyone who turns out to vote. These are rules that zero states will follow in November. And the caucus states require attendees to have all-day availability, which means only a certain type of person can attend. Because of this, they are notoriously difficult to poll, and it is more likely that the pollster error is just due to a bad turnout model than it is anything else. This is a reasonable choice (as Silver actually seems to concede). 

And while it is always good to have as many datapoints as possible, in my estimation, including caucus states meant that I would be comparing apples with oranges, which is actually the last thing you want to do when hypothesis-testing. 

Silver knows this, given that in his recent piece on cellphones, he excludes a number of pollsters from his dataset, for a variety of (perfectly justifiable) reasons. While it would have given him more datapoints if he had included pollsters who only recently began polling or who conducted internet polling, Silver correctly decided that including them would damage the integrity of his data. It’s the same process with excluding caucus states.

 Next, I excluded Florida because it was a part of the Old Confederacy (see below) and also because, as we were reminded again and again in the primaries by Obama supporters, neither candidate campaigned there. It is therefore difficult to use its results as indicative of how the candidates would have fared in a full-on election.

Finally, I excluded the results from the Old Confederacy, but left in Texas. My reasoning for excluding these states is simple, and was (perhaps too succinctly) summarized in the following phrase in my post, where I described the South as a region “where [Obama] was buoyed by unusually high African American turnout...” In other words, the makeup of the electorate in those states is fundamentally dissimilar from the other states in the dataset. African Americans made up 48% of the electorate in Alabama, 49% in Georgia, 47% in Louisiana, 48% in Mississippi, 34% in North Carolina, 55% in South Carolina, 29% in Tennessee, 19% in Texas, and 29% in Virginia. This is unique to the Confederacy: The only other states where African American turnout exceeded 20% of the electorate were Delaware (29%), Illinois (21%), New Jersey (23%) and Maryland (37%).

And it is pretty clear that the African American population has a significant effect on whether Obama over- or under-performed in the polls- - the r-square when you compare AA% in a state to the pro- or anti-Obama effect in the state is .45, with a t-stat of 4.6 for the variable (which seems to argue for the existence of such an effect). Since the goal is to figure out how white voting behavior will change on election day relative to the polls, if at all, it seemed to argue for excluding states where whites make up an unusually small portion of the electorate. I guess another approach in states like NC and SC would be to compare the proportion of white voters Obama was predicted to get by particular pollsters versus what exit polls showed, but this has its own problems. And I’m lazy.

Silver argues that “the particular geographics [sic] of the Confederacy are not especially relevant electorally.” In many contexts that may be true, but in this context it is not. Having African Americans comprise around 50% of the electorate – something, incidentally, that many pollsters weren’t predicting, especially at first – would drown out any Bradley effect in a way that wouldn’t occur in other states where African Americans comprise a much smaller portion of the electorate. Moreover, the South behaved differently than the rest of the country in the primary season. You could explain 80% of the variance – at the county level! – of the voting in the South between Hillary and Obama solely by looking at the percentage of African American and college-educated voters in the county. That is unique to the South, and did not hold up elsewhere. Finally, there aren’t any states in the country that will have African Americans making up 40-50% of the electorate, except maybe Mississippi.

 I’ll also add that Silver’s assertion that Kentucky and Tennessee are two peas in a pod is just silly. Tennessee has no analogue to the Old Seventh congressional district in Kentucky, which is mining country organized by the UMW in the 30s, and which is basically an extension of West Virginia. Kentucky has no good analogue to Memphis, and its Fifth District is only a much smaller and weaker analogue to Tennessee’s First, Second, and Third Districts. And more importantly, the African American percentage in the Kentucky Democratic primary electorate was 9%, versus 29% in Tennessee. For whatever similarities they might have, their Democratic primary electorates are very dissimilar in the way that is most germane to this model.

 The decision to re-include Texas is probably the best criticism of my methodology, but it is also the one with the least overall effect, given that the Obama barely underperformed there. The reason for including it is pretty obvious if you look at the statistics above, and consider my overall reasoning for excluding the Confederacy. The AA population is comparatively small relative to the remaining Southern states, and is more akin to the general Democratic electorate. Perhaps it would have been simpler to say “exclude all states where African Americans comprised over 25% of the voting electorate,” which would have had the same effect, although Maryland would have been excluded, and would have been more consistently applied. The only problem is that any percentage applied would have been probably even more arbitrary than the methodology I chose – why not 20%, which would have excluded New Jersey? Why not 30%, which would have included TN and VA? Regardless, if it makes people feel better, we can still exclude Texas, which changes my results a couple hundredths of a point.

I’m not 100% certain, and am genuinely curious, how Silver is calculating the confidence intervals for my results, so I can’t really respond to the statistical significance charge. I must admit, however, that I find it odd for Silver to chide me for not reaching the 90th percentile in statistical significance, given that his data dredging...er...stepwise regression process demands significance only at the 85th percentile (just do cntrl-F and enter 85 to find it). The use of RCP averages versus pollster.com is much easier to defend. As you know from watching the primary results, Obama surged in late January after his South Carolina win. I’m no expert in Pollster.com’s methodology, but my impression from looking at some of their results is that they are much slower to phase out old polls than was RCP. Indeed looking at the FAQ, I’m not certain they phase out old polls at all. In some ways and in some applications, their estimates are superior to RCP’s, but in a race where you have a last minute surge by a candidate, an average that only includes the last few days’ polling is going to be the best estimate to use.

 There are several examples to point to of how this affects the results, but perhaps California is the best one. RCP’s final average for California included polls concluded only four days before the primary, which fully captured the poll bounce Obama was seeing. This is demonstrated in their chart of the race. The Pollster.com polling shows a much more gradual improvement in Obama’s polling, in part (I think) because the relatively recent, but nonetheless outdated polling from a few weeks prior was dragging their analysis down. In other words, my sense was that, especially in the Super Tuesday polls, the Pollster.com method was understating Obama’s strength in the polls, and hence overstating the degree to which he overperformed. 

Indeed, if you look at Silver’s chart, which is organized sequentially, the numbers become an awful lot redder (indicating Obama underperforming) when you get past Super Tuesday, and there is a lot less difference between his findings regarding Obama’s performance and mine. Take the following chart, which shows my result (a negative value means Obama underperformed), 538’s results, and then the difference between the two. I’ve highlighted any state where I found a pro-Obama effect at least two points higher than Silver’s in blue, and any state where I found an anti-Obama effect at least two points higher than Silver’s in red. The Super Tuesday states are between AL and TN. Super Tuesday is really where the Silver and I find different results, after that we are rarely more than a couple points off in our results. I think this is almost entirely due to the last-minute poll bounce that RCP captured, and which Pollster didn’t. 

 The criticism that the results aren’t robust if they change when the averaging mechanism is changed is also silly. If one averaging mechanism somehow systemically biases the results relative to other averaging mechanisms, which I think is the case here, then of course which one you choose makes a difference, and it should make a difference. This is especially true if the averaging mechanisms are interpreting different data, which I also suspect is the case here. 

In the end, I suppose reasonable minds can differ over whether to use Pollster.com or realclearpolitics.com. Without really knowing how Pollster’s regression works, it is probably impossible to argue it conclusively. But I really thought Silver's response to using RCP rather than Pollster was silly. It tipped me off that his principal interest is in NOT finding the Bradley Effect, rather than letting the chips fall where they may. If he finds no Bradley effect and I find a Bradley effect, there is evidence for the Bradley effect. That doesn't mean that it is there or not – and remember, the real point of the article was to speculate on what would have to happen for McCain to win, not to prove or disprove anything -- all it means is that more research must be done. 

Finally, Silver writes:

The other, more important question is why we should simply dismiss the results in the South, where Obama significantly overperformed his numbers, by 7.2 points on average, according to my definition of the region and by 9.9 points according to his…

The easy answer is that I don’t dismiss it. Had Silver bothered to read the entire article, he would see that I wrote, under the heading “Youth/African American Vote”:

Regardless, I’ve covered this to some extent here, with the salient point being this: yes, Obama will likely increase African American turnout, but the states where this could make a real difference – with the exception of Virginia – are either so deeply red or so deeply blue that it is unlikely that AAs will be game-changers. Improving Obama’s vote share in Mississippi from 40% to 43% doesn’t do him a lick of good.

Emphasis added. Silver and I seemingly agree that any pro-Obama effect from high African American turnout is likely to be muted in the general election, with AA’s share of the electorate likely to be at least halved relative to the primary election. Higher African American turnout might absolutely flip Virginia if McCain is only leading by a couple of points on election day, as he is today, and as I concede in my article. It might make a difference in North Carolina, but given that McCain is, as of this writing, leading by nine in the RCP average and five in Pollster.com, I’m not sure it will happen.  Even Silver’s own averages have McCain up five in NC, though they only have him up one in Florida (notwithstanding that only one poll this month has Obama leading, and that only three polls this month have McCain’s margin at less than five, but the shortcomings of his model are a story for another time).  I don’t think a reverse Bradley effect will have much of an effect in Florida either, where African Americans make up an even smaller portion of the population than they made up in Texas: only 12% in 2004. 

But as of this writing, McCain (using Pollster.com) is down three in PA, and down three in MI, and up two in OH. This all may change by election day – heck, it might change this week – but as of right now, I would take a couple extra points for McCain in PA, MI, and OH in exchange for giving up a couple of points in VA, NC, and FL. In a heartbeat, without thinking twice. 

And now I’m going to go play with my kid.