Sean Oxendine's blog

On Pennsylvania Polling

There’s been quite a dust-up in the blogosphere over the ARG poll showing Obama up four in Pennsylvania, but also showing a 53-39 Democratic/Republican registration edge. This was observed by Jim Geraghty at the Campaign Spot, and followed up by writers like Kavon Nikrad at Race42008.com. Today, there’s a SurveyUSA poll out showing a six-point Obama edge, but also showing a 51-38 D/R edge.

Some have called this weighting into question. There’s some reason for this. The 2006 exit poll – a very good year – showed a 43-38 Democratic edge in the actual electorate. The 2004 exit poll showed a 41-39 Democratic edge. 2002 data aren’t available, but the unweighted 2000 data show a 42-39 Democratic edge.

So in a very good Democratic year, a mildly bad Democratic year, and a middling year, we’ve seen Democrats fairly steadily trading in a 41-43 percent range, while Republicans are in the 38-39 percent range.

The defense for SurveyUSA and ARG is that they are pegged directly to the Pennsylvania voter registration statistics, which presently shows a 51-38 Democratic edge.

One question we can ask is “what did the party registration numbers look like in 2000, 2004, and 2006?” Do Democrats traditionally underperform their registration?

The answer to this is “yes, they do historically underperform.” In 2000, Democrats held a 48-42% edge over Republicans. In other words, the Democratic electorate was 6 percent less Democratic and 3 percent less Republican than registration numbers showed

In 2004, the registered electorate was 48-41% Democratic. Democrats underperformed their registration by seven points, while Republicans underperformed by two points.

In 2006, the registered electorate was 48-40% Democratic. Democrats underperformed by five points while Republicans, again, underperformed by two.

So there is something to the argument that Democrats typically perform worse than their voter registration numbers would indicate.

Unfortunately, comparing registration data to exit polls results is a little like comparing apples to oranges. Someone might be registered Democrat, but consider themselves an independent who is registered in one party or the other for purposes of the primary or whatever. So the registration statistics might show one thing while the exit polls show another. That would actually make things a wash. While we might be able to expect the election day registration numbers to be more heavily independent than registration statistics would show, we can probably also assume that Obama will get more Democrats than polls predict, since independents are disproportionately included in the Democrats’ polling sample. A more intriguing possibility is that much of the Democrats’ surge in voter registration comes from Democratic-leaning independents becoming actual Democrats. Regardless, I don’t think that you can engage in the kind of re-weighting that Geraghty engaged in with ARG, because I think that you’re conflating two different data sets. And incidentally, though SUSA tended to be pro-Kerry in PA in 2004, its final poll pretty well nailed it.

(1) I don't like the decision to suspend his campaign.  For one thing, I think it is lose-lose for him.  If there is a deal, it is going to be extremely unpopular with large swaths of the electorate.  The politically savvy decision, at least in the short term, is to rail against Wall Street bailouts while hoping the deal goes through.  If there is no deal, John McCain looks like he can't get things done.

(2) I also don't like it because it reeks of desperation.  This is especially apparent at a time when the bottom really is falling out for McCain's polling numbers.  It's still a long way to November, but it isn't THAT long.

(3) On the other hand, just because I don't like it, doesn't mean that he didn't have to do it.  It's a desperate sounding move, but I do think with what's going on in the polls, he had to do something.  And he may have already been in a no-win situation.  As a friend of mine put it, "worst of all worlds he let Paulson and Bush drive bailout story for a week, then show up at a national security debate and have America think of him as the out of touch guy still hung up on the Surge.   Now he is a leader, seeking to achieve bipartisan consensus to avoid catastrophe.   He neutralizes Obama's advantage because Obama has to fall in line behind the bailout Dodd will no-doubt drive through congress.  Now the economy is moot as they both support the same deal, only McCain is seen as a leader and Obama a follower.  Debates resume next week, and McCain not seen as out of touch."

(4) Since a lot of the cash for the campaign is going to come from the RNC and 527 groups, I'm not sure what the decision to "suspend his campaign" really means, in practical terms.

(5) So it may well be that this is the best way to play an awful hand.  Time will tell.

In this weeks' edition of the OxMethod -- which weights polls based on their prior accuracy and on their age (phasing out polls after 30 days) -- we see some of John McCain's bounce starting to fade. This will likely become more dramatic over the next few days. Senator Obama leads with 273 votes to McCain's 265. Interestingly, the closest state to flipping to McCain is New Hampshire -- if it were to do so, it would be an electoral tie. Also, it is worth noting that shifting the map 3.4 points toward Senator McCain would make it reflect a 2.5 point win for McCain -- the same win Bush had. If that happened, 34 of the 50 states would be polling within 4 points of the actual spread in 2004. This is amazing. 8 states have moved four or more points toward McCain relative to 2004 (MN, WA, ID, NY, AZ, KS, MA, TN) and 8 states have moved four or more points toward Obama relative to 2004 (WV, IA, UT, WY, MT, ND, IN, and HI). That is remarkable stability in the map. This Map: Last Map Trend Chart

 At fivethirtyeight.com, Nate Silver has a formula for allocating undecideds.  It is a pretty simple formula.  He looks at how Obama overperformed or underperformed relative to the polls in the Democratic primaries, and then applies the formula to the general election.  I’m having trouble replicating his results – the best I could get it was an r-square of .61, with only three of the variables at significant at 95%, but that could be due to different datasets used and different estimates of variables (I’m also not entirely certain when he’s using dummy variables). 

There are a lot of little nits one could make (Kentucky doesn’t border Illinois?), and I think there may be some multi-collinearity jacking up his r-square (hard to say), but the major complaint is this:  One cannot make the jump from primary electorate to general electorate without first knowing who the undecideds were in the primary, and then determining that the same groups are similarly undecided in general.  In other words, if 10% of the undecideds in SC were African American in the primary (plausible, given Obama’s late surge and a choice between two Democrats) but only 1% of the undecideds are African American now (again, plausible, though I could make an argument either way – maybe the 10% of AA’s who vote Republican are all undecided now), it is hard to predict with accuracy how the undecideds will break overall in the general based on how they broke in the primary. 

 Even if you could, it is difficult to draw hard conclusions, because it is impossible to know, say, if the African Americans who are undecided in the general share salient characteristics with African Americans who were undecided in the primary (ie, if the undecided AAs today typically vote Republican, they may well be more likely to break toward McCain than AAs were to break toward Hillary, though again, I could make the argument either way). 

 And what if pollsters are correcting now for an anticipated surge in AA turnout?  What happens when we move from an electorate where AAs comprise around 50% of the electorate to one where AAs will comprise no more than 30%, save in a few states?  Won’t this mute any reverse-Bradley effect significantly?

 And unfortunately, at the state level, this is impossible to do at more than the anecdotal level.  Only about 5% of any given state self-describes as “undecided” at this point, so you’re allocating a very small dataset.  And given that subsamples have even larger margins of error than samples, it becomes impossible to say scientifically that “5% of the undecideds are white, 1% of the undecideds are African American” and make any type of conclusions about the makeup of undecideds.   If you see the same pattern repeating in poll after poll (as we do with older voters being more undecided), then you can make some observational comments, but you can’t make scientific comments. (Yes, error margins shrink as you approach an extreme, but not enough to completely fix this problem).

 Silver recognizes this, and writes that “it does not necessarily follow that the patterns exhibited by undecided voters in the primaries will match those in the general election.”  His response is a non-answer:  “based both on my research and on what I've been hearing from people on the ground, it's apparent that the public polling in general is not terrific, and that if we have an instinct about where the polls are more likely to come in high or low, we probably ought to follow it.”  Which is fine, but for a regression model that is supposed to be scientific and non-arbitrary, simply going with our instincts seems to be awfully arbitrary (especially where part of the model for what our instincts are is going to be based on polling data regarding the number of undecideds).

 I do think, though, that we can glean some information about the undecideds at the national level.  Gallup is kind enough to give the demographic crosstabs for the entire weeks at a time.  This works out to around 6,000 voters.  The crosstabs then, even for a group that makes up only 10% of the population (like African Americans), would have reasonable error margins (around +/- 4% here).  We can make some observations  -- though it is difficult to draw definitive conclusions -- based on this:

Someone in the comments was kind enough to point out that Nate Silver has penned a little ditty over at FiveThirtyEight.com entitled “Bad Math and the Bradley Effect”, which purports to be a response to my earlier article speculating on what the scenario is for McCain to win. Ideally I’d spend my Sunday early afternoon doing something more productive, but since he’s basically challenging my intellectual honesty by accusing me of cherry-picking polls, I suppose that it deserves a response.

Silver’s biggest complaint with my methodology seems to be that I define my dataset as something less than every election that was held during the primary season. Of course, one of the biggest challenges of any type of statistical analysis is defining your dataset. I did not go into my reasoning for my dataset selections, since (a) I was trying to explain the scenario (which I considered unlikely) for McCain winning, not explore under strict adherence to political science principles whether a given effect occurred or not and (b) it is a blog post, not an academic paper. But since it has been brought up, the reasoning was as follows.

I excluded caucus states, such as Iowa and Nevada. I think anyone with a basic understanding of the dynamic of caucuses would understand my decision to do this, and would know that it had nothing to do with whether or not the results in those states fit any particular hypothesis. They were excluded because they are relatively low turnout affairs (Iowa being something of an exception to this) with Byzantine voting rules where it is almost impossible to know the true first-choice preferences of everyone who turns out to vote. These are rules that zero states will follow in November. And the caucus states require attendees to have all-day availability, which means only a certain type of person can attend. Because of this, they are notoriously difficult to poll, and it is more likely that the pollster error is just due to a bad turnout model than it is anything else. This is a reasonable choice (as Silver actually seems to concede). 

And while it is always good to have as many datapoints as possible, in my estimation, including caucus states meant that I would be comparing apples with oranges, which is actually the last thing you want to do when hypothesis-testing. 

Silver knows this, given that in his recent piece on cellphones, he excludes a number of pollsters from his dataset, for a variety of (perfectly justifiable) reasons. While it would have given him more datapoints if he had included pollsters who only recently began polling or who conducted internet polling, Silver correctly decided that including them would damage the integrity of his data. It’s the same process with excluding caucus states.

 Next, I excluded Florida because it was a part of the Old Confederacy (see below) and also because, as we were reminded again and again in the primaries by Obama supporters, neither candidate campaigned there. It is therefore difficult to use its results as indicative of how the candidates would have fared in a full-on election.

Finally, I excluded the results from the Old Confederacy, but left in Texas. My reasoning for excluding these states is simple, and was (perhaps too succinctly) summarized in the following phrase in my post, where I described the South as a region “where [Obama] was buoyed by unusually high African American turnout...” In other words, the makeup of the electorate in those states is fundamentally dissimilar from the other states in the dataset. African Americans made up 48% of the electorate in Alabama, 49% in Georgia, 47% in Louisiana, 48% in Mississippi, 34% in North Carolina, 55% in South Carolina, 29% in Tennessee, 19% in Texas, and 29% in Virginia. This is unique to the Confederacy: The only other states where African American turnout exceeded 20% of the electorate were Delaware (29%), Illinois (21%), New Jersey (23%) and Maryland (37%).

And it is pretty clear that the African American population has a significant effect on whether Obama over- or under-performed in the polls- - the r-square when you compare AA% in a state to the pro- or anti-Obama effect in the state is .45, with a t-stat of 4.6 for the variable (which seems to argue for the existence of such an effect). Since the goal is to figure out how white voting behavior will change on election day relative to the polls, if at all, it seemed to argue for excluding states where whites make up an unusually small portion of the electorate. I guess another approach in states like NC and SC would be to compare the proportion of white voters Obama was predicted to get by particular pollsters versus what exit polls showed, but this has its own problems. And I’m lazy.

Silver argues that “the particular geographics [sic] of the Confederacy are not especially relevant electorally.” In many contexts that may be true, but in this context it is not. Having African Americans comprise around 50% of the electorate – something, incidentally, that many pollsters weren’t predicting, especially at first – would drown out any Bradley effect in a way that wouldn’t occur in other states where African Americans comprise a much smaller portion of the electorate. Moreover, the South behaved differently than the rest of the country in the primary season. You could explain 80% of the variance – at the county level! – of the voting in the South between Hillary and Obama solely by looking at the percentage of African American and college-educated voters in the county. That is unique to the South, and did not hold up elsewhere. Finally, there aren’t any states in the country that will have African Americans making up 40-50% of the electorate, except maybe Mississippi.

 I’ll also add that Silver’s assertion that Kentucky and Tennessee are two peas in a pod is just silly. Tennessee has no analogue to the Old Seventh congressional district in Kentucky, which is mining country organized by the UMW in the 30s, and which is basically an extension of West Virginia. Kentucky has no good analogue to Memphis, and its Fifth District is only a much smaller and weaker analogue to Tennessee’s First, Second, and Third Districts. And more importantly, the African American percentage in the Kentucky Democratic primary electorate was 9%, versus 29% in Tennessee. For whatever similarities they might have, their Democratic primary electorates are very dissimilar in the way that is most germane to this model.

 The decision to re-include Texas is probably the best criticism of my methodology, but it is also the one with the least overall effect, given that the Obama barely underperformed there. The reason for including it is pretty obvious if you look at the statistics above, and consider my overall reasoning for excluding the Confederacy. The AA population is comparatively small relative to the remaining Southern states, and is more akin to the general Democratic electorate. Perhaps it would have been simpler to say “exclude all states where African Americans comprised over 25% of the voting electorate,” which would have had the same effect, although Maryland would have been excluded, and would have been more consistently applied. The only problem is that any percentage applied would have been probably even more arbitrary than the methodology I chose – why not 20%, which would have excluded New Jersey? Why not 30%, which would have included TN and VA? Regardless, if it makes people feel better, we can still exclude Texas, which changes my results a couple hundredths of a point.

I’m not 100% certain, and am genuinely curious, how Silver is calculating the confidence intervals for my results, so I can’t really respond to the statistical significance charge. I must admit, however, that I find it odd for Silver to chide me for not reaching the 90th percentile in statistical significance, given that his data dredging...er...stepwise regression process demands significance only at the 85th percentile (just do cntrl-F and enter 85 to find it). The use of RCP averages versus pollster.com is much easier to defend. As you know from watching the primary results, Obama surged in late January after his South Carolina win. I’m no expert in Pollster.com’s methodology, but my impression from looking at some of their results is that they are much slower to phase out old polls than was RCP. Indeed looking at the FAQ, I’m not certain they phase out old polls at all. In some ways and in some applications, their estimates are superior to RCP’s, but in a race where you have a last minute surge by a candidate, an average that only includes the last few days’ polling is going to be the best estimate to use.

 There are several examples to point to of how this affects the results, but perhaps California is the best one. RCP’s final average for California included polls concluded only four days before the primary, which fully captured the poll bounce Obama was seeing. This is demonstrated in their chart of the race. The Pollster.com polling shows a much more gradual improvement in Obama’s polling, in part (I think) because the relatively recent, but nonetheless outdated polling from a few weeks prior was dragging their analysis down. In other words, my sense was that, especially in the Super Tuesday polls, the Pollster.com method was understating Obama’s strength in the polls, and hence overstating the degree to which he overperformed. 

Indeed, if you look at Silver’s chart, which is organized sequentially, the numbers become an awful lot redder (indicating Obama underperforming) when you get past Super Tuesday, and there is a lot less difference between his findings regarding Obama’s performance and mine. Take the following chart, which shows my result (a negative value means Obama underperformed), 538’s results, and then the difference between the two. I’ve highlighted any state where I found a pro-Obama effect at least two points higher than Silver’s in blue, and any state where I found an anti-Obama effect at least two points higher than Silver’s in red. The Super Tuesday states are between AL and TN. Super Tuesday is really where the Silver and I find different results, after that we are rarely more than a couple points off in our results. I think this is almost entirely due to the last-minute poll bounce that RCP captured, and which Pollster didn’t. 

 The criticism that the results aren’t robust if they change when the averaging mechanism is changed is also silly. If one averaging mechanism somehow systemically biases the results relative to other averaging mechanisms, which I think is the case here, then of course which one you choose makes a difference, and it should make a difference. This is especially true if the averaging mechanisms are interpreting different data, which I also suspect is the case here. 

In the end, I suppose reasonable minds can differ over whether to use Pollster.com or realclearpolitics.com. Without really knowing how Pollster’s regression works, it is probably impossible to argue it conclusively. But I really thought Silver's response to using RCP rather than Pollster was silly. It tipped me off that his principal interest is in NOT finding the Bradley Effect, rather than letting the chips fall where they may. If he finds no Bradley effect and I find a Bradley effect, there is evidence for the Bradley effect. That doesn't mean that it is there or not – and remember, the real point of the article was to speculate on what would have to happen for McCain to win, not to prove or disprove anything -- all it means is that more research must be done. 

Finally, Silver writes:

The other, more important question is why we should simply dismiss the results in the South, where Obama significantly overperformed his numbers, by 7.2 points on average, according to my definition of the region and by 9.9 points according to his…

The easy answer is that I don’t dismiss it. Had Silver bothered to read the entire article, he would see that I wrote, under the heading “Youth/African American Vote”:

Regardless, I’ve covered this to some extent here, with the salient point being this: yes, Obama will likely increase African American turnout, but the states where this could make a real difference – with the exception of Virginia – are either so deeply red or so deeply blue that it is unlikely that AAs will be game-changers. Improving Obama’s vote share in Mississippi from 40% to 43% doesn’t do him a lick of good.

Emphasis added. Silver and I seemingly agree that any pro-Obama effect from high African American turnout is likely to be muted in the general election, with AA’s share of the electorate likely to be at least halved relative to the primary election. Higher African American turnout might absolutely flip Virginia if McCain is only leading by a couple of points on election day, as he is today, and as I concede in my article. It might make a difference in North Carolina, but given that McCain is, as of this writing, leading by nine in the RCP average and five in Pollster.com, I’m not sure it will happen.  Even Silver’s own averages have McCain up five in NC, though they only have him up one in Florida (notwithstanding that only one poll this month has Obama leading, and that only three polls this month have McCain’s margin at less than five, but the shortcomings of his model are a story for another time).  I don’t think a reverse Bradley effect will have much of an effect in Florida either, where African Americans make up an even smaller portion of the population than they made up in Texas: only 12% in 2004. 

But as of this writing, McCain (using Pollster.com) is down three in PA, and down three in MI, and up two in OH. This all may change by election day – heck, it might change this week – but as of right now, I would take a couple extra points for McCain in PA, MI, and OH in exchange for giving up a couple of points in VA, NC, and FL. In a heartbeat, without thinking twice. 

And now I’m going to go play with my kid.

InTrade currently has the candidates roughly tied. As of this writing, McCain is at around 49 and falling, while Obama is at around 51 and rising. Still, that’s basically a tie. The question is, where do we go from here?

The arguments for why Obama will ultimately win this have been explored ad nauseum over the last several months in a variety of contexts. They are straightforward: Parties rarely win third terms, President Bush is unpopular, Obama is charismatic (while McCain is not), the country is ready for change (which McCain does not embody), and Obama’s cash and ground game advantages will translate into a massive surge in Democratic support on election day, which will improve Democratic performance up and down the ticket.

Less has been said about why McCain will win. Maybe this is because the possibility seemed so absurd to many viewers of the political process not two weeks ago, that little commentary has been done on the actual scenario for a McCain win, or why things look different than they looked a few months – or even weeks – ago.

Let me be clear from the outset though – this isn’t to say I think McCain will win. I actually still think Obama is the favorite by reasonable odds -- probably 55-45. What I’m interested in is this: If we wake up the morning after election day looking at a McCain Presidency, and people talk about the “shocking” result, what were the warning signs? What should we have been looking at in order to see this coming?

I think a good way to approach this is to take a look at all of the supposed Obama advantages, and look at the evidence why they won’t play out the way that we have expected that they would

Well, the bounce period is over, and we can finally get some real sense of where the race stands post-Convention.  The bottom line is a lot has moved, but not much of importance has changed.  We see red states have gotten a lot redder, blue states have gotten a little redder, and swing states have pretty much stayed the same.  McCain still trails narrowly in the electoral college, 273-265.  Interestingly, the population-weighted state poll average shows a national vote of 46.5-45.6 McCain, while the national polls show  a 46-45.8 McCain lead.

Note:  Post-convention, I only weight polls taken within the last 30 days.  If there are no polls, I will use the last available poll.  In October, this will switch to 15 days.

Previous map:

Latest map:

Results trend:

For much of this cycle, the news networks have been atwitter that Barack Obama has been pursuing a 50-state strategy, or something akin to that. The premise of the 50-state strategy is, of course, that a campaign should not leave electoral votes on the table, but rather should attempt to compete in all 50 states. Wikipedia's entry for Howard Dean describes the strategy succinctly:

After Dean became Chairman of the DNC, he pledged to bring reform to the Party. Rather than focusing just on 'swing states,' Dean proposed what has come to be known as the 50-State Strategy. The goal, the DNC says, is for the Democratic Party to be committed to winning elections at every level in every region of the country, with Democrats organized in every single voting precinct in the country. State party chairs have lauded Dean for raising money directly for the individual state parties. Dean's strategy uses a post-Watergate model taken from the Republicans of the mid-seventies. Working at the local, state and national level, the GOP built the party from the ground up. Dean's plan is to seed the local level with young and committed candidates, building them into state candidates in future races. Dean has traveled extensively throughout the country with the plan, including places like Utah, Mississippi, and Texas, states in which Republicans have dominated the political landscape.

It is an incredibly popular idea with the netroots, and may even be considered their organizing principle. I also think, and have always thought, that it is an uncommonly dumb idea.

Res Ipsa Loquitor.

This is incredibly important.  The Democrats' main argument for why they couldn't lose this election is the enthusiasm gap.

A few days ago, Joe Trippi published an blog post indicating that he believed it was essentially a tied election, but that the tie-breakers went to Obama.  We're still just at the very beginning of the actual election season, so its a little hard to predict how things will play out.  But I think he's probably off-base, and I think a good argument can be made that the tie-breakers favor McCain.  Consider Trippi's analysis:

Polls are likely underestimating the turnout of young voters because many of these voters use cell phones and pollsters are having a difficult time including their views with accuracy. Obama has a big advantage with these voters. As a potential tie-breaker – ADVANTAGE OBAMA.

Nothing drives me battier than this argument about why we can't trust the polls, which we will hear over and over again.  This has been floating around at least since 2002.  And yet there is zero evidence that cell phone-only households are having an effect on the reliability of telephones. 

Riddle me this:  Were the elections in 2002, 2004, and 2006 skewed -- even a point or two -- toward the Democrats or Republicans relative to the polls?  If cell-phone only households are even starting to have an effect, I think we should have at least started to see it in the results, with Democrats overperforming the polls.