2006 And The Bradley Effect

The debate about the Bradley Effect is flaring up again, as witnessed in this back-and-forth between Bill Greener and Nate Silver.  For those just tuning in, Silver and I have had our own back-and-forth about whether the effect occurred in the 2008 primaries.  See here , here, and here.  In short, I believe there is evidence to support something that looks an awful lot like the Bradley effect in the Democratic primaries. Silver does not. 

To be honest, as of right now the whole debate about the Bradley Effect is kind of silly, given that Obama is above 50% in enough swing states to win the election even if McCain wins every undecided.  And there is something surreal about liberal commentators protesting that America has substantially moved on past race in the last 20 years, while conservatives argue that we've barely evolved racially.

Nonetheless, this is of some academic interest, and if polling models are overstating turnout among the youth and Democrats (or understating Republicans), and the electorate looks more like what IBD and/or Battleground have been seeing, it may well be relevant. 

Overall Numbers

I think Silver’s criticism that Greener cherry-picked polls to make his best argument is a fair one.  Nonetheless, overall his argument about 2006 is much weaker than he admits.  The best we can say is that there is no conclusive evidence for a Bradley effect in 2006.  But there is still some evidence, and I think it is substantial.  Indeed, this is shown in his own table, which shows that in the five major races pitting African American candidates against white candidates, the white candidate overperformed his polling numbers on average by 3.6 percent, while African Americans overperformed by only 1.6 percent.   

Silver’s rejoinder is that this is not statistically significant.  But when you are averaging multiple polls (as we are with the RCP average) it isn’t clear how the error margin should be calculated.  The theory behind poll aggregation is that we can treat each poll the same as an  individual pollster treats the people reading the scripts within the larger polling company; just as pollsters aggregate employees’ results to get a large polling group, so too we can aggregate the pollsters to get a "super sample." 

There are obviously some problems with this – different pollsters use different methodologies – but I think if we get to the point of talking about statistical significance of aggregated polling, we’re already accepting that aggregating polling is an acceptable methodology.  With Tennessee, we are therefore looking at a sample of about 2700 respondents, which would be an error margin of +/- 1.89% for that race alone.  For all five races combined our sample size would be about 10,000, for an error margin of +/- .098%, meaning that we’re 95% certain that in 2006 the white candidate performed better than the polls said he should vis-à-vis the black candidate.

 And that is if we use 95% as our confidence level.  95% is a great if you need to be 95% certain of the answer.  But it isn’t written in stone that we have to use this if all we're looking for is at least some evidence of a phenomenon.  Silver knows this, since he only requires 85% confidence for variables in his regression analysis.  

Let’s say we’re just looking for a “more likely than not” standard, e.g., we just want to be 51% certain there’s a difference.  For Tennessee alone, the error margin would be about +/-.69%, and for the whole 2006 sample the error margin would be +/-.35%, meaning that we would be well beyond the threshold of being able to say “more likely than not” that the white candidate overperformed the polls. 

Again, there are fair criticisms of this approach (e.g. it is an apples to oranges comparison to compare Mason Dixon to Rasmussen, since they have different models and transcripts), but they are also criticisms of poll aggregations writ large.  Once you’ve accepted poll aggregation, you’ve accepted all of this. 

Tennessee Senate 

Now let’s look at the Tennessee Senate race a little closer, since this is often pointed to by Bradley effect skeptics as an example of a race where the Bradley effect didn’t occur.  First, it is worth noting that Corker’s lead in the RCP average was certainly inflated by an outlying Mason-Dixon poll showing him up 12.  If you remove that poll, you add two points to Ford’s polling average, and the break toward him is much less pronounced.

 Regardless, let’s take a look at the white vote over time in Tennessee.  SurveyUSA gives us a nice time-sequence of how whites said they planned to vote:

 9/9-9/11:  Corker 49%, Ford 44%

10/7-10/9:  Corker 51%, Ford 44%

10/22-10/24:  Corker 53%, Ford 42%

11/3-11/5:  Corker 57%, Ford 41%

Final CNN Exit Poll:  Corker 59%, Ford 40% 

In other words, whites behaved exactly like we would expect them to behave with a Bradley Effect.  It isn’t so much that Ford underperformed his polling among whites – though he did – it is that undecided whites steadily broke for Corker.   

Now maybe this isn’t Bradley effect; maybe whites really did respond to the short-lived Playboy bunny ad (which I personally think is a ludicrous claim).  Absent that advertisement, maybe they would have broken differently.  We don’t have crystal balls and can’t settle that argument, but what we can say is that there actually was behavior consistent with the Bradley effect in Tennessee. 

What caused Ford to overperform was that while polls showed Corker getting around 15% of the black vote, in reality he got 4%.  This is consistent with the “reverse Bradley effect” that Silver and I have both posited might help Obama in states with high African American populations if they are close (as VA and NC now are). 

Indeed, we can compare the 2006 race on a county-by-county basis to the 2002 race.  In 2002, a much better year for Republicans nationally than 2006, Lamar! Alexander beat Congressman and political scion Bob Clement by ten points.  Six years later, Corker beat Ford by a narrower 51-48 percent vote.  That's a swing of seven points toward Ford. 

But if you get into the microdata, the swing was by no means uniform.  Much of the swing was attributable to Shelby county – i.e. Memphis – which is 44.6% black, swinging 27 points toward Ford and Davidson county – i.e. Nashville – which is 23.2% black, swinging ten points toward Ford.    This was accompanied by a surge in turnout.  Corker actually got about the same number of votes that Alexander got in Shelby county.  But Ford received about 60,000 more votes than did Clement, in a race where turnout was only up 300,000 votes statewide. 

Indeed, of Tennessee’s 95 counties, only 18 showed a swing of 7 points or more toward Ford (i.e. swung more toward Ford than average), and therefore account for his swing.  55 of them actually swung toward Corker.  The counties that swung toward Ford were on average 19% black, the counties that swung toward Corker were on average 7% black.  

In other words, Tennessee actually displayed highly racialized voting patterns, and the white undecideds behaved exactly as we would assume they would behave with a Bradley effect.  And this is against the backdrop of an aggressive outreach by Ford for conservative white voters.

Elephant Effect?

Tennessee certainly doesn’t look like an “elephant effect,” given that much of the state actually swung toward Republicans when comparing the second-best Republican year in almost twenty-five years to the worst Republican year in the same cycle. 

But what about Maryland and Ohio, where Steele and Blackwell, both Republicans underperformed the polls?   

In Maryland, Steele underperformed the polls by six points.  In Ohio, Blackwell underperformed by four points.  At the same time Ehrlich only underperformed by five points.  DeWine only underperformed by two points.  So while white Republicans in these states also underperformed their polls (and this was by no means uniform nationwide, as Senators Allen and Burns overperformed in two of the most hotly-contested races, and House Republicans outdid the generic congressional ballot considerably), they did so slightly less than did their black counterparts. 

It is also worth noting that Ehrlich and DeWine were both incumbents, so we would expect to see undecideds break against them to a larger degree than we saw for Blackwell or Steele.

Regardless, given that the white vote -- which is what we're talking about in terms of Bradley effect -- also broke against Ford and Patrick (more in a second), I think we're talking about more than just an "Elephant effect."

Other Races

 Finally, let’s look to see how the white vote behaved over time in races where SurveyUSA provided us with crosstabs.  First, let’s look at Healey-Patrick:

 9/19-9/21:  Healey 26%, Patrick 63%

10/8-10/10:  Healey 35%, Patrick 50%

10/21-10/23:  Healey 32%, Patrick 55%

10/3-11/1:  Healey 35%, Patrick 53%

Final CNN Exit Poll:  Healey 39%, Patrick 51%

Next, Ohio Gov:

9/18-9/20:  Strickland 55%, Blackwell 36%

10/23 -10/25:  Strickland 63%, Blackwell 32%

11/3 – 11/5: Strickland 55%, Blackwell 38%

Final CNN Exit Poll: Strickland 58%, Blackwell 40%

And finally, MD Senate: 

9/17-9/19:  Steele 54%, Cardin 41%

10/15-10/17:  Steele 53%, Cardin 41%

11/3-11/5:  Steele 53%, Cardin 44%

Final CNN Exit Poll:  Steele 50%, Cardin 48% 

In three of the four races, we see a break of the white vote toward the white candidate.  In three of the four races, the white candidate ends up doing better with white voters than he had done in every SurveyUSA poll taken in the race, while the black candidate gets at or below his best polling among whites.  The one exception – the 10/23-25 Ohio Gov SUSA poll, showed Strickland with his second- largest overall lead of any poll taken in the cycle, and is about ten points ahead of what pollsters in the field at about the same time were showing.  In other words, that poll may be something of any outlier. 

Maybe this is a problem with SurveyUSA’s methodology, but they’re generally one of the better pollsters out there.  Maybe this is attributable to sampling error, although sampling errors should cancel themselves out; they shouldn’t consistently appear on one side or another of the equation.

 In the end, it is impossible to say whether what we see is attributable to the Bradley effect or is a purely localized phenomenon.  But it is at least evidence of the Bradley effect's existence in 2006.