November 5, 2009
By Mark Blumenthal
I had intended to post a "quick" summary of what Tuesday night's results say about how the polls did, but like a thread pulled on a sweater, my outline kept getting longer. So apologies for the delay in getting this summary posted. What follows is a review of how the polls performed this year, with a closer look at the question posed yesterday by our own Brian Schaffner, was it "a victory for IVR polling?"
New Jersey. Our final trend estimate based on all pre-election polls was dead even, with each major party candidate receiving 42.0% of the vote and independent Chris Christie 10.1%. Christie had a one-point lead on the RealClearPolitics average of the last five non-partisan polls (+1.0%), roughly the same margin as using our more "sensitive" trend line (+1.1%).
The unofficial count, as of this writing, has Christie leading by 4.3% (though as noted yesterday, all of these unofficial results are likely to change slightly as provisional and absentee ballots are counted). So the average polling error in New Jersey was between 3.3% and 4.3% depending on the average. Nate Silver did a compilation of comparable New Jersey polling errors (compared to final averages) on 9 previous elections that ranged from a low of 0.5 to a high of 4.8. So the error yesterday, while higher than average, fell well within recent experience.
At the same time, nearly everyone has noticed that the average of the final polls from three organizations using an automated methodology (sometimes refered to as "interactive voice response" or IVR) had Christie ahead by four percentage points (46% to 42%) -- roughly the same as his unofficial margin -- while the last three live-interviewer telephone polls had Corzine leading by an average of one point (41% to 40%)
As I wrote on Monday night, what makes that gap between automated and live-interviewer polling interesting is that it was not some random fluke on the last few polls, but persisted throughout the campaign to a degree that we did not see in Virginia this year or in most states during the 2008 presidential election. My conclusion was that the consistency in the estimate of Corzine's vote on so many recent polls suggested a looming "incumbent effect," that voters had largely made up their mind on Corzine but that a small but critically important number were still weighing whether to support Christie or Daggett. So, the theory goes, the IVR polls did better by removing the live interviewer and simulating a secret ballot, thus pushing voters harder to make a choice and more accurately recording their true intentions over the phone.
And what happened to Daggett? Our final trend estimate had him at 10%, but he received only 5.8% of the vote. Although it had been rising until mid-October, Daggett's support ultimately followed the traditional pattern. Many voters that had been intrigued by his candidacy ultimately concluded that their votes would be wasted and opted to support either Christie or Corzine. The Fairleigh Dickinson Unversity poll provided a hint of where Daggett's support was heading in an experiment conducted on their last survey: They found that Daggett received just 6% -- the same number he won on election day -- when they only named Corzine and Christie as candidates but accepted Daggett as a volunteered choice. When they offered a three-way-choice that included Daggett, his support jumped to 14%.
Virginia. Republican Bob McDonnell's victory in Virginia was never in doubt during the final weeks of the campaign, so political junkies were less obsessed with the polling numbers, but the polling errors in Virginia were, on average, about the same as in New Jersey. Our final trend estimate had McDonnell ahead by 13.4% (54.7% to 41.0%). The unofficial tally has McDonnell leading by 17.4% (58.7% to 41.3%) so the error, as of this writing, averages 3.7 points on the margin.
In Virginia, the gap between the results of automated and live interviewer polls was not nearly as big or as consistent as in New Jersey. The average of the final automated polls in Virginia conducted by PPP, SurveyUSA and Rasmussen had McDonnell at 56% compared to 54% on the final polls in the last week conducted by five organizations using live interviewers, while both sets of poll gave Democrat Creigh Deeds an average of 41% of the vote. However, the final automated polls by SurveyUSA and PPP along with the live interviewer survey by Virginia Commonwealth University are closest to the final margin (as of this writing).
New York City. Our final trend estimate had Mayor Michael Bloomberg leading Democratic challenger William Thompson by a 14-point margin, (53.1% to 39.0%), but Bloomberg won by less than five (50.6% to 46.0%) so the polling error is large (9 points on the margin) -- roughly the same as the infamous New Hampshire polling debacle).
What happened? Marist pollster Lee Miringoff describes it as a "text book case of pre-election poll analysis:"
It is not unusual in contests between a well-known incumbent (Bloomberg) and a relatively unknown challenger (Thompson) that the incumbent ends up getting pretty much the same number he was attracting in pre-election polls. Undecided voters tend to find the challenger or not vote at all, having already rejected the incumbent.
He refers, of course, to the "incumbent rule," a subject I speculated about at length in 2004, only to see it generally not apply that year, in close races in 2006 or 2008. That said, it does appear to have returned in New Jersey and New York City on Tuesday.
But that apparent reemergence raises an important question: If the rule is no longer a "rule," but rather a phenomenon that occurs only occasionally, how do we know to expect it? Miringoff wrote yesterday that Marist's polls "showed the trend that Democratic voters were 'coming home' to Thompson." That result would have been a helpful warning sign. Problem is, I can't find any reference to it in Marist's final poll release. Instead, I find this prediction: "If today were Election Day," they wrote on Wednesday without qualification, "Mayor Michael Bloomberg would handily win a third term."
If anyone deserves to say "I told you so" in New York, it is Thompson pollster Geoff Garin, who released a survey last week showing Thompson gaining (he said), trailing by only 8 points (38% to 46%) and by only 3 points (41% to 44%) among those who said they were certain to vote. The release prompted Bloomberg spokesman Howard Wolfson to retort that it "gives new meaning to the term margin of error." Not exactly. (And yes, we managed to miss this poll and omit it from our chart -- apologies to Garin and our readers for that oversight).
I asked Garin for his thoughts and he agrees that "undecideds split against incumbent" in the New York race and that such a split was knowable in advance, but argues:
[I]t is stupid to think they would split 100 to nothing. There was a high undecided in NYC because voters were cross pressured -- they did not want to reward Bloomberg for his bad behavior on term limits, but they didn't know enough about Thompson to know whether he would be up to the job.
Garin also thinks their sample made a difference:
I think the main reason we did better and the public polls were off is that we worked off the voter file, and were persnickity about who we took into what was very likely to be a low turnout election. Even among whites, the smaller the turnout scenario the better for Thompson. I am sure the public polls let in too many people.
Maine Question 1. Polling on the gay-marriage referendum was far more limited -- just seven public polls released over the course of the campaign -- and the complicated ballot language and the error prone nature of prior referenda poll warned us to expect the unexpected. Yet while the differences between the final polls were relatively small, it is worth noting that the automated survey from PPP was the only one that showed more support for the anti-gay marriage position than opposition. Our final trend estimate showed the No side (pro gay marriage) with a two point lead (49.4% to 47.1%) but Question 1 won by nearly six (52.8% to 47.2%).
While this one experience is far from a conclusive test, there are at least theoretical reasons to think that automated surveys have an advantage in measuring true preferences on issues like gay marriage, where the presence of a live interviewer might introduce some "social discomfort" that would make the respondent reluctant to reveal their true preference.
* * *
So were automated IVR polls the big winners on Tuesday, as Mickey Kaus, Taegan Goddard and PPP's Tom Jensen argue? If what you care about most is predicting the winners, it is clear that the automated surveys provided a more accurate gauge of the outcome, especially in New Jersey where the closer simulation of the secret ballot probably gave us a heads up of an imminent "incumbent rule" effect favoring Christie. SurveyUSA also deserves credit for coming closer than most pollsters to the final margin in New Jersey, Virginia and New York City.
But that said, consider that we count on polls to do much more than predict the outcome. In addition to the points raised by Brian Schaffner here yesterday, consider two things:
First, as a live-interviewer media pollster pointed out to me yesterday, there were some inconsistencies with subgroups, particularly by race. As the table below shows, despite relatively small sample sizes, the three automated surveys showed Republicans Christie and McDonnell winning a greater percentage of the African American vote than the final live-interviewer surveys and the exit polls (though there were a few inconsistencies; namely Rasmussen in New Jersey and Marist in New York City).
If you believe the exit poll result, then the automated surveys provided a generally misleading sense of whether the Republican candidates were about to make bigger inroads than they did among African-American voters (consider also commenter RussTC3's observation about big differences between job approval ratings as measured by PPP and the exit polls -- as Mike Mokrzycki reminds us we do polls for reasons other than predicting the outcome).
Second, there is one last contest we need to review....
New York 23. Although three last minute polls on the special election in New York's 23rd Congressional District conducted after Republican Dede Scozzafava withdrew from the race last Saturday showed Conservative Doug Hoffman leading Democrat Bill Owens by margins of between 5 and 17 points, Owens prevailed by 4 points (49.0% to 45.9%). Whatever shortcomings we might identify in the polling, the far bigger error was the interpretation applied by pundits, most notably me, who foolishly assumed that the trend in Hoffman's direction was unstoppable and that normal assumptions about last minute developments would apply. In retrospect, it is obvious that there was nothing normal about the last 72 hours of this particular campaign.
Moreover, we should have paid closer attention to the evidence of growing voter uncertainty in the final Siena Research Institute poll. Their final survey, conducted on Sunday night, showed Hoffman with modest but not quite statistically significant lead (41% to 36%) but also a doubling of the undecided (from 9% to 18%) in just a few days. So their poll showed that voter uncertainty was surging at a time when it is usually nonexistent. To his great credit, Siena pollster Steven Greenberg also argued that Owens might still gain from the Scozzafava endorsement on Sunday since "most voters are not political junkies" and had not yet heard the news" (an argument I boldly dismissed since few undecided voters had a favorable impression of Scozzafava -- apologies to Greenberg for that).
But while we might plausibly reconcile the results of the Siena poll with the outcome, the PPP survey is another story. While their estimate of Owens' support (34%) was within a few points of the other polls, PPP had Hoffman receiving five percentage points more support (51%) than he ultimately received (45.9%). A late shift among the undecided voters cannot explain the difference.
I am planning to look more closely at this example, but the important point for now is that while the automated polls turned in a strong performance in New Jersey, Virginia and Maine, the PPP poll in NY-23 was highly misleading.
The larger lesson is this: Automated polls have been maligned, unfairly in my view, as inherently "unreliable." Yet when it comes to predicting election outcomes they continue to prove, NY-23 aside, at least as reliable as surveys done by conventional means. In New Jersey this week, they were more accurate in predicting the winner. At the same time, however, it would be wrong to jump to the opposite conclusion and place inherently greater trust in all automated surveys, especially when used for purposes other than predicting election outcomes.
All polls have their limitations. Rather than trying to divide them into two categories, "reliable" and "crap," we might do better to try to understand their limitations and interpret the results we see accordingly.
By Mark Blumenthal on November 5, 2009 5:23 PM
| Permalink
| Comments (6)
| TrackBacks (0)
November 4, 2009
By Mark Blumenthal
For whatever reason, the usual network web sites did not post full tabulations of the exit poll results from last night or at least did not post them in an easily discoverable place. However, for those interested -- and our readers always are -- the New York Times did post full tabulations for the exit polls conducted by Edison Research in New Jersey, Virginia and New York City.
Also, the following analysts and organizations have posted in-depth analyses of the exit poll results:
[I'm confident this list is missing similar analysis from other organizations, so please add a comment or email us if you see articles worth linking to].
By Mark Blumenthal on November 4, 2009 1:37 PM
| Permalink
| Comments (0)
| TrackBacks (0)
By Mark Blumenthal
"So who was the most accurate pollster yesterday?"
If I had $100 for every time I've been asked that question by a reporter on the Wednesday morning after an election, I could retire early. And after five years of blogging on this beat, it's a question I'm determined to refuse to answer today.
Why?
First, all the votes are not yet counted (including 7% of the precincts in NY-23), and the counts that are available do not yet include the absentee and provisional ballots that will be added later and are not reflected in those percentage-of-precincts-reporting statistics you see on all the media vote counts morning. Take a look at this snap judgement from November 5, 2008. It declared a "big winner" among prognosticators on the assumption that Barack Obama won by 6.1 percentage points (52.4 to 46.3), but when all the ballots were counted the margin was 7.2 (52.9% to 45.7%). So that particular snap judgement picked the wrong "big winner."
Second, the whole notion of crowing a "big winner" based on a handful of polls in a handful of states is foolish. The final polls yesterday had random sampling error of at least +/- 3 percentage points. If a poll produces a forecast outside its margin of error, that's important. But if several polls capture the actual result within their standard error, chance alone is as likely as anything else to determine which one "nails it" and which miss by a point or two.
Third, there are sometimes other problems with making too much of "hitting a bullet with a bullet" on the final poll, when the polls leading up to it provide different results.
Yes, there are several good stories about what went right and what went wrong with yesterday's polling, including some important lessons about the value of automated polling. Some pollsters certainly did better yesterday than others. And I'm hoping to have something written and posted on that subject later today, provided that I don't get bogged down by the calls and emails from reporters wanting me to tell them, "who was the most accurate pollster yesterday?
By Mark Blumenthal on November 4, 2009 11:33 AM
| Permalink
| Comments (2)
| TrackBacks (0)
By Mark Blumenthal
Since 2002 (and probably earlier), you could do pretty well in predicting the outcomes of races for President, Senate, Governor and even the U.S. House by collecting the final polls in each race and averaging them. In fact, in 2008, the final Pollster.com trend estimates and RealClearPolitics averages did as well or better at calling election outcomes as those more "sophisticated" models you heard so much more about last year.
The reason is that while highly variable, the final polls were largely unbiased in the aggregate. Any one poll might be way off from the final result, but the average of all of them usually comes reasonably close to the final result. There have certainly been exceptions in individual states, but in 2002, 2004, 2006 and 2008, the polls looked reasonably accurate once averaged across all states.
We may perceive things differently tonight. First, instead of watching polling across 20 or 30 contests, most of us are focused on just three or four races and for two of these -- New York's 23rd District and the Maine Question 1 -- we have only one or two recent polls to consider.
Second, as Nate Silver pointed out yesterday, the challenges in some of today's elections -- again, especially Maine and NY-23 -- may be more like what pollsters faced during last year's presidential primaries, where poll averages often missed the mark by wide margins.
Silver also posted a handy comparison of final poll averages in New Jersey elections since 2000 (below), which helps make two important points. First, as he writes, despite conventional wisdom to the contrary there has not been "any particular tendency by Democrats to outperform their numbers once the final polls are in." Second, though usually very close to the result, final poll averages in individual states typically missed the final margin by a few percentage points. So even though our final New Jersey trend estimate is a remarkable 42.0% to 42.0% tie, for example, the final margin will be close but probably not that close.
Which brings me to our final polling-wrap up for 2009. Here's what the final polls and our trend estimates are showing:
- New Jersey, again, ends up as 42.0% to 42.0% tie on our trend estimate, a contest simply too close to call between Democratic incumbent Jon Corzine and Republican challenger Chris Christie, with independent Chris Daggett running far behind at 10.1% and likely falling. My hunch, explained last night, is that Christie prevails.
- New York's 23rd District special election for Congress was the focus of much speculation over the weekend. The last two polls, both conducted immediately after original nominee Dede Scozzafava withdrew from the race, each had Conservative Doug Hoffman leading Democrat Bill Owens, but by widely different margins (5 and 17 percentage points). Our trend estimate, which has Hoffman leading by 7 points (43% to 36%) also factors in previous polling that showed a closer contest. The ultimate margin is anyone's guess, but my sense is that Hoffman will win comfortably.
- The outcome of Virginia's race for Governor has never been in much doubt. Republican Bob McDonnell began with a roughly 7-point lead over Democrat Creigh Deeds that never significantly wavered, widening to nine points by Labor Day and ending at our final trend estimate of roughly 14 points (55% to 41%).
- Ditto for the New York City Mayor's Race. Incumbent Michael Bloomberg led Democratic challenger William Thompson consistently on polling throughout the race, although his lead on our final trend estimate (53% to 39%) is sightly narrower than earlier in the year.
- And the very few polls on Maine's Question 1, the gay marriage referendum, show a close contest, although as I wrote earlier this afternoon, referenda polling is notoriously error prone. The age composition of the PPP survey seems closer to plausible than the final DailyKos/Research 2000 survey, but beyond that, your guess is probably as good as mine.
Two notes on what's coming up later tonight. First, the consortium of network news organizations (known formally as the National Election Pool or NEP) is conducting exit polls in New Jersey and Virginia tonight. While official results will not begin to appear until the polls close, some early leaked estimates will probably start to bounce around the internet sometime after 6:00 p.m. As I explained at about this time last year (on on most election days since 2004), these are not likely to be much more accurate than the pre-election polls summarized above. Very large grains of salt are in order.
And finally, we will be live blogging here once again tonight. If all goes well, we should be using a more advanced tool that will allow our all-star line-up of contributors (Charles Franklin, Kristen Soltis, Margie Omero, Steve Lombardo and hopefully more) to join in. We hope you'll join us starting at about 6:30 eastern time.
By Mark Blumenthal on November 3, 2009 2:58 PM
| Permalink
| Comments (3)
| TrackBacks (0)
By Mark Blumenthal
I have to admit that I had been hoping to take a closer look at polling on Maine's Question 1 on Gay Marriage over the weekend, but got distracted by the fuss over the New York 23rd District special election. The polling is difficult to evaluate partly because there has been so little of it. While I have a lot of confidence in our trend estimates in states with large numbers of polls, the small number of polls in Maine (7 total since Labor Dayl) allow for just crude linear trend lines.
Another reason why the Maine polls are difficult to evaluate is that issue referenda polling is so treacherous and prone to error. A 2004 paper by Joe Shipman, then director of election polling for SurveyUSA, showed that polling on ballot measures had triple the rate of error (9.5 average error on the margin) as polls in presidential elections (3.4) and nearly double that of contests for statewide offices (4.6). I summarized the assumed reasons for that greater error rate in a long post four years ago today, but the most relevant to Maine are a greater difficulty modeling the likely electorate and the problem of accurately conveying ballot language.
A particularly painful example followed a few days after that post, when a set of ballot initiatives in Ohio produced some of the biggest polling errors in recent memory. The combination of failing to poll late and not accurately reproducing the actual ballot language were likely culprits.
Let's start with the ballot language in Maine that voters are confronting right now:
Do you want to reject the new law that lets same-sex couples marry and allows individuals and religious groups to refuse to perform these marriages?
So, a "Yes" vote is a vote against gay marriage, and a "No" vote is for gay marriage. Confused? Imagine the uncertainty some Maine voters may be experiencing without that extra bit of explanation. As you can see in the table at the bottom of our chart, only the Pan Atlantic SMS surveys reproduce the actual ballot language -- and nothing else -- while the other pollsters provide a line of explanation to clarify the meaning of "Yes" and "No."
The final round of polling has shown a relatively close race, although results have varied. A survey conducted two weeks ago by Pan Atlantic SMS, shows the No side prevailing by an 11-point margin (53% to 42%), while a Daily Kos/Research 2000 poll conducted late last week shows a dead-heat (No 48%, Yes, 47%) Finally, the Democratic automated polling firm PPP conducted a survey over the weekend that had the Yes side ahead by a not-quite-statistically significant four points (51% to 47%) despite a very large (n=1,133) sample.
Complicating the issue further is that the final poll from PPP differed in both the age of the "likely voters" they selected and the way they interviewed (via an automated, recorded voice methodology). Less than a third of PPP's likely voters (32%) were under age 45, compared to more than half (51%) of the Research 2000/DailyKos survey. Both showed much more support for the No side from younger voters.
On the question of age, the Research 2000 sample was even younger than the Maine exit poll in November 2008 (43% age 18-45) and far younger than in November 2006 (36% age 18-45). Of course, the PPP sample was older than both, but keep in mind that exit poll estimates are sometimes too young.
The question of the automated mode is more complicated. The automated polls conducted by SurveyUSA in California 2008 may have picked up more support for ultimately successful anti-gay marriage Proposition 8 than on live interviewer surveys conducted at the same time. However, the convoluted nature of the timing of the various polls and the final result from SurveyUSA (showing Prop 8 narrowly failing) make it impossible to draw firm conclusions.
So what conclusions can we reach about tonight's outcome in Maine? I have more faith in the age composition of the PPP poll than the one from Research 2000, but given the much larger potential for error in ballot referenda and the close margins on the two final polls, your guess is probably as good as mine.
By Mark Blumenthal on November 3, 2009 1:58 PM
| Permalink
| Comments (0)
| TrackBacks (0)
November 2, 2009
By Mark Blumenthal
With all but perhaps or or two final polls logged, our trend estimate in New Jersey, as of this writing, stands at a 42.0% to 42.0% deadlock between Corzine and Christie, with Chris Daggett falling to 10.1%. That amazingly close result will likely change if we add another poll or two tomorrow, but a shift of a half point or so in either direction will have little meaning. The polling on this race is as close as it every gets, and as our standard trend line (below) shows, has been for the last few weeks.
The bottom line is that our final estimate is too close, in and of itself, to forecast winner. As I noted last Friday, our final estimates for the 2008 election included four states with final Obama-McCain margins that rounded to a percentage point or less. The nominal leader won in two of these states (North Carolina and North Dakota) but lost in two others (Missouri and Indiana).
But wait. Does our standard trend line ignore a last minute trend to Christie, analogous to the presumed movement to Hillary Clinton in the New Hampshire primary? If you use our chart's Smoothing Tool to change to the "more sensitive" setting (as illustrated below), you will see a hint of a trend toward Christie. The Republican challenger's support ticks up slightly (to 42.5% as of this writing), while Corzine's line moves slightly (to 41.4%).
Intriguing as it seems, given the mix of different methodologies and field periods on our chart, we cannot be sure that the twitch in the more sensitive line represents a real change and not just random noise. The slight move to Christie is because three of the five surveys released today show nominal movement to Christie, while only one shows a nominal shift to Corzine and one shows no change in the margin. I'm not certain of the odds calculation on that outcome, but the probability that it occurred by chance alone is far more than the usual 5% we usually require to say it is statistically significant (using different calculations, Monmouth University pollster Patrick Murray reaches a similar conclusion).
Even if real, the 1.1 point margin favoring Christie on the more sensitive trend estimate is still not large enough to characterize the race as "leaning" Christie's way. So, for better or worse, if you are looking for a purely objective, empirical "call" of the New Jersey race, our trend estimates are not much help. The finals snapshot is just too close.
If, on the other hand, you are interested in some purely subjective speculation -- and that's what all of the various predictions of the outcome amount to -- read on.
Let's start with a review of the biggest New Jersey polling puzzle, the consistent difference in results between automated and live interviewer surveys, and then consider what it may imply about two things that polls are least able to measure consistently and accurately: Who will turn out to vote and what voters really mean when they say they are undecided.
The automated-vs-live-interviewer puzzle. As reviewed here on Friday, the three pollsters that use an automated, recorded voice methodology -- SurveyUSA, Rasmussen Reports and PPP -- have produced results consistently better for Christie and worse for Corzine than the other live-interviewer telephone surveys.
As the chart above shows, that difference persists through the final round of surveys released today and late last week. On their final polls PPP, SurveyUSA and Rasmussen have Christie leading by an average of 4 points (46% go 42%) while the three live-interviewer surveys released earlier today by Quinnipiac University, Monmouth University and Democracy Corps had Corzine leading by an average of one point (41% to 40%).
Modeling turnout: Probably the most important result in today's data comes from a single table buried in the cross-tabulations of the poll released by the Democratic-affilliated Democracy Corps. Their poll puts Corzine ahead by the largest margin 41% to 36%. The cross-tab shows that virtually all of Corzine's lead on the Democracy Corps poll comes from voters who did not cast a ballot in the 2005 election (and since Democracy Corps samples from a voter list, this classification is based on actual vote history, not a self-report):
Among those who voted in 2005 (84% of the Democracy Corps sample), Corzine leads by only a single percentage point (39% to 38%) in the Democracy Corps poll. Among those who have voted in other elections but not 2005 (and every respondent to this poll self-reported having voted in 2008), Corzine leads by more than two-to-one (54% to 26%). Thus the much discussed "all out push" by the Corzine campaign to win over Obama backers.
Corzine backers were cheered today by tabulations on two of the new polls showing the Governor with a significant lead among early voters. SurveyUSA's new poll found 14% had already voted and Corzine led among these voters by 12 points (50% to 38%). Monmouth found half as many early voters (6%) but an even wider Corzine lead among them (51% to 31%). Tantalizing as it is, we will not know until this time tomorrow whether the Corzine campaign is truly mobilizing the new Obama voters from 2008 or whether they are simply getting a lot of hard core Democrats that would have voted anyway to cast their ballots early. If Corzine wins, it will surely be because of this organizations advantage.
Last week, Nate Silver speculated better Corzine performance on automated polls might be due to an effectively "tighter screen" on those surveys:
An automated poll tends to be associated with lower response rates, since an automated script can't do as much a human to coax someone into an interview, and therefore sometimes tends to reach a more enthusiastic set of respondents (in effect, it may serve some of the same functions as a very tight likely voter screen).
Since Republicans tend to be more enthusiastic right now, that may be what's causing the automated polls to be more favorable to them. But since none of us yet know how the enthusiasm gap is going to play out in practice, it would be premature to come to any conclusion about whether the voter universe that Rasmussen and PPP are coming up with is "too tight" or "just right.
Perhaps, although as I noted on Friday, the automated-vs-live-inteviewer gap is significantly smaller in Virginia. On the final round of polls, the lead by Republican Bob McDonnell is only two points greater on the automated polls (56% to 41%) than the last five live interviewer polls.
Either way, my sense is that overall turnout will be driven less by the respective campaign field organizations than by the underlying enthusiasm gap driving voter decisions. It is one thing to help an already enthusiastic voter cast an early ballot. It is something else to convince a complacent voter to get excited about a candidate for whom they have mixed feelings.
Measuring those who are undecided. If you spend time with horse race polling numbers, it doesn't take long to discover that some of the biggest differences among pollsters involve the undecided percentage. One reason is that voters do not fall neatly into "decided" and "undecided" categories. Vote decisions fall along a continuum from completely committed to totally undecided, with most voters falling somewhere in between. The size of the undecided category on a poll may depend on the wording or structure of the question or how hard interviewers pushes for a decision. Automated surveys frequently obtain a smaller undecided percentage, and one reason may be that voters feel less comfortable revealing their "secret ballot" choice with a live interviewer.
Complicating this issue further is that saying "I'm undecided" on a survey may imply something other than total indecision. In a three-way race, it may mean that the voter has decided against voting for one candidate, but has not settled on which of the two alternatives deserves their choice (see some evidence of this sort of uncertainty in the focus group conducted by Monmouth University pollster Patrick Murray).
The theory behind the incumbent rule (that I spent a lot of time speculating about in 2004) is that the undecided category swells with voters that have decided against supporting the incumbent but are not yet ready to embrace the challenger.
Over the last eight to ten years, it has been hard to find much evidence of an automatic "break" of undecided voters toward challengers especially in highly competitive races, probably because incumbents have become so much more adept at turning the tables and "going negative" on their opponents.
However, if we take a closer look at the automated-vs-live-interviewer puzzle in New Jersey, we see a pattern that a few years ago I would have treated as clear evidence of the incumbent rule in action. All of the surveys, regardless of their methods, are yielding consistent results for Corzine -- most have him within a point of 42%. But the final automated surveys show a much smaller undecided vote and a consistently higher percentage for Christie (45-47%) than the live interviewer surveys (36-42%). (And NRO's Jim Gerhaghty notices that this pattern extends back to far more surveys than those in the table below).
So what does all this tell us about the too-close-to-call final estimate we are showing for New Jersey? This now nearly three-year-old comment from Republican pollster Neil Newhouse sums up my feeling:
[N]ewhouse noted the example of his client, incumbent Republican Jim Gerlach (Pennsylvania-6), who was in a 44% to 44% tie on their final internal poll conducted a week before the [2006] election. In the "old days," Newhouse said, we would have assumed an easy Murphy victory. However, Gerlach ultimately prevailed (51% to 49%) after a closing with a final television ad featuring a personal appeal by Gerlach that Newhouse credited for the victory. As for the incumbent rule, Newhouse said, "we are seeing a bit of a change, but not much consistency." While he still tends to give challengers the "benefit of the doubt" when incumbents are under 50%, Newhouse believes it is no longer "carte blanche automatic" that the undecided vote on the final poll will all go to the challenger.
All other things being equal -- and 42.0% to 42.0% is about as equal as they get -- I still tend to give a challenger like Christie the "benefit of the doubt" when up against an incumbent like Corzine even though recent examples of the "incumbent rule" are few and far between. That instinct is reinforced by the large number of voters that are either undecided or still leaning to independent Daggett (with Daggett's support falling) and the fact that no matter how hard pollsters appear to push, Corzine does not seem to rise beyond 42%.
So while the empirical evidence says this race is still too close to call, my hunch is that Christie will emerge the narrow victor.
By Mark Blumenthal on November 2, 2009 11:30 PM
| Permalink
| Comments (3)
| TrackBacks (0)
By Mark Blumenthal
Since I typically file a column on Friday that runs on Monday, and since NationalJournal.com is focused on the theme of energy policy this week, today's topic is the challenge of polling on a little known public policy issue like Cap and Trade. Hope you click through.
By Mark Blumenthal on November 2, 2009 8:59 AM
| Permalink
| Comments (0)
| TrackBacks (0)
By Mark Blumenthal
We got two new pieces of polling news last night from New York's 23rd District. The first is the one and only survey from Public Policy Polling (PPP), a firm that does automated surveys for Democratic candidates but also conducts and releases surveys in high profile races like NY-23 as a marketing tool. PPP's poll, also the first conducted since Republican nominee Dede Scozzafava withdrew on Saturday morning, forecasts a different outcome than previous surveys, including the Siena Research Institute poll conducted last week: They show Conservative candidate Doug Hoffman with a huge lead over Democrat Bill Owens -- 17 points (51% to 34%) on a three-way matchup that includes Scozzafava, whose name will remain on the ballot), 16 points on a question that only asked about Hoffman and Owens (54% to 38%).
The second piece of news was a release from the Siena Research Institute announcing that they will release another new poll this morning a little after 9:30 a.m. Eastern time.
The new PPP survey suggests a significant acceleration of the trend evident in other polling including Siena's -- a collapse of Scozzafava's support while Hoffman's vote soars and Owens creeps up slowly. Strictly speaking, Scozzavfava's withdrawal and subsequent endorsement of Owens over the weekend render previous horse race results virtually useless as predictors of the outcome. We are probably best advised to throw out the previous polls (and the trend estimate based on them in the chart above) and simply examine the two post-withdrawal poll we'll have available later this morning.
If the Siena results are consistent with PPP, this discussion will be mostly academic. The Siena and PPP surveys use very methodologies, and given Charles Franklin's analysis of the last Siena survey on Saturday, I would not be surprised to see their update produce a closer result than PPP. I will update this post accordingly when we have those results.
UPDATE: Surprise, surprise, the Siena results show Hoffman narrowly ahead of Owens, but by a smaller, five-point margin (41% to 36%) than PPP, with Scozzafava getting only 6% of the vote and 18% undecided. With 606 interviews, that margin is not quite statistically significant given the usual 95% confidence interval.
Given that Siena uses a classic random digit dial (RDD) sample and live interviewers, while PPP uses a voter list sample and an automated recorded-voice method, some are going to want to ignore the PPP results and focus on the large number of undecided voters (18%) in the Siena survey. Even Siena pollster Steven Greenberg is arguing that Hoffman might not have the advantage going into tomorrow's election since "most voters are not political junkies like I am and didn't know" as of yesterday, that Scozzafava had endorsed Owens.
Apologies to my Democratic friends for the pessimism, but I don't see it. First of all, even if we focus only on the Siena survey, the crosstabs offer little hope of a decisive rebound among those undecided as of last night. Scozzafava's rating among the undecided is 28% favorable, 22% unfavorable, while half (50%) have no opinion. In case it's not obvious: If a voter doesn't like Scozzafava by now, there's not much chance her endorsement of Owens will mean much to them.
Keep in mind that before asking who they would support, the Siena question informed every voter that while Scozzafava's name would remain on the ballot she has "suspended her campaign" and "released those individuals supporting her campaign to transfer their support as they see fit." As such, nearly two-thirds (65%) of those still planning to vote for her say they are "absolutely certain" about their choice with "no chance I will change my mind."
Among the undecided, the ratings of Owens and Hoffman are similar, although Hoffman's negatives are slightly higher (by a margin that is far from statistically significant):
- Owens - 26% favorable, 20% unfavorable, 54% don't know
- Hoffman - 24% favorable, 25 unfavorable, 51% don't know
The overriding message from the big "don't know" numbers among the undecided is that most are not likely to vote. None of these numbers suggests a late, decisive break toward Owens.
There's also the matter of the trend. Scozzafava's withdrawal accelerated the trend to Hoffman's that was already quite strong. On the Siena surveys, for example, Hoffman's vote has increased from 16% to 23% to 35% to 41% on four surveys conducted since October 1. Count me as skeptical that the six-point bump in Hoffman's support seen in last night's poll fully captured the benefit to Hoffman from Scozzafava's departure.
And then, finally, we come back to the PPP poll and its bigger Hoffman margin. Rather than go on at length (again) about the differences between random digit dial (RDD) samples and voter lists and about the trade-offs between live interviewers and an automated method, let's consider it this way: Both are blunt instruments for sampling adults, selecting "likely voters" and measuring their preferences. Neither can be considered a gold-standard, a true random sample that perfectly covers, models or represents those who will vote tomorrow.
However, my experience conducting surveys for political campaigns, especially in Congressional districts in non-presidential year races, taught me the value of the vote history available on registered voter lists. More often than not, surveys I helped conduct based on such lists came closer representing the true likely electorate than media RDD samples which, like the Siena survey, disclose little to nothing about their likely voter screen or demographic composition.
Add to that the potential advantages of a self-administered automated survey in getting voters to provide more honest answers about whether they plan to vote and who they plan to vote for, and I find it difficult to ignore the PPP results. Hoffman looks like he's headed to a comfortable victory.
By Mark Blumenthal on November 2, 2009 7:34 AM
| Permalink
| Comments (2)
| TrackBacks (0)
November 1, 2009
By Mark Blumenthal
Public Policy Polling (D)
10/31-11/1/09; 994 likely voters, 3.1% margin of error
Mode: Automated phone
(PPP release)
New Jersey
2009 Governor
Christie 47%, Corzine 41%, Daggett 11% (chart)
Is your second choice for Governor Chris Christie or Jon Corzine? (Asked only of Daggett voters)
Corzine 45%, Christie 36%
Favorable / Unfavorable
Chris Christie (R): 43 / 42
Jon Corzine (D): 34 / 58 (chart)
Chris Daggett (i): 24 / 35
Job approval / Disapproval
Pres. Obama: 45 / 45 (chart)
By Mark Blumenthal on November 1, 2009 11:42 PM
| Permalink
| Comments (14)
| TrackBacks (0)
By Mark Blumenthal
Public Policy Polling (D)
10/30-11/01/09; 1,747 likely voters, +/- 2.3% margin of error
Mode: Automated phone
(PPP blog entry and release)
New York 23rd Congressional District
2009 House: Special Election
51% Hoffman, 34% Owens, 13% Scozzafava (chart)
54% Hoffman, 38% Owens (on 2-way choice question)
Favorable / Unfavorable
Bill Owens (D): 35 / 45
Doug Hoffman (C): 50 / 37
Dede Scozzafava (R): 31 / 52
By Mark Blumenthal on November 1, 2009 11:02 PM
| Permalink
| Comments (18)
| TrackBacks (0)
By Mark Blumenthal
Jon Lerner shares his take on Scozzafava's departure from his CFG sponsored poll.
Micah Sifry proposes a way for Chris Daggett to overcome fear of wasted voting; 55,000+ (as of this hour) take the Daggett Pledge.
Jennifer Agiesta says Deeds is underperforming outside NoVA.
Tom Jensen says even Obama ran better in rural VA; finds Corzine will need 17% from NJ voters who don't like him.
Anthony Salvanto and Mark Gersh drill down on NY-23.
Charlie Cook sees no bellwethers in the 2009 races.
Stuart Rothenberg sums up the outlook for 2009.
Chris Bowers unveils his 2009 election forecast and debuts his National House Ballot monitor.
Jim Geraghty ponders the undecideds in New Jersey.
Ben Smith chronicles the role of pollster Joel Benenson in the Corzine campaign.
Sarah Dutton breaks down Obama's approval slide.
Gallup presents five key realities of public opinion and health care reform.
Frank Luntz releases another health care strategy memo (via Lundry); Brian Beutler summarizes.
Nicholas Thompson says the NBC/WSJ polls shows Americans have different health priorities than Obama.
Alex Bratty sees GOP opportunity amidst low trust in government.
Tom Schaller outlines a general theory of Democratic disgruntlement.
Steve Benen has his usual fun with the Fox News polls.
Tom Edsall expands on the Hart Research findings that average people don't see government stimulus helping them.
Bloomberg/Selzer says investors still fear Rout in Stocks.
Josh Tucker finds factoid in SUSA tabs: Corzine hits 50% among Springsteen fans.
Flowing Data puts cell size in perspective (the other kind of cell size).
And Ana Marie Cox explains "scozzafava'd."
By Mark Blumenthal on November 1, 2009 12:08 PM
| Permalink
| Comments (0)
| TrackBacks (0)
October 31, 2009
By Mark Blumenthal
A quick update to Charles' post on the bad news for Democrat Bill Owens and good news for Conservative Doug Hoffman in the new Siena Research Institute survey in the wake of the withdrawl this morning by Republican Dede Scozzafava.
Charles noted that Scozzafava supporters in the survey (20% of all likely voters) rate Owens almost a negatively (50% unfavorable) as Hoffman (57%). That's important, but so are the party leanings of those voters. The Siena crosstabs did not include party identification tabulation, but I emailed the folks at Siena and they kindly shared the following data:
So Scozzafava voters, as of this past week, identify Republican by a more than three-to-one (64% to 19%) margin. At a minimum, it is clear that many Scozzafava supporters that choose to vote for someone else on Tuesday will have to resolve some attitudes in conflict, especially if Scozzafava herself makes no formal endorsement (her name will remain on the ballot; Nate Silver also points out that they also rate Barack Obama 64% favorable, 31% unfavorable).
Also, a post-script for your how-do-polls-influence-campaigns file: As Josh Marshall noted, in an local newspaper interview, Scozzafava attributed her decision in part to the results of the same Siena poll: "Ms. Scozzafava told the Watertown Daily Times that Siena Research Institute poll numbers show her too far behind to catch up - and she lacks enough money to spend on advertising in the last three days to make a difference."
Finally, PPP put a NY-23 poll in the field this morning that obviously got a little scrambled. Be sure to read Tom Jensen's review of their "NY-23 Polling Odyssey."
Update: Charles has much more here.
By Mark Blumenthal on October 31, 2009 4:22 PM
| Permalink
| Comments (0)
| TrackBacks (0)
October 30, 2009
By Mark Blumenthal
The most recent polling in New Jersey shows an excruciatingly close race between incumbent Democrat Jon Corzine and Republican challenger Chris Christie. As of this writing, our standard trend estimate (below) puts Corzine "ahead" by a negligible 0.8% (41.4% to 40.6%). The more sensitive setting on our smoothing tool makes the Corzine margin slightly narrower (0.6%), the less sensitive setting makes it slightly larger (0.9%). Any way you look at it though, the differences between the estimates -- and more importantly, between Corzine and Christie -- are virtually meaningless. Right now, the current polling snapshot of this race is a close as these things get.
For perspective on the closeness of the margin you might want to stroll down memory lane and revisit my final Election Day update from Tuesday, November 4, 2008. We showed only four states where the Obama-McCain margin on our trend-estimates was less than 2 percentage points, and the leader ultimately won the state in 2 of 4 states. So a margin of under two percentage points puts us well within true toss-up territory in terms of predictive accuracy, especially with a weekend of polling still to go.
Understandably, the close nature of the race has political junkies turning these numbers upside down and reading every possible tea leaf and in search of the key to the outcome. After doing much of the same (while out with the flu) the last few days, the best answer I can give based on the empirical evidence -- for the moment at least -- is that this race is currently looking very close.
Are things trending toward Corzine? Yes, when compared to early September, our chart indicates a decline of roughly four percentage points for Christie and an increase of roughly three points for Corzine. Over the course of the summer, Christie had been dropping (from a high of roughly 49% in early July), while Corzine remained flat.
What is less clear is whether the closing trend has continued over the last two weeks. As of this writing, only three pollsters have tracked more than once since mid-October, allowing apples-to-apples trend comparisons. Two, SurveyUSA and Democracy Corps -- show Corzine's margin two percentage points better. One, Rasmussen, shows it one point worse. None of these differences are statistically significant alone and the patterns are obviously small and inconsistent.
That said, the trend over the next four days may not be as smooth, and the Daggett "wild card" that everyone has focused on for the last few months is the reason. Consider at least three ways that the Daggett effect leaves us even more uncertain about the outcome:
Individual level uncertainty -- The Monmouth University Polling Institute reported yesterday on a focus group they convened earlier this week in Edison, NJ among voters who are still either undecided or just leaning to a candidate. While they explicitly warn against treating the findings as representative of all undecided voters, the most clear finding was a sense of unhappiness with both major candidates: "These voters claim that this is the most difficult election choice they have ever faced. Nearly all said that Jon Corzine has not done a good enough job to deserve reelection. They simply have not heard enough from Chris Christie to cast their lot with him." Their final decision about Daggett, the report says, may come down to whether he has a chance of winning.
Aggregate level uncertainty -- One statistic worth pondering: On the last ten polls, all conducted in the last week, the portion of the electorate that is either undecided or supporting a candidate other than Corzine or Christie averages 16.5% (with a range of 11% to 23%). As a crude measure of voter uncertainty, that's considerably more than 5% or so we saw at this stage of last year's presidential election.
Measurement artifacts? -- Complicating this issue even further are the measurement challenges that pollsters face when testing lesser known independent candidates, especially when voters are unhappy with the top two choices. Offer just three choices and no explicit undecided category and some undecided voters will choose the independent as their way of expressing uncertainty. On the other hand, fail to prompt for the independent and you may measure a number that's much lower (see, for example, the intriguing experiment embedded in the Fairleigh Dickinson poll). Reality likely falls somewhere in between. And no one can be certain of the effect that the other 9 candidates will have.
And finally, there is the intriguing pattern noted earlier this week by PPP's Tom Jensen and explored last night by Nate Silver. Christie has done consistently better on telephone polls conducted using an automated, recorded voice than on those using live interviewers. Using the filter tool on our chart, as of this writing, Christie runs roughly three points ahead of Corzine on the automated polls, but Corzine runs a little less than three points ahead on live interviewer polls. The chart below, which Charles Franklin kindly prepared this afternoon, shows that the difference has been consistent throughout the race (his margins are likely different than on our interactive chart due to his use of slightly different smoothing levels).
We also see a similar though far less pronounced and consistent effect in Virginia, and then only since Labor Day.
What this effect is about, and what it portends for the outcome in New Jersey, I cannot say. Nate Silver has some plausible speculation about automated surveys being potentially more sensitive to an enthusiasm gap between Republicans and Democrats, although if that is true, I have no explanation for why we saw no such consistent difference between automated and live interviewer surveys in the Obama-McCain polling last year. We should have new surveys over the weekend or on Monday from all three automated pollsters in New Jersey (SurveyUSA, PPP and Rasmussen) and from at least three of the live-interviewer polls. So this phenomenon will be interesting to watch.
Either way, the combination of a very close snapshot and many indicators of potential volatility makes for a very uncertain outcome.
By Mark Blumenthal on October 30, 2009 4:33 PM
| Permalink
| Comments (9)
| TrackBacks (0)
October 28, 2009
By Mark Blumenthal
Teagan Goddard and Andrew Sullivan wonder what country Rasmussen Reports is polling; Scott Rasmussen responds.
David Hill reviews the Gallup ideology trends.
Jennifer Agiesta examines the Obama factor in the Virginia governor's race.
Mark Mellman says what's controversial with voters on health reforms is not with lawmakers, and vice versa.
Glen Bolger and Jim Hobart recommend an economic message for Republicans.
Resurgent Republic quarrels with CNN/ORC's cap and trade question wording.
Nate Silver lampoons a misleading Zogby question.
Jim Geraghty ponders divergent polls in New Jersey.
Alan Reifman tracks health care interest with Google Trends.
Megan McArdle explores polling mysteries.
Salena Zito polls opinion polls and finds Americans fed up (via Reynolds).
Jessica Hagy says is it all (via Flowing Data & Sullivan):
P.S.: I'm a bit under the weather with a bug bearing a strong resemblance to H1N1. Hopefully back to full speed soon.
By Mark Blumenthal on October 28, 2009 2:07 PM
| Permalink
| Comments (1)
| TrackBacks (0)
October 27, 2009
By Mark Blumenthal
Yesterday, the Club for Growth, an organization that backs conservative Republicans, released a new survey it conducted on the special election in New York's 23rd District that showed Doug Hofffman, the Conservative Party candidate it has endorsed, running a few points ahead of Democrat Bill Owens with Republican nominee Dede Scozzafava in third. This result differed from two surveys conducted earlier in October by Siena College and Daily Kos/Research 2000 that showed Democratic nominee Bill Owens narrowly leading Republican Dede Scozzafava with Hoffman in third.
Is the Club for Growth result cooked? That's what Nate Silver strongly implied in post last night headlined, "Reality Check: NY-23 Poll May Seek to Alter, Not Reflect, Reality." Let's take a closer look.
The table above shows the results from all six polls that have been publicly released so far for this race. The Club for Growth/Basswood survey is different, in that it shows Hoffman running seven percentage points higher than the two surveys conducted earlier in the month. (UPDATE: a new survey conducted by Neighborhood Research, sponsored by another Conservative group and released while I was drafting this post has results consistent with the CFG/Basswood survey).
However, the trend is consistent with earlier results and recent news. The two surveys from Siena College in late September and mid-October track a seven point decline in Scozzafava's support and a six point increase for Hoffman. Quite a bit also happened over the last week. Last Wednesday night, while the Daily Kos/Research 2000 poll was still in the field, Sarah Palin endorsed Hoffman on her Facebook page, a development that subsequently received national attention. Hoffman also received endorsements from Steve Forbes and Rick Santorum on Friday. Basswood Research conducted their survey on Saturday and Sunday.
Let's be clear: It is always sensible to treat sponsored, internal surveys with extra skepticism when they are publicly released. Political scientists that have studied public polls (examples here and here) find that partisan surveys typically show a an average bias of 2 to 4 percentage points favoring the sponsoring party. One reason for this phenomenon is that most internal polls never see the light of day. Campaigns typically choose to share only those polls showing good news, not bad.
But in this case, Nate Silver is making a considerably stronger accusation. After running through a list of concerns, Silver concludes:
[T]his is very probably not a case, a la Strategic Vision, where the numbers were simply fabricated. But there's an awful lot that a pollster can do short of making up numbers -- asking leading questions, applying implausible likely voter models or demographic weightings, selecting an unorthodox sample frame, etc. -- to produce a result that fits its desired narrative.
Do we have any evidence that Basswood Research used leading questions, implausible likely voter models or demographic weights or an unorthodox sample frame used to produce its survey? Let me take these issues one by one.
Leading questions? Silver concedes in an update that Club for Growth posted a complete filled-in questionnaire that he had not seen when writing his post, although he hints at "fresh" questions raised by seeing the full text. I am not sure what he's referring to, as I see nothing that would obviously bias the result in Hoffman's favor or that deviates sharply from the standard practice of campaign pollsters. For what it's worth, the Basswood questionnaire provides more complete disclosure than the other public polls, in that it provides full text and results of the demographics (omitted by Siena) and the full text of the likely voter screen questions (omitted by both Research 2000 and Siena).
Implausible demographic weighting? Silver is concerned that "[o]nly 14 percent of the likely voters in this poll are age 40 or under, as compared with about 40 percent in the Research 2000 poll." I'd agree with FiveThirtyEight commenter Matt Hogan that if anything, it's the Research 2000 age composition that's implausible. Nearly half (49%) of their likely voters are under 45 years of age. Both the national and New York exit polls for the 2006 general election report only 36% in that age category, and if anything, exit poll estimates tend to be too young.
The sample was also weighted geographically, according to Basswood pollster Jon Lerner, so that the percentage contributed by each county in the sample conforms to the distribution of voters in the 2008 and 2006 elections. I have not attempted to gather county level vote returns for NY-23, but Basswood included the weighted value for each county in the filled-in questionnaire so anyone can evaluate its geographic representation. Among campaign pollsters, that sort of geographic weighting is standard practice.
Unorthodox sample frame? Hardly, although there is an important difference in the sample frames being used in NY-23. Siena College and Research 2000 are using a random digit dial sample -- one that reaches every working landline phone in the district by randomly varying the final digits of telephone numbers in exchanges within the District. When I spoke to him by phone last night, Basswood pollster Jon Lerner confirms that he sampled from a list of registered voters, selecting those who had cast ballots in either the 2006 or 2008 general elections.
While pollsters continue to debate the merits of samples drawn from voters lists versus random digits, the use of lists to survey congressional districts is hardly unorthodox. Pollsters have used list samples to conduct the vast majority of congressional district polling over the last several decades, since gerrymandered district boundaries make random digit sampling impractical in most districts. Telephone exchanges are a crude match to geography below the county level and very few voters can identify their district number when asked. The only reason that an RDD sample is even an option in NY-23 is that most of the district falls within eight undivided counties, leaving only a small portion in three counties that are divided between districts.
Implausible likely voter model? I don't see it. While pollsters differ wildly in their likely voter selection or modeling techniques, the screen used by Basswood seems reasonable and appears to fall within the norms of typical pollster practice.
Let's run the numbers: In last year's general election, according to the Almanac of Politics, just over 253,000 voted for either Obama or McCain in the 23rd District (Voter Contact Services puts the total turnout at 258,000). According to Wikipedia, 199,103 cast a ballot for Congress there in 2008. The nearby 20th District of New York provides another useful statistic -- 160,940 showed up for a special election held there in April. While no one knows for sure how many will turn out next week, it is likely to fall somewhere in the neighborhood of 200,000 (or lower; see the first update below).
So how does the Basswood model compare? The firm Voter Contact Services, which sells list samples, reports that it has 267,599 voters identified as having voted in a general election between 2006 and 2008 -- so that's roughly the population that Basswood sampled. The key question is how many of the sampled voters passed Basswood's screen question, which accepted those who say they are "very likely" to vote in next week's election, but terminates those who say they are only "somewhat" or "not likely" to vote. I do not have the terminate data from Basswood, but when the AP/IPSOS poll asked a similar question of adults using a 10-point screen in early October 2006 (via the Roper Center iPoll database), 69% choose the most extreme "completely certain to vote" response.
That result is typical. Screens based on self-reported intent to vote may look "very tight" but are usually not, as respondents vastly overstate their true intentions. My guess is that the Basswood "very likely" percentage would be higher than the IPSOS, all things being equal, since it offers just three response categories to the AP/IPSOS ten. Regardless, if we assume that the Basswood question identified 60% to 70% of their registered voters as "very likely" voters, that would project to a turnout of something in the range of 160,000 to 190,00, which seems more than plausible.
[Update: I guessed low. According to Jon Lerner, 85% of the
registered voters they called said they were "very likely" to vote in
the special election. See update 2 below].
Of course, reasonable pollsters can and will quibble over how to select likely voters. Recent efforts to validate turnout on list samples have revealed problems with self-reported likelihood questions. But the notion that this particular poll was cooked, that it used leading questions, an unorthodox frame and an implausible likely voter model is not supported by facts available.
Update: I will gladly defer to others with more expertise on predicting turnout, but I probably should have set 200,000 as a high side (small-l) liberal guess at turnout. David Wasserman, the House Editor at the Cook Political Report who spends a lot more time thinking about these things, writes today that in the NY-20 special election earlier this year, "roughly half the number of voters who turned out for the 2008
presidential election showed up for the special election, which
suggests between 110,000 and 130,000 voters could show up for this race."
Via email, David adds, "anything above 150,000 is a pipe dream."
It is also worth considering the advice Twittered yesterday by NBC's Chuck Todd, who saw more than his share of partisan, congressional district polling in his years as Hotline editor:
Be very cautious of ALL NY 23 polling. Why? There's nothing driving turnout; figuring out WHO is going to vote is near impossible.
Which gets back to my larger point. Yes, caution is in order, but it's foolish to single out the Club for Growth/Basswood poll as somehow inherently implausible on such flimsy evidence.
Update 2 - Jon Lerner of Basswood Research emails:
In the CFG poll in NY-23, the percentage of those who were
contacted and screened out for lack of being "very likely" voters
was 15%. In my experience, in a likely low-turnout race such as an
off-year special election, using voter lists with vote history is far
preferable to random digit dialing. On the assumption that turnout in a
special general election will be far lower than turnout in a normal general
election, you want to begin with people with vote history. Still, even
that alone is not good enough, because many on-year general election voters
will not vote in an off-year special election (especially one in which the
highest office on the ballot is congress). So, we try to narrow the scope
further by including only those respondents who have vote history and who say
they are "very likely" to vote in the special election.
Although we do not screen further, my assumption is that even with that screen we
will have a small number of non-voters, as respondents tend to overstate their
likelihood of voting. Thus, while I believe the sample we derived for the
NY-23 survey was as accurate as can be, if it is off at all, it is likely to be
over-inclusive rather than under-inclusive.
[Typos and grammer corrected]
By Mark Blumenthal on October 27, 2009 12:46 PM
| Permalink
| Comments (2)
| TrackBacks (0)
October 26, 2009
By Mark Blumenthal
On Friday I speculated, here and in my column, about what it might take for independent Chris Daggett to win in New Jersey. Several developments over the weekend appear to make that unlikely possibility even less likely.
The short version of the column is that given all the obstacles -- Daggett's weak name recognition, his apparent inability to compete with television advertising and the way all three candidates compete with home town news on New York City and Philadelphia media outlets -- the independent would need something truly extraordinary to change the nature of "free" media media coverage in order to win.
One scenario that seemed at least plausible as of late last week might be for Daggett to pick up the remaining newspaper endorsements to provide a jolt of final week "momentum" coverage. Three things slammed the door on that possibility over the weekend.
First, six newspapers endorsed on Sunday and unlike the earlier endorsement of the New Jersey Star Ledger, none went for Daggett. Republican Chris Christie won endorsements from the New York Post, Press of Atlantic City, the Asbury Park Press and the Cherry Hill Courier-Post, while incumbent Democrat Jon Corzine got the nod from the Trenton Times and the Bergen Record (links via Hotline Wake-up Call).
Second, don't laugh: The New York Yankees won last night, clinching a spot against the Philadelphia Phillies in the World Series, which begins on Wednesday night. Thus, baseball games will compete with political news for five of the next eight nights left before Election Day (and on the front pages the next morning). For a sense of how hard it will be for any of the New Jersey candidates to make news in this environment, consider the front pages of today's New York Post, New York Daily News, the New York Times, even the Philadelphia Inquirer, New Jersey Star Ledger and Asbury Park Press! All feature the Yankees prominently (though the Times tucks the news discretely below the fold), while none but the Asbury Park Press say one word about the New Jersey governor's race.
Third, two polls released since Friday (by Suffolk University and by Democratic aligned Democracy Corps) show Daggett running well below the 19% and 20% measured by SurveyUSA and the Rutgers Eagleton poll late last week. Whatever we might conclude about the methodology of these surveys, the bottom line is that no headlines are currently heralding an independent candidate surging into the mid 20s.
The Suffolk University poll out this morning does look like a bit of an outlier, at least for now, both in terms of the nine-point lead it fives Corzine (42% to 33%) and the very large percentage of undecided voters (15%). The uncertainty they measured, however, may result from the interesting choice the Suffolk pollsters made about their vote question: They attempted to replicate the actual New Jersey ballot by first reading the names of the two major party candidates, Corzine and Christie, and then reading the names of all 10 independent and third party candidates in random order, including Daggett.
On Friday, Monmouth University pollster Patrick Murray speculated that Daggett faces the challenge of being "buried on the ballot" amidst the many independent candidates, while Corzine and Christie always appear first. The Suffolk question, which shows Daggett at only 7%, provides some suggestive evidence to support that hypothesis.
P.S.: Speaking of Murray, he notes on his blog today that the Suffolk poll "appears to have weighted party ID to party registration, a common mistake by pollsters unfamiliar with the New Jersey electorate." He says that because the Suffolk party question -- "Q1. Which political party do you feel closest to - Democrat, Republican or Independent?" -- has results that match New Jersey's voter registration (34% Democrat, 21% Republican, 46% independent or unaffiliated). I'll let Murray explain:
Being "unaffiliated" in one's registration is not the same as being "independent" in one's thinking. We consistently find that at least 1-in-5 unaffiliated New Jersey voters actually see themselves as partisan.
This is a byproduct of New Jersey's semi-open primary system. Why bother registering with a party if you can wait until primary day and do it on the spot? And why bother to vote in primaries if they are rarely competitive? So, New Jersey ends up with a lot of "party-line" voters who never bother to register with their preferred party. They just see no need.
Having done my share of polling in New Jersey, I can confirm Murray's finding. Many New Jersey voters registered as unaffiliated think of themselves as Democrats or Republicans. Murray's post on understanding unaffiliated voters is worth reading in full, especially if you're a pollster hearing this for the first time.
P.P.S.: Democratic pollsters PPP are teasing partial results from a new New Jersey survey that will finish interviewing tonight. PPP's Tom Jensen describes a big increase in Daggett's unfavorable rating among Republicans and says the independent "has the same level of support he did two weeks ago, but now he seems to be hurting Jon Corzine more than Christie."
By Mark Blumenthal on October 26, 2009 1:37 PM
| Permalink
| Comments (0)
| TrackBacks (0)
October 24, 2009
By Mark Blumenthal
NEP wins a ruling against an exit polling ban near NJ polling places; Danny Shea has more.
Gallup's quarterly Obama job approval slips to 53%.
Democracy Corps responds to Glen Bolger's posts on their "semi-secret" poll; Bolger adds more.
Charlie Cook reviews intensity of opposition to Obama in Democracy Corps focus groups
Sarah Dutton examines polling on the public option.
Jon Cohen finds strong support for restricting pay of bailed out executives.
Jennifer De Pinto assesses whether Obama can help Corzine.
Gary Andres weighs the Democrats' Debt Dilemma
Tom Jensen tests a generic congressional vote offering independent/third party as an option.
David Hill is long on Apple, not its politics.
Mark Mellman celebrates a GOP brand in tatters.
Chris Cillizza ponders the rise of independent identification.
Charles Blow fawns over Michelle Obama.
David Paul Kuhn looks forward to an Obama decline below 50%.
Kevin Drum asks about "poll flippery;" John Sides responds.
James Joyner considers the Gallup marijuana poll (via Sullivan).
Andrew Gelman finds older wealthier Americans oppose government health care nationwide.
Junk Charts finds even obviously wrong charts leave many confused.
PPP polls on, what else, balloon boy.
Mary Landrieu suggests an alternative public option poll question.
Evans McMorris-Santoro shares the Bill Clinton polling quote of the week.
And a pollster that can sing! Kristen Soltis channels Janis Joplin (at around 2:42):
By Mark Blumenthal on October 24, 2009 10:22 AM
| Permalink
| Comments (0)
| TrackBacks (0)
October 23, 2009
By Mark Blumenthal
My column makes an early appearance this weekend -- Friday when filed rather than the usual Monday morning -- because it covers the very timely topic of whether independent candidate Chris Daggett might "pull a Jesse Ventura" and score a come-from-nowhere victory in the New Jersey Governor's race. The parallels between the trial heat numbers of Daggett and Ventura are intriguing, but the fact that Ventura had considerably better name identification and a host of other issue make Daggett's climb steeper. Please click through and read in full.
When you are finished, here are some other challenges that got left on the cutting room floor, so to speak. Let's call them items seven through nine (since the column includes a list of six):
7) Ballot position - Monmouth University pollster Patrick Murray points out via email that Daggett's name will be harder to find on the ballot.
Daggett is an independent who will be buried on the ballot. NJ gives the first two lines on the ballot to the D[emocratic] and R[epublican] nominees (randomly assigned). The remaining candidates (there are 9 this year) are all relegated to a 3rd line - again randomly assigned, which means Daggett's name will be buried amidst a bunch of no hopers and thus very difficult to find.
I'm not sure how the ballot order was handled in Minnesota in 1998.
8) Attitude and Turnout -- Former Star Tribune polling editor Rob Daves argues that a key to Ventura's success is that he "ran on attitude and not agenda." In other words, he used his celebrity persona to stake out a different sort of message, one less about traditional issue positions and more about what one biographer called a "down to earth," non political attitude. That message helped Ventura surge among younger voters who do not typically vote in off year elections. As the Almanac of Politics (2002) put it:
In a year when turnout nationally mostly sagged, turnout surged in Minnesota, especially in the outer counties of the Twin Cities media market; in many counties turnout rose 40% or more from the last off-year election, and was even above the presidential year of 1996. This was the area where Ventura ran best, with 45%, to 34% for Coleman and only 21% for Humphrey.
By contrast, as Patrick Murray notes, it was Chris Christie that tried to run on attitude rather than agenda in New Jersey this year:
Christie tried to use his prosecutorial reputation to paint himself as someone who would be tough with the special interests. And therefore New Jersey would be a better place.
This had two effects. One, it opened him up to opposition research on his character. And it emphasized for voters that his campaign was lacking specifics. That opened up a hole for mild-mannered Daggett to fill with his very specific proposal to address New Jersey's number one issue - property taxes. The kind of specifics that some anti-Corzine voters were looking for, but were not getting from Christie.
Whether this contrast makes Daggett's task easier or harder relative to Ventura remains to be seen. The point is, they are running very different races.
9) Same Day Registration -- Jennifer Duffy of the Cook Political Report passes along one more difference: Minnesota had same day registration in 1998, which helped enable the surge in turnout that Ventura's campaign produced. New Jersey in 2009 does not.
By Mark Blumenthal on October 23, 2009 4:33 PM
| Permalink
| Comments (1)
| TrackBacks (0)
October 22, 2009
By Mark Blumenthal
Self promotion alert: Tomorrow (Friday), I'm participating in briefing on Capitol Hill -- "Public Opinion on Health Reform: What Do the Polls Mean? -- sponsored by the Alliance for Health Reform. The other panelists include Karlyn Bowman, a senior fellow at the American Enterprise Institute; Humphrey Taylor, chairman of Harris Interactive; and Mollyann Brodie, vice president of the Kaiser Family Foundation. Brodie will be presenting new results from Kaiser's October Health Tracking Poll.
The briefing will be held at the Hart Senate Office Building from 12:15 to 2:00 PM, and I am told that on site registration will be available. Click here for more details. Hope to see you there.
By Mark Blumenthal on October 22, 2009 10:42 PM
| Permalink
| Comments (0)
| TrackBacks (0)
By Mark Blumenthal
The conservative reaction to Tuesday's new ABC News/Washington Post poll did not stop with claims that the partisan balance of the respondents was "rigged." It also included a furious push-back over the wording of the "public option" question, no doubt fueled by the Post's decision to make that particular result the lede of their front page story (something even Nate Silver found "somewhat bizarre"). In reviewing some of the criticism, I discovered a result overlooked in June that should cheer advocates of the public option almost as much as this week's ABC/Post poll.
Let's start with the text of the ABC/Post question: "Would you support or oppose having the government create a new health insurance plan to compete with private health insurance plans?" Their most recent survey of adults, conducted October 15-18, found 57% supporting the proposal, 40% opposing it and 3% with no opinion.
A chorus of conservative critics jumped on the question language:
- Pollster Ed Goeas: "Nowhere does this question indicate that the program would be government run, and it is a quite a stretch to conclude that the 57% support is for the public option.
- Blogger Jay Cost: "ABC News/WaPo presents the idea that the government insurance plan would 'compete' with private insurance plans. This is a contested notion, as Republicans think that the public option will drive private insurance away."
- Fox anchor Gretchen Carlson: "That's a little bit different than "do you support the government run option or not ... saying 'do you like the government run option. That would be more straightforward."
- Pollsters Gary Andres and Whit Ayres: "When Americans are asked a one-sided question about whether they support a public option that competes with private insurance, it's not surprising a majority says 'yes.' It's just another 'choice,' 'more competition' and it's perceived as a way to make health care more affordable. Why wouldn't a proposal like that generate wide support? Just like 'world peace' or 'ending poverty.'"
There is some truth to this criticism. Given that only 56% of Americans are able to associate the phrase "public option" with the health care debate, it is safe to assume that with questions like the ABC/Post public option measure are closer to testing reactions to possibly unfamiliar concepts than to measuring pre-existing attitudes about the "public option." When you do that, the results are very sensitive to question wording.
In July, for example, the Kaiser Family Foundation found 59% in favor of "creating a government administered public health insurance option similar to Medicare to compete with private insurance plans." But when they threw one-sided arguments against the public option at supporters and one-sided arguments for it at opponents, they found they could push support as low as 35% or as high as 72% (something I reviewed in August in a post that Andres and Ayres linked to yesterday).
I am not opposed to questions that test reactions to unfamiliar concepts. They are part of understanding public opinion on many subjects, including the health care debate. Americans may want something, after all, even if they are not familiar with the terminology. When we test reactions, however, my own preference runs to questions -- mostly shunned by media pollsters -- that present both sides of an argument using the rhetoric typically lobbed by partisans. That's why this passage in the Andres-Ayres post caught my eye:
[I]n our Resurgent Republic Health Care poll we provided voters real world arguments about proposals - the up sides and the down sides - before asking for a response . . . We have no doubt that public attitudes about a public health insurance plan could change. But that all depends on the information presented. As is evidenced from the three questions in the Resurgent Republic health care poll, providing voters with more background and arguments produces mixed results for the public plan option.
"Mixed?"
I had forgotten about Resurgent Republic's June health care poll so I pulled up the toplines. Here are the results to those three questions:
Q15:
-47% agree: Congressman A says Americans need a public health insurance plan administered by the federal government to expand choices and control costs by competing with private health insurance companies.
-45% agree: Congressman B says a government-run health insurance plan will use taxpayer subsidies to undercut private insurance rates, and force private companies out of business, resulting in everyone going into a government-run plan.
-8% unsure
Q16:
-57% agree: Congressman A says a public insurance plan will allow people to keep the plan they have now if they want, or give them the choice of a public plan. It will shift power from insurance bureaucrats to consumers.
-38% agree: Congressman B says a public insurance plan will inevitably force everyone into a "one size fits all" government-run plan that will take away choices. It will shift power from consumers to government bureaucrats.
-5% unsure
Q17:
-53% agree: Congressman A says a public insurance plan is a limited option to allow citizens to have one more choice for health insurance and will force private plans to stay competitive on costs and services.
-43% agree: Congressman B says a public insurance plan is the first step toward a government take-over of health care similar to Europe and Canada, with fewer covered procedures, long wait times for surgery, and more government bureaucracy.
5% unsure
With the possible exception of the first question, these are not results I would describe as "mixed." All three show more support for the public option than opposition. The last two questions produced majority support among registered voters in June that is fairly close to what the ABC/Post poll found last week among all adults. (I also assume that most public option advocates would react to the language of Q15 as conservatives reacted this week to the ABC/Post question. Does "expand choices" really capture the promised benefit as much as "give them the choice of a public plan?" But I digress...).
The main point here is that a group of Republican pollsters took their best shot at a set of questions that would capture both the costs and benefits of the public option, presenting the very arguments so many found lacking in the Post/ABC question this week. In all three instances, more voters favored the public option than opposed it.
I wonder if they tried "to get the same amount of Democrats and Republicans" in their sample, to be "fair and balanced" and all?
P.S.: To be fair, the Resurgent Public poll was conducted in June, just as general opposition to health care reform was increasing most rapidly. Public Option questions included in polls conducted at about the same time by ABC/Washington Post and CBS News found support that was 5 to 10 percentage points higher than on their most recent surveys. Still, I would expect far more stability in measures like those tested by Resurgent Republic that provide respondents with far more information. And even if a few points closer than what they found in June, the Resurgent Republic tests suggest more robust support for the Public Option than some assume.
By Mark Blumenthal on October 22, 2009 3:39 PM
| Permalink
| Comments (3)
| TrackBacks (0)
October 21, 2009
By Mark Blumenthal
Few things are more perennial in the survey world than partisan attacks on high profile polls that produce a result that partisans don't like. More often than not, those attacks involve the issue of party identification. One of my first posts as a blogger five years ago involved an argument about party weighting (Democrats thought the polls were skewed too Republican) and a subject has come back like crab grass ever since. Yesterday, the object was the ABC News/Washington Post poll, and the complaints came from a Republican pollster and former Republican Speaker of the House Newt Gingrich.
The complaint from Gingrich drew a response worth reading from ABC News polling director Gary Langer, so I'll start there. The tempest was probably triggered by findings highlighted in the last two paragraphs of the Washington Post's front page poll story:
Only 20 percent of adults identify themselves as Republicans, little changed in recent months, but still the lowest single number in Post-ABC polls since 1983. Political independents continue to make up the largest group, at 42 percent of respondents; 33 percent call themselves Democrats.
The wide gap in partisan leanings and the lack of confidence in the GOP carries into early assessments of the November 2010 midterm elections: Fifty-one percent say they would back the Democratic candidate in their congressional district if the elections were held now, while 39 percent would vote for the Republican. Independents split 45 percent for the Democrat, 41 percent for the Republican.
Gingrich responded by telling a Salt Lake City radio station that he considered the ABC/Post poll "deliberately rigged." According to Langer, when asked about the finding that only 20% of Americans consider themselves Republican, Gingrich replied:
Well, it tells me first of all that the poll's almost certainly wrong. It's fundamentally different from Rasmussen. It's fundamentally different from Zogby. It's fundamentally different from Gallup. It's a typical Washington Post effort to slant the world in favor of liberal Democrats.
Langer produced the following table, showing that the ABC/Post estimate falls roughly in the middle of other recent national surveys of adults:
Now as both Langer and PPP's Tom Jensen point out, Gingrich was conflating two issues: Concerns about party identification and about the so-called generic house vote. In the same radio interview, for example, he pointed to a recent Gallup poll showing a closer margin in the national House vote and, according to Langer, emphasized the generic vote in a second interview later in the day,
Here Gingrich is on somewhat firmer ground: The polls that he cites -- especially Rasmussen and Zogby -- report on "likely voters" that tend to look more Republican than the adults sampled by other pollsters. While neither pollster provides much information on how they currently define likely voters, the universe of actual voters in 2010 will likely be far more Republican than the full adult population. Keep in mind, though, that the generic ballot is a blunt instrument that tends to produce wide variation among pollsters even in the week before the election when all are doing their best to identify or model the likely electorate. Bottom line: While the generic ballot is a useful measure (see these posts on efforts to use it to model the outcome) it is not infallible.
Back to the argument about party ID. Why does Langer leave out Rasmussen and Zogby, two of the three pollsters that Gingrich cites as looking fundamentally different? The implied answer is that ABC News considers neither pollster "air-worthy" (they explain their standards here), but even if you quarrel with that judgement, there are good reasons why such a comparison would be foolish.
Let's start with Zogby. First, Zogby reports results among "likely voters," not adults. Second, they do not -- as far as I know -- report their party ID results to non-subscribers (if readers can point to recent examples to the contrary, or to results from behind their subscription wall, I will gladly correct this post). Third, unlike the organizations listed in Langer's table, Zogby weights every survey by party ID, usually to match the estimate from exit polls in a prior election. So there is no point in comparing Zogby's weighted party ID numbers to those from recent polls of adults, even if they were available. It's like comparing an apple to fruit cocktail.
And what about Rasmussen? Like Zogby, Rasmussen typically reports results among likely voters and weights by party. However, Rasmussen does a service by routinely releasing their monthly party identification numbers among all adults, weighted only by demographics. Their results for September do look a lot different from any of the other pollsters: 37.5% Democrat, 32.1% Republican, 30.4% other.
Why might Rasmussen's party ID results look so different? It might be because of the kinds of people they sample as compared to other polls, but there are two other huge differences to consider. Rasmussen calls with a different mode (automated rather than live interviewer) and asks a different question. Other pollsters begin by asking respondents what they "consider" themselves to be, prefaced by the phrase "generally speaking" or "in politics, as of today," with the options typically Republican, Democrat, independent or "something else." Rasmussen simply asks:
If you are a Republican, press 1. If a Democrat, press 2. If you belong to some other political party, press 3. If you are independent, press 4. If you are not sure, press 5
If you believe that party ID is like eye color, that we are all either Democrats, Republicans or something else and that we will always provide the same answer under any circumstances, even if shaken awake during a deep sleep, well...it probably doesn't matter how the pollster measures it. But there is a ton of evidence that although the aggregate party ID numbers change very, very slowly, at the individual level all sorts of things can alter the answers that respondents give, especially if they are borderline between independence and identifying with a party: the wording, when the question is asked, what questions come before, how hard the interviewer pushes for an answer, and so on. So it is quite possible that people are willing to report their party identification differently when asked by an automated recording rather than a live interviewer, especially when the text of the questions differ. Comparisons between Rasmussen and other pollsters on this score prove little.
Was the ABC/Post poll right that Republican identification among adults is lower than ever? There we do have some pretty convincing evidence. As our Party ID chart (above) shows, their result was far from an outlier. Even with the sensitivity set to low (to diminish short term statistical noise from "house effects), the downward trend in both Democratic and Republican identification is evident. Click on the red dots below (to connect-the-dots for individual pollsters) and you will see that almost every pollster shows a decline in Republican identification over the course. The clear rise in independent identification may not help us forecast the results of the 2010 elections, but the trend is nevertheless undeniable. The notion that ABC and the Post "rigged" their result is laughable.
By Mark Blumenthal on October 21, 2009 4:01 PM
| Permalink
| Comments (4)
| TrackBacks (0)
October 20, 2009
By Mark Blumenthal
The New York Times rails against a proposal that will "waste money and ruin the Census."
Gary Langer digs deeper into the ABC/Post results on Iran.
Jennifer Agiesta shares follow-up interviews among ABC/Post respondents on Obama's accomplishments.
Lymari Morales explains why Gallup measures well being.
Anthony Salvanto and Mark Gersh crunch the New Jersey numbers.
Patrick Murray finds a lot of individual level "churn" in NJ.
Ed Kilgore considers two "ceilings" in NJ.
Rob Farbman finds just enough NJ love for Corzine.
Tom Jensen thinks automated polls may be more accurate in Maine.
Nate Silver examines turnout's impact on Maine's Question 1.
Glen Bolger reviews the "semi-secret" Democracy Corps poll.
Andrew Sullivan mines a Democracy Corps focus group report.
Greg Sargent notes that the ABC/Post generic House ballot results look a lot like 2006 & 2008.
Ed Goeas attacks the ABC/Post poll showing increasing support for the public option.
Resurgent Republic examines the Pew Research results on health care.
The Franklin & Marshall pollsters call for better polling disclosure.
Survey Practice publishes its October issue on cell phone-only households, mail ballot registration and more.
Renard Sexton follows the Afghanistan election audit.
Patricia Cohen (NYT) examines the relevance of Political Science and Tom Coburn's effort to gut its funding (via Sides).
Henry Farrell has more on the value of Political Science.
And while I was busy, Katie Couric interviewed Frank Luntz about his new book and, yes, his focus group of Playboy Playmates (video explains my headline, among other things):
By Mark Blumenthal on October 20, 2009 3:58 PM
| Permalink
| Comments (0)
| TrackBacks (0)
By Mark Blumenthal
My column on "opt-in" Internet panel surveys yesterday ended a bit abruptly, leaving at least one reader confused about my meaning. So let me "revise and extend" as we say here in the nation's capital, beginning with the column's two final paragraphs:
[Prof. Jon] Krosnick wants to be clear that he sees no evidence yet that "opt-in sample surveys are as accurate as or more accurate than probability sample surveys," and given their lack of foundation in probability sampling, he is not optimistic that they ever will be. "It's essential for us to be honest about what our data are and what they are not," he added.
True enough, but I would add one thought. Our honesty should extend to the limitations of probability samples as well. In the Krosnick-Yeager study, for example, despite very sophisticated weighting, that very expensive, very rigorous telephone survey still produced errors outside the margin of error on 4 of 13 benchmarks. By random chance alone, it should have produced no more than 1.
To be clear, I did not mean to imply that David Yeager, Krosnick or any of their co-authors** were hiding anything about the margin of error. The statistics they produced on the percentage of "significant differences from benchmark" appear in Table 2 of their report and were cited in Gary Langer's initial post on the study.
Rather, the point I am trying to make is that some have reacted to this study in ways that feed a reflexive, binary, good vs. evil view of the differences between probability samples and those from opt-in internet panels. Many hear the praise of standard probability sample as "valid and reliable" or "very consistently accurate" as implying infallibility. These statements also lead many to believe that any survey conducted via the Internet is cheap "crap" that any "reputable pollster would stay away from."
My point: All surveys involve some sort of trade-off between cost and accuracy. Studies like the one we've been discussing give us some tangible sense of the differences in accuracy when using opt-in samples.
The point about the potential fallibility of conventional probability sampling was made by the same Jon Krosnick (and co-author Morris Fiorina) in a 2004 paper that I linked to in last week's column:
To be sure, though, the +/- x percent margins of error that accompany many widely- publicized survey results are misleading. This is true partly because these margins of error describe only sampling error, whereas we know many other sorts of error are present in survey data, including errors caused by interviewers and respondents when reporting and recording answers to questions. But in addition, at least in theory, these margins of error underestimate sampling error itself. Such estimates are accurate only if the respondents ultimately interviewed are a random draw from the original sample, and the less than perfect response rates that typify public opinion polls certainly come with non-response bias in terms of demographics. It is not an exaggeration to say that conventional public opinion surveying today begins with probability samples, then loses successive portions of the sample but hopes that in the end, the losses will cancel out or be corrected by statistical weighting using demographics, so the sample that remains will be a reasonable approximation of the population.
What is most encouraging about the Yeager, Krosnick, et. al. study is that so many in the survey world are looking to the results as a useful way to judge the true accuracy of different kinds of surveys. There is much disagreement about what to make of the differences reported, but hidden silver lining is that nearly everyone agrees that this sort of "results based analysis" has merit. That's progress.
**The original version of the column may have left the impression that David Yeager and Jon Krosnick wrote the paper on their own, which did a disservice to co-authors LinChiat Chang, Harold S. Javitz, Matthew S. Levendusky, Alberto Simpser and Rui Wang. Apologies for the oversight.
By Mark Blumenthal on October 20, 2009 1:20 PM
| Permalink
| Comments (1)
| TrackBacks (0)
October 19, 2009
By Mark Blumenthal
My column this week concludes a two-part series on the subject of "opt in" internet panel surveys and a recently released, much debated accuracy study by Jon Krosnick, David Yeager and their colleagues.
For further reading: last week's column, the blog post by ABC's Gary Langer that shared the Krosnick-Yeager report, the reaction by Doug Rivers of YouGov/Polimetrix (which is also the owner and principle sponsor of Pollster.com) and a response from Krosnick, Yeager, et. al.
By Mark Blumenthal on October 19, 2009 9:15 AM
| Permalink
| Comments (0)
| TrackBacks (0)
October 16, 2009
By Mark Blumenthal
Here's a quick apology for my lack of blogging today and (in advance) for tomorrow too. I was traveling for much of today to Baton Rouge, Louisiana where tomorrow I'm participating all day in the John Breaux Symposium at the Reilly Center for Media & Public Affairs at Louisiana State University.
If you happen to be in Baton Rouge and have some free time tomorrow, tomorrow's discussion should be terrific. The topic is "Redefining Public Opinion Polling in an Age of Segmented Markets and Personalized Communication." In addition to yours truly, the panelists include our own Charles Franklin, Charlie Cook of the Cook Political Report, Scott Keeter of the Pew Research Center, Anna Greenberg of Greenberg Quinlan Rosner and Susan Herbst of Georgia Tech.
Meanwhile, two quick "outliers":
Louis Jacobson of Politfact.com did a fact check on Glenn Beck's citation of a result from the IBD/TIPP poll of physicians that I discussed on Pollster and in a column last month. Jacobson's piece includes considerable new reporting on the issue -- including the full text of the questions asked in the survey. It's worth a click.
Finally, we overlooked a new Rasmussen poll in New Jersey today. I just added the trial heat question to our chart; we should have the usual poll update post in the morning.
By Mark Blumenthal on October 16, 2009 12:21 AM
| Permalink
| Comments (0)
| TrackBacks (0)
October 14, 2009
By Mark Blumenthal
Regular readers may recall my lament that despite the many, many times pollsters have asked Americans to react to descriptions of the so-called "public option" in health care reform, very few have attempted to probe the depth of Americans' knowledge about the proposal.
The point of debate is often whether Americans "want" a public option. These arguments sometimes get expressed in the context of representative democracy. "65% of Americans are begging Congress for an inclusive public option," wrote one Daily Kos diarist a few weeks ago, yet "our Representatives in the Senate are REFUSING to give The People what they overwhelmingly want." Poll questions like the one cited by the diarist typically measure how Americans react to a brief description of the Public Option concept. Those are helpful, but if Americans are really "begging" for a public option, we might also want to measure how many know what the public option is before hearing the pollster's description.
Unless I've missed it, the only effort to tackle this question was an opt-in internet panel survey conducted by Penn, Schoen and Berland Associates and presented at an event they sponsored with AARP [see correction below]. They found that only 37% of their adult sample could correctly identify the Public Option from a list of three possible choices. As Nate Silver pointed out, if the respondents had picked randomly, a third (33%) would have chosen correctly.
Today, we have a new measure from a respected source that has often probed Americans about their level of political knowledge: The Pew Research Center today released an update of their semi-annual "News IQ Quiz," a survey that poses a dozen multiple choice or true/false questions about key facts in the news.
Today's update includes three questions about the health care debate, reproduced in the box below:
The Pew survey finds that a majority of adults (56%) are able to associate "public option" with health care rather than another issue. On the one hand, as the report points out, that awareness ranks toward the high end of Pew's awareness questions. On the other, as a gauge of knowledge, the bar is pretty low. Recognizing that the term "public option" has something to do with health care does not mean you can explain what the term means, how it might work or who it might cover. At least we know that nearly half of Americans (44%) have no clue that the term even involves the health care debate.
How many Americans both know what the public option is and "want" it? Unfortunately, the Pew survey includes no favor-or-oppose questions, so we have to guess, but the number probably falls far short of a majority. The Pew Report does tell us that Democrats are just as likely as Republicans to correctly answer the public option question (59% for each), so it's unlikely that the more knowledgeable are radically different in terms of their attitudes about specific reform proposals. In other words, are 65% really "begging" for a public option? Not likely.
Of course, attitudes about public policy are seldom static, and it's fair to expect our representatives to take into account attitudes that are "latent" and might flower into actual opinions as awareness grows later in the debate or during some future election campaign. That said, health care has been about as high profile as legislative debates get, so we are not likely to see big swings in awareness or opinion over the next few months. What people "want" now is pretty much what they will want when members of Congress cast their floor votes on health care legislation later this year.
Correction: The original version of this post said the Penn, Schoen, Berland poll was "sponsored by AARP (an organization that has not taken a formal position on the public option)." PSB emailed to say that while they presented their survey at an event they sponsored with AARP, the seniors groups did not pay for the poll itself.
By Mark Blumenthal on October 14, 2009 3:31 PM
| Permalink
| Comments (0)
| TrackBacks (0)
October 13, 2009
By Mark Blumenthal
My column this week takes up the subject of "opt in" internet panel surveys and a recently released, much debated accuracy study by a team of Stanford researchers. As this subject is too big for a single column, I've split it into two parts. This week, in Part I, I take up the issue at the heart of the controversy: random probability sampling.
By Mark Blumenthal on October 13, 2009 10:25 AM
| Permalink
| Comments (0)
| TrackBacks (0)
October 10, 2009
By Mark Blumenthal
Barack Obama wins the Nobel Peace Prize; The Pew Research Center, Frank Newport, Gary Langer and the Transatlantic Trends Survey provide the public opinion context.
Tom Coburn proposes to eliminate NSF funding for political science, Monkey Cagers John Sides, Andrew Gelman, Joshua Tucker and Henry Farrell react.
Tom Jensen sees little change in health care reform support since September.
Mark Mellman offers an answer for why Jews are liberals.
Democracy Corps finds growing damage to the Republican brand.
Tom Schaller examines the national Congressional ballot 300 days out.
Eric Kleefeld recalls a history of late Democratic surges in New Jersey.
Andrew Sullivan parses the Quinnipiac Poll.
Greg Sargent endorses a Research2000 question on health care and partisanship.
Evan McMorris-Santaro reviews reports a state investigation of opinion research conducted with tax dollars for CT Governor Jodi Rell.
Calbuzz publishes its Standards for Polling, Decency and Free Lunch.
Amanda Lenhart reports that new Pew Internet data shows Twitter usage rising to 19% of American adult internet users, up from 11% in April (via Christine Matthews).
By Mark Blumenthal on October 10, 2009 11:19 AM
| Permalink
| Comments (0)
| TrackBacks (0)
By Mark Blumenthal
We've had a flurry of new polls this week in New Jersey which are generally showing a much closer contest than in September or earlier in the summer, but there is a huge wild card in this race and it's all about independent Chris Daggett.
The best news for the campaign of Democrat Jon Corzine, however, is not the margin but the fact that the Governor's support is starting to rise slightly. You can see the upward tick in the blue line from roughly 39%, where it stood for much of the summer to 40.6% as of this afternoon.
If you're a skeptic of our trend lines, you can also see the improvement in apples-to-apples comparisons in individual polls. Five surveys released in the last week (by Monmouth/Gannett, DailyKos/Research2000, Rasmussen, Democracy Corps and Neighborhood) all showed increases for Corzine of 1 to 3 percentage points as compared to prior tracking polls done by the same organizations in September. In any one survey, these minor shifts would be too small to achieve statistical significance, but the consistency across the five surveys gives us greater confidence that the change is real.
That news is important, because Corzine will need to continue growing his support. The encouraging news for Republicans and not-so-great news for Democrats is that a roughly seven point drop in Republican Chris Christie's support since the summer has gone mostly to independent Chris Daggett. A very large percentage of the New Jersey likely voters (an average of 18% on the surveys released this past week) are either undecided or supporting Daggett or are undecided. Daggett's support is the most important wild card in the race.
Consider this finding buried within the extensive cross-tabs provided by the Democracy Corps poll released on Thursday, that includes the subgroup of 89 respondents currently supporting Daggett. Not surprisingly, supporters of the independent rate both Corzine and Christie negatively. Using a "thermometer" rating that allows respondents to rate the degree of warmth or coolness the feel for each candidate, Daggett voters give Christie a overwhelming negative rating (8% warm, 59% cool) but are even more negative about Corzine (10% warm, 76% cool rating; see p. 44).
What should be worrisome for Democrats, however, is how these two ratings compare. Far more of Daggett's supporters rate Christie more favorably than Corzine (50%) than rate Corzine more favorably than Christie (24%; see p. 59). We know that support for independent and third party candidates often falls as election day approaches. That may be because some voters are strategic -- feeling reluctant to "waste a vote" when the contest between the top two candidates is close. It may be because the nature of three-way polling question makes support for the independent a sort of holding place for undecided (a polite way to say "I'm not sure" while selecting one of the offered choices).
But either way, Corzine's prospects depend on Daggett retaining his current support. Election Day is a little over three weeks away and for now Daggett's trajectory is up, not down, based partly on his debate performance a a week ago, so Daggett's trend line may not follow the traditional pattern. Democrats should hope so, because a collapse in Daggett's support would be a scary scenario for Corzine.
By Mark Blumenthal on October 10, 2009 8:06 AM
| Permalink
| Comments (0)
| TrackBacks (0)
October 8, 2009
By Mark Blumenthal
Just a quick note to let everyone know that our "poll updates" and my blogging will be a little slower and less frequent than usual through the weekend, as Emily is taking some time to travel to a family commitment. We'll be back up to full speed next week.
By Mark Blumenthal on October 8, 2009 1:00 PM
| Permalink
| Comments (1)
| TrackBacks (0)
October 7, 2009
By Mark Blumenthal
Marc Ambinder reports on how the Democrats won the data war in 2008, posts the after action report from Catalist and a summary of what SEIU learned from its efforts.
John Sides sees similarities in the Catalist report to political science research on voter turnout.
Gary Langer reviews the evolution of attitudes on the Afghanistan War.
Bruce Drake rounds up Obama approval by state.
Chris Weigant updates his Obama job approval watch.
Nate Silver weighs in on New Jersey.
Chris Good takes issue with Bobby Jindal on public opinion on health reform; Mickey Kaus takes issue with Good.
Andrew Sullivan highlights a global branding survey that shows the U.S. as most admired; David Frum thinks its a silly survey; Sullivan responds.
Jonathan Singer uses new SurveyUSA numbers in Minnesota to smack back at Rasmussen.
David Hill thinks the H1N1 vaccination drive could use a pollster.
Alex Bratty shows that Republicans are behind the 2009 increase in interest in politics.
Brent Seaborn charts the shifts in party identification.
Tom Jensen is bullish on Beau Biden against Mike Castle.
Dana Stanley reviews a new iPhone survey application.
Josh Marshall catches Chris Christie making an odd polling reference.
And to the latest Marist poll release we say, "whatever."
(And a note: Blogging will be a little slow through the weekend as I cover for Emily on poll updates. She is taking some time off for a family commitment. We'll be back up to full speed next week).
By Mark Blumenthal on October 7, 2009 4:36 PM
| Permalink
| Comments (0)
| TrackBacks (0)
By Mark Blumenthal
The Wall Street Journal's Carl Bialik weighed in today on the Strategic Vision, LLC controversy in both his print column and a separate blog item. Collectively, like Shaila Dewan's New York Times story on Sunday, they provide a decent, concise overview of the story for those that have not been following it. Unfortunately, there is little news for those of us that have been following the story closely. According to Bialik, Strategic Vision CEO David Johnson "didn't respond to Wall Street Journal requests for comment."
Bialik did seek comment from various mathematicians regarding the claims of statistical irregularities in Strategic Vision's results made by Nate Silver (here, here and here) and others at FiveThirtyEight.com:
This week, Mr. Silver brought in a physicist and commenter on his blog to calculate the probability, which shrank to 5,000 to 1 against, when removing what he said was an unproven assumption that each digit should appear equally often. Several mathematicians said the shift in odds doesn't diminish Mr. Silver's finding that the Strategic Vision numbers were unlikely to arise by a quirk of fate.
He added additional details in the blog post:
Mathematicians said the Silver analysis -- finding that certain digits showed up far more often than others in Strategic Vision polls -- was troubling but want to see more evidence. Jordan Ellenberg, a University of Wisconsin, Madison, mathematician, blogged that the case isn't as persuasive as investigations into possible fraud in the Iranian election. "It's not so substantial that I would have gone public with it, if it were me," Ellenberg said, but he does think it merits further investigation.
"To strengthen the argument that Strategic Vision's (or any other polling group's) numbers seem unusual, the next step would be to assess the observed variation across a number of similar polling organizations and see where various groups fall," said Lance Waller, a biostatistician at Emory University.
One bit of news that Bialik passes along is that some of Strategic Vision's clients are now pressing the company for more verification of their data:
Two think tanks that are clients of Strategic Vision also are seeking more details on the firm's methods in light of Silver's analysis. The Goldwater Institute, which calls itself a free-market think tank, and the Oklahoma Council of Public Affairs hired Strategic Vision to test high-school students' civic knowledge in Arizona and Oklahoma, respectively.
After Silver questioned the Oklahoma results as being too bleak, both think tanks sought verification from Strategic Vision. "Although I find it very unlikely that Strategic Vision manufactured this data, I have asked for receipts from the marketing firm from which they purchased the contact data just to make certain," Matthew Ladner, vice president of research for Goldwater Institute, said.
Brandon Dutcher, vice president for policy for the Oklahoma group, isn't making up his mind just yet. "I have requested voluminous survey data from them, as well as answers to some methodological questions -- all of which I expect they can and will provide so that they can go about defending their firm and I can go about defending this survey," Dutcher said. "If not, however, then of course I would want my money back and wouldn't hire them again.
See both the article and blog post for all the details.
Bialik observes that while the controversy has gotten "bogged down in threats of litigation and arcane calculations," the controversy "has shed light on an inconvenient truth about widely reported political polls: Verifying their numbers is nearly impossible." That conclusion is hard to argue with and a big reason why better disclosure -- the issue behind the AAPOR reprimand that helped bring these issues to a head -- is so important. Only greater transparency will prevent these sorts of controversies in the future.
By Mark Blumenthal on October 7, 2009 1:51 PM
| Permalink
| Comments (0)
| TrackBacks (0)
October 6, 2009
By Mark Blumenthal
Back in early September, I wrote a column (with some added commentary here) with a suggestion for President Obama and the Democrats:
Challenge Congress to pass a reform bill that requires all members to obtain their health insurance the same way as those without employer-provided health insurance -- through the newly created health care exchanges, rather than the Federal Employee Health Benefit Plan [FEHBP]. The two systems are conceptually similar -- similar enough that the pledge could help sell voters on the benefits of the exchange itself.
Well, in doing research on what aspects of the Senate Finance health reform bill take effect when, I noticed this interesting provision as described by the very helpful Kaiser Family Foundation tool for making Side-by-Side comparisons of the various Congressional bills. The Senate Finance Bill appears to be the only one that includes this provision: "Provide elected officials and federal employees the option of purchasing coverage through the exchanges or through FEHBP."
In other words, while not mandatory, any member of Congress could choose to buy their coverage through the same mechanism that will be offered to those who currently lack insurance.
So, a bit more unsolicited advice for any member of Congress thinking about voting for health care reform: First, make sure this provision stays in whatever version of the bill gets to the floor of the House and Senate. Second, make a pledge, loud and clear, to buy your coverage though the exchange once it goes into effect in 2013.
And pollsters, if you haven't tested this message yet for your clients, you should. The script for the TV ad practically writes itself.
By Mark Blumenthal on October 6, 2009 4:04 PM
| Permalink
| Comments (1)
| TrackBacks (0)
By Mark Blumenthal
In the spirit of transparency, we need to provide full disclosure of a mistake we made and an apology to our readers and to Rasmussen Reports.
In the year since the election, we have worked to create a new collection of charts that track all available surveys not just for election trial heat results but also a series of national measures, including presidential job approval ratings, favorable ratings, the national "generic" congressional ballot, the classic "right direction wrong track" question and a handful of measures of perceptions of the economy.
We started putting up new charts after the 2008 election knowing that some would "work" -- we would find enough reasonably comparable data from a variety of sources to make for a robust trend line -- and some would not. It was also probably inevitable that we would make a mistake or two along the way.
Well, as I discovered this past week, we did. For a handful of charts, we have been republishing some extraneous data from behind the gated subscriber pages on RasmussenReports.com. The affected charts are two that track perceptions of the economy (excellent/good/fair/poor and getting better/getting worse), our Obama favorable rating chart and the three charts that track the Obama job rating by party (Democrat, Republican and independent). Rasmussen does provide some results from these questions (usually just from one answer category) on their free-to-all "By The Numbers" page. For the economic charts, that represent the bulk of the data we misused, we were filling in results from the subscriber tabs for data omitted on the By-The-Numbers page. Compounding the error, as noted last week, in one instance we were including numbers on our Obama favorable rating chart that were actually mislabeled job approval rating results.
I could tell a long story about a small error that cascaded, but it all boils down to a lack of clear communication by me. As such I deserve and will take full blame. So there is no confusion in the future, our policy henceforth is iron-clad: We will not republish a single number in our charts unless it has already been published or released into the public domain by the pollster or sponsor.
Knowing that Josh Tucker has raised some good questions about the whole notion of gated, subscriber-only crosstabs, I want to make clear that no one at Rasmussen complained to us about this issue. We discovered it ourselves and subsequently reached out to apologize, an apology I repeat publicly today. Except for the erroneously labeled data which we have already taken down, Scott Rasmussen has kindly granted us permission to leave the remaining data in place.
In correcting our error, however, it is now clear that two of our charts -- those tracking current and retrospective assessments of the economy -- will no longer "work" as intended. Virtually all of the data going forward would be coming from the Gallup Daily tracking and, as such, our chart would add no real value to those that Gallup publishes itself (here and here). We may rework our charts using only monthly values at some point in the future, but if we do, those charts will be based on monthly releases from other organizations, and from Gallup or Rasmussen should they ever opt to put monthly summaries into the public domain.
By Mark Blumenthal on October 6, 2009 3:52 PM
| Permalink
| Comments (1)
| TrackBacks (0)
October 5, 2009
By Mark Blumenthal
My column this week looks at the evolution of impressions of the Medicare prescription drug benefit among seniors in the years following its passage as a way to anticipate how public opinion toward health care reform might evolve should Congress pass a bill this year. Seniors were initially skeptical of the new benefit, and struggled with its complexity during the initial enrollment period, but grew much more positive as they started to experience the program's full benefits. The big lesson from that experience is that the timing of the initial benefits or initial costs will matter a great deal.
After filing the column on Friday, my colleague Ron Brownstein alerted me to a portion of panel discussion he moderated that covered this topic. The panel was hosted two weeks ago by the Bipartisan Policy Center. In the second video segment, a question comes up at about the 12:00 minute mark i(transcription that follows is mine):
Q: Assuming something does pass... next year when the Democrats are up for reelection, what the things they can point to in this bill that are will mean something to people since so much of the discussion is about things that will take years to accomplish?
Ron Pollack (Executive Director, Families USA): That's a critical question because as you correctly indicated, 2013 is essentially the implementation date for so many of these things. I was telling an anecdote before we got up here. My sister-in-law called last night. She's moved and she's trying to purchase individual coverage and because she's had a health problem, she's tried to get coverage on the individual market and because she's had some health problems, she's been turned down. So she said, "I'm really rooting for health care reform to happen so I can get my coverage this year," and I said, "sorry, it's not going to happen." I think the legislation will pass, but so much of health reform is not going to be implemented until 2013.
Note: The current draft of the Senate Finance Committee bill includes a provision to create a high-risk pool in 2010 to help provide coverage to those denied it due to pre-existing conditions. Pollack continues:
One of [the] things we've counseled to folks on the Hill and the White House is you've got to line up a number of things that are concrete that people can actually feel and touch with respect to what happens between now and 2010 and its difficult to do that because of the fiscal constraints. You can't put a whole new regimen that's going to spend a lot of money, but there are going to be, I think, a number of things that are going to be improvements. Take one example: Billy [Tauzin] and PhRMA worked out arrangements in terms of helping seniors with respect to the [Medicare drug benefit] donut hole. At least they're going to get discounts for brand name drugs that will subsidize fifty percent of the cost in the donut hole. That's a significant thing that I think is going to be very helpful to seniors.
I think there will be some efforts at insurance market reforms such as prohibiting rescissions of policies. I think you're going to see, as this bill goes to conference, much greater attention given to what can we do that's implemented in 2010 so that when people go to the polls in 2010 then can feel some concrete benefit.
John Rother (Executive Vice President, Policy and Strategy, AARP): All of us know, after the prescription drug benefit was enacted, which was by the way an exact parallel, politically, to this debate except for the parties switched, we did enact a discount card that people could use immediately before the full program went into effect precisely to offer people something tangible of benefit before they could realize the full program. So I'm hoping we can do something similar.
Billy Tauzin (President and CEO, PhRMA): That was a six hundred dollar per senior benefit, twelve hundred dollars in a couple, that was available immediately, and not everybody took advantage of that. I think it was about an eighteen percent take, but it was there immediately to help people in the interim before implementation. We're going to need probably to look at that as we get through this process. Where are those points were we can do something immediately.
Of course, that initial benefit did not eliminate seniors' suspicions about the program. More had unfavorable impressions than favorable on the Kaiser Family Foundation tracking until enrollment in the full program ramped up in late 2006. See the column for full details.
By Mark Blumenthal on October 5, 2009 9:29 AM
| Permalink
| Comments (1)
| TrackBacks (0)
October 4, 2009
By Mark Blumenthal
The New York Times' Shaila Dewan reviews the Strategic Vision story, confirms that the Atlanta Journal Constitution never received "supporting documentation" for their polls after repeated requests.
The National Council on Public Polls issues its own call for full disclosure: "A refusal to disclose can ignite the suspicion that there is something to hide."
Joshua Tucker calls on Rasmussen to free its cross-tabs.
The Pew Research Center finds slipping support for abortion; Jon Cohen and Gary Langer see no such evidence on other polls; John Sides has Deja Vu.
Ezra Klein and Brenden Nyhan note that Republican Party favorable ratings are much lower now than in 1994; Ed Kilgore amplifies.
Jennifer Agiesta looks at support for the Public Option in the 13 states represented by Finance Committee Senators that opposed it; Alan Reifman reacts.
Tom Jensen shares a word on party ID on their Virginia survey.
Glen Bolger sees an opening in Obama's weak approval ratings on top issue priorities.
Doug Schoen says Americans want entrepreneurship-friendly policies.
Resurgent Republic summarizes how the cost of health care reform cuts into its support.
Eric Kleefeld thinks polling gives Obama a reason to smile, also sums up the polling in NJ and VA.
Tom Edsall examines the income divide among Democrats.
Andrew Gelman assesses Norman Podhoretz' Why Are Jews Liberal.
Chris Bowers requests tabs showing Sestak leading Specter among PA voters who know both.
Another RNC survey earns the "push poll" label (via Smith).
The Associated Press reports on call center Western Wats employing children as young as 13.
By Mark Blumenthal on October 4, 2009 10:03 AM
| Permalink
| Comments (1)
| TrackBacks (0)
October 2, 2009
By Mark Blumenthal
Andrew Sullivan calls this the chart of the day. I'll second that:
It was posted a few days ago by David Leonhardt of the New York Times and is comes from a survey of registered voters conducted by Democratic pollster Geoff Garin for the Economic Policy Institute and it captures in a single image the profound challenge now facing Barack Obama and the Democrats.
Voters appear to be aware that the federal government is spending at unprecedented levels, but they believe the benefits are accruing mostly to large banks, Wall Street and manufacturing firms (which I'd wager really means General Motors), but not to average people. Not to "me."
Meanwhile, the Garin study shows that 85% believe the country is still in recession, nearly all see unemployment as a very big (59%) or somewhat big (24%) problem, and 58% say that someone they know, someone in their household or family or a close friend, has been laid off or has lost their job.
The survey also includes tests of Democratic-friendly messages on the stimulus and the economy, and those interested in Garin's strategic advice will want to review his presentation. But Democrats should be sobered by these results. Given yesterday's news of unemployment hitting 9.8% (with 17% unemployed, underemployed or no longer looking) expect the next twitch of the Obama job approval trend line to be down.
Related: Gary Langer has more on "what an ugly time it is for the American workforce," including the reminder that the ABC/Post poll found 47% of Americans reporting that a job loss or pay cut in their household in the last year. His conclusion:
[E]conomists suggest the downturn's bottomed out and the recession may be over. By the classic metrics, OK. it can take months, even years, for rising GDP to make itself felt at the kitchen table. For many Americans, as today's data show, economic realities remain dire - with potential political, policy and public health impacts yet to unfold.
By Mark Blumenthal on October 2, 2009 4:39 PM
| Permalink
| Comments (2)
| TrackBacks (0)
October 1, 2009
By Mark Blumenthal
The crux of the reprimand issued last week by the American Association for Public Opinion Research (AAPOR) against Strategic Vision, LLC is that the pollster failed to provide "any information about response rates, weighting, or estimating procedures." But if you look closely at the materials posted online in connection with AAPOR's "Ad Hoc" investigation of the primary polling mishaps of 2008, you will see several other pollsters for whom no response rate or weighting information is available. So why did AAPOR single out Strategic Vision? And why isn't AAPOR itself more transparent about the identify of the person that filed the complaint or about their communication with Strategic Vision's CEO, David Johnson? Let's take a closer look.
As of this writing, AAPOR has published information disclosed by pollsters in response to the requests of its primary polling investigation in two ways. Their final report, released this past April, summarizes the information that had been disclosed at the time (see especially Tables 4, 5, 7, 9 and 18). In partnership with the Roper Center, AAPOR has also created an online archive that includes responses and data received from pollsters, as well as many of their initial public reports. Some of these responses on the Roper site were received since they wrote the report.
If you take the time to sift through the various documents, you will still find (as of this writing) no responses on weighting procedures from three organizations: Strategic Vision, Clemson University and Ebony/Jet. Response rate information is still missing for those three plus two more, LA Times and Rasmussen Reports. Both response rates and weighting information are among the "minimal disclosure" items that the AAPOR code mandates that all pollsters disclose. So why did AAPOR single out Strategic Vision for public condemnation and not any of the others?
I put that question to AAPOR and received a two-part answer from standards chair Stephen Blumberg. First, the Roper/AAPOR archive does not include all of the latest information:
We recognize that there may be discrepancies between the ad hoc committee report, the information on the Roper Center site, and the information available to the ad hoc committee. Some information that was received after the ad hoc committee report was finalized has not yet been posted. More information will be posted soon to update the Roper Center site.
Second, while some organizations were apparently unable to provide all the the information requested, they apparently convinced AAPOR that they had made a good faith effort to disclose whatever information they had retained or otherwise had available:
Several organizations provided responses indicating that they did not produce, obtain, or retain sufficient information to provide the methodological information listed in the AAPOR Code and requested by the ad hoc Committee. Hence, it was not always possible for each organization to provide equally detailed information.
So why was Strategic Vision singled out for public reprimand?
Strategic Vision LLC, however, was the only polling firm that explicitly refused to provide such information in response to multiple requests. Strategic Vision LLC never indicated that such information was not produced, obtained, or retained.
Blumberg also expanded on why AAPOR is not commenting on the actions of other pollsters or disclosing the identify of the person that filed the initial complaint against Strategic Vision:
Regarding any judgments that may have been made during an AAPOR Standards Investigation of the adequacy of disclosure for any organization, you are aware (as an active AAPOR member and former Council member) that the confidentiality provisions in our Procedures do not permit AAPOR to comment. We cannot reveal whether complaints were filed, evaluation committees were formed, judgments were made, or actions other than public censure were taken.
[And yes, interests disclosed once again: I am an active AAPOR member and served as a member of its Executive Council from 2006 to 2008].
Blumberg's reference to confidentiality raises an objection voiced frequently by Strategic Vision CEO David Johnson in response to the AAPOR action. "We've asked for a copy of the complaint that was filed against us, and who filed it," Johnson told Jim Galloway of the Atlanta Journal Constitution. "How can you respond to something when you don't know who filed the complaint." He also told the website Research that he "find[s] it unusual that an organisation that says they are all about transparency won't supply us with details of the complaint. What they were asking for were trade secrets."
AAPOR's refusal to name the person that filed the complaint is, as Blumberg says, consistent with its extensive "Schedule of Procedures for Code Violations" that includes numerous safeguards to "maintain confidentiality of the subject(s), information sources, and methods of investigation." Why the lack of transparency?
A good clue to the answer can be found in Sidney Hollander's chapter of the official AAPOR history that is posted on the organization's website. Ironically, emphasis on anonymity and confidentiality was partly a reaction to the concern about potential lawsuits and legal liability of the sort that David Johnson is now threatening. Hollander writes (p. 76):
In early 1974, some Council members began exploring what legal liability the organization might incur if it were to adopt stronger measures. The legal advice obtained recommended explicit procedures that could be applied uniformly as a means of minimizing the possibility of retaliation by liability suits.
Hollander does not address the issue, but it seems likely that the authors of those procedures wanted to protect against those who might try to use their process to promote frivolous or unfounded complaints. So they set up a procedure to carefully evaluate and investigate reported violations of their code before making any comment.
The chapter also reports that complaints about unidentified complainants are not new. He cites a 1973 complaint against a polling organization that (p 76),
declined to respond to the complaint without knowing the identity of the complainants. Anonymity of the complaint's source was an issue that has been continually debated as the Code developed. Although Council member Cisin said that the concealing complainants' identities make the Standards Committee party to a 'security action,' the Standards Committee took the position that once a claim is accepted, the committee itself becomes the plaintiff in criminal law. (p. 76)
And what about another of David Johnson's complaints: Why would AAPOR expect a non-member to conform to its rules? That issue, Hollander writes, was also the subject of internal debate from the very beginning. He writes that in 1964, an argument in favor of acting on complaints against non-members was that AAPOR members had shown "overwhelming support for action against pseudo-surveys, for instance, and other violations by non-members that threatened to impair interviewer access to respondents" (p. 74). As a professional organization, AAPOR has always been concerned about unethical actions that threaten the image of their profession.
They resolved the debate, Hollander writes, by setting up rules that would hold practitioners "responsible for their work" through disclosure requirements, but not proscribe specific standards or best practices for how survey research should be conducted. And that brings us back to the Strategic vision reprimand.
What is striking about AAPOR's action last week, especially in light of the responses of other organizations, is that other pollsters that fell short of full disclosure were not the subject of public reprimand. For example, the AAPOR Code says researchers should release response rates with their public reports, but as far as I know, only 1 of 21 pollsters disclosed a response rate at the time their surveys were released in 2008. However, the other organizations either released response rate information on request or responded in good faith about the information they could and could not "produce, obtain, or retain." AAPOR singled out Strategic Vision because it was, they say, the only organization that flat out refused to answer even cursory questions about its response rates and weighting procedures. It was the only organization, in effect, that refused to take responsibility for its work.
By Mark Blumenthal on October 1, 2009 1:47 PM
| Permalink
| Comments (0)
| TrackBacks (0)
September 30, 2009
By Mark Blumenthal
David Yeager and Jon Krosnick respond to critics on their study of opt-in Internet panel surveys.
The Kaiser Family Foundation finds increased support for health reform in September; Charles Blow, Ezra Klein, Katie Connolly and Bruce Drake comment.
Margie Connelly notes a decline in support for universal health insurance.
Ande Coller reviews the "dizzying deluge of hard-to-reconcile" health reform polls.
Andy Barr assesses the leveling out of Barack Obama's approval ratings.
Ben Smith shares the AJC's new survey of American Jews.
Jennifer Agiesta examines the sharp issue divide between NoVA and the rest of Virginia.
Carrie Dann reports on Republican Patrick McHenry's efforts to promote the Census.
David Hill sees growing dealignment and desire for a third party.
Mark Mellman urges Democrats to work hard, not worry about 2010.
Alex Bratty says independents have buyer's remorse about Obama.
Alan Abramowitz thinks a repeat of 1994 in 2010 is unlikely.
Steve Singiser sees no decline in Obama's favorable since the campaign.
Ruy Teixeira reviews NBC/WSJ data showing greater support for health reform.
Nate Silver considers the connection between interest in news and response bias.
Tom Jensen finds a way to call Rasmussen an "East Coast, left-wing Democratic smear group."
Taegan Goddard catches a Bill Clinton quote pollsters will want to save.
By Mark Blumenthal on September 30, 2009 7:41 AM
| Permalink
| Comments (0)
| TrackBacks (0)
September 29, 2009
By Mark Blumenthal
Regular readers will know that I'm a fan of the growing use of "word cloud" graphics to depict the results of open-ended questions. As such, it's a pleasure to pass along some interesting new examples posted today by Patrick Murray, the director of the Monmouth University Polling Institute. The examples below shows the word or phrase that "mainly unaffiliated voters" mention when asked about the candidates for Governor in New Jersey:
Jon Corzine:
Chris Christie:
Do click through for full-size versions as well as the word clouds for independent Chris Daggett and for the image or message that voters say stand out from the campaign ads they have seen.
Another intriguing aspect of this project is that it comes from "an online panel study with mainly unaffiliated voters" that the Murray says Monmouth is conducting "as part of our polling coverage." I emailed him to ask for more details. Here is his response:
We expected that the New Jersey Governor's race would be volatile and so it would be interesting to supplement our standard trend telephone surveys with a panel to track individual level changes over the final six weeks of the campaign. We also wanted to experiment with some ideas that can't be easily done in a telephone poll (i.e. reactions to visual images). That is why we are not releasing marginals (i.e. full percentages). We'll continue to be conducting our standard telephone polls for representative sampling purposes.
Approximately 1/3rd of the panel respondents were recruited from prior Monmouth University telephone poll respondents who indicated an interest in participating in online surveys. The remainder came from a purchased random voter list (of which 60% were "unaffiliated", meaning they are registered with no party - although that does not necessarily mean they are politically independent). Our target was to arrive at a fairly even partisan distribution and we assumed that unaffiliated/independents would be less likely to participate. Self-identified party affiliation among the panel's Wave 1 participants was 29% Democrat, 31% Republican, 41% independent (i.e. a little less D and a little more R than a representative New Jersey voter poll). The panel is also somewhat more male and Caucasian than our representative voter polls.
We found that the demographic skews in the panel have little effect on the relative frequency of the words mentioned for each candidate, which is what the clouds illustrate.
The subject of online panels and their use is source of great controversy, but this experiment is indicative of the way pollsters trained in traditional methods are starting to experiment with online techniques: They are cautious and careful, but also eager to exploit the opportunities created by the online mode.
By Mark Blumenthal on September 29, 2009 3:38 PM
| Permalink
| Comments (0)
| TrackBacks (0)
By Mark Blumenthal
Over the weekend, our friend Chuck Todd shared this bit of poll skepticism via Twitter:
Why I'm always skeptical of Rasmussen robo-polls: it's results like this: NO WAY this guy Bob Krause has name I.D. of 63% in Iowa SEN.
I know little about Bob Krause except that he has not held public office since serving in the Iowa state representative in the 1970s and running unsuccessfully for Iowa Treasurer in 1978. As such, handicappers assume he begins his campaign with little true name recognition statewide. So how could Rasmussen show 63% of Iowa likely voters able to rate him?
Well, they start by asking a favorable rating question that offers no neutral category and that lacks an explicit prompt to say that the name is unknown or unfamiliar. Here is the text:
I'm going to read you a short list of people in the News. For each, please let me know if you have a very favorable, somewhat favorable, somewhat unfavorable, or very unfavorable impression.
According to Scott Rasmussen, their script then reads each name, with a prompts that "If 'very favorable, press 1; if somewhat favorable, press 2" and so on. It concludes with, "If you are not sure, press 5."
Next, they ask favorable ratings after asking the vote preference question, Rasmussen's standard practice. Doing so tells respondents two things about Krause they probably didn't know: He is a Democratic candidate for Senate and the presumed opponent of Republican Senator Charles Grassley in 2010.
Now combine all of that with the operating theory that survey methodologists use to explain how inattentive respondents answer pollsters' questions: They "satisfice," which is shorthand to say that they work just hard enough to provide a satisfactory answer. So suppose you're responding to the poll, you really not sure who Bob Krause is, but the pollster (a) does tell you that "never heard of him" is an acceptable answer and (b) has already told you that Krause is the Democratic candidate for Senate. Under those circumstances, you might say "somewhat" favorable or unfavorable (reasoning, "well, I've never heard of him, but he's a Democrat, so I'll say I'm just somewhat [favorable or unfavorable]"). That answer is especially likely if you're in a hurry and not in a mood to wait for the automated system to additional answers after those listed in the introduction of the question.
Look at the the Rasmussen results, and you see a pattern consistent with that theory: Only 13% express a "strongly favorable" (5%) or "strongly unfavorable" (8%) opinion of Krause, and I'd wager that most of those were strong partisans reacting mostly to Krause's Democratic affiliation. Most of the rest with an opinion said it was either "somewhat favorable" (28%) or "somewhat unfavorable" (22%). And of course, more than a third (37%) did hang on long enough to choose "not sure."
Now, is it possible that aside from the way respondents answer questions, automated polls attract a different kind of respondent that is better informed about politics? Yes, that's possible, but not nearly proven by this example. Most of what makes this result seem so improbable to Chuck Todd and the other political pros that he talks to can be explained by the combination of question text and order. I share Chuck's skepticism with Rasmussen's favorable ratings, but necessarily of their underlying sample or their vote preference results.
P.S. A related issue: We discovered over the last few days that what we believed to be a weekly update of Barack Obama's favorable rating on RasmussenReports.com had been changed at some point to a weekly job approval rating, even though the labels still read "favorable" and "unfavorable" rather than "approve" and "disapprove" up until about a week ago. As such, we have removed the non-favorable ratings from our national Barack Obama favorable rating chart.
By Mark Blumenthal on September 29, 2009 2:00 PM
| Permalink
| Comments (3)
| TrackBacks (0)
September 28, 2009
By Mark Blumenthal
My National Journal column for the week is now posted. Filed on Friday, it reviews the reprimand of polling firm Strategic Vision, LLC issued last week by the American Association for Public Opinion Research and the ensuing controversy sparked by Nate Silver's allegations that firm's polls "exhibit unusual patterns" that ""suggest, perhaps strongly, the possibility of fraud."
For further reading: I first reacted to Nate Silver's first "trailing digit" analysis on Friday and reviewed some of the contradictions and odd facts surrounding Johnson's reaction to AAPOR, the absence of Strategic Vision cross-tabs and the difficulty of locating a physical Strategic Vision office on Saturday. Over the weekend, Nate Silver did some additional analysis using Quinnipiac surveys as a control, raised more questions about a Strategic Vision survey of students in Oklahoma and tracked down** what appears to be a Strategic Vision office in rural Georgia. Meanwhile, Strategic Vision CEO David Johnson promises cross-tabs to the Atlanta Journal Constitution's Jim Galloway once again and tells the St. Petersburg Times that he plans to sue Nate Silver. Also, both Politico's Ben Smith and The HIll's Aaron Blake did roundups with reaction from Johnson.
**Update: Though he gave due credit to commenters both here and on FiveThirtyEight. I can say I heard it first, here, from socio-logic.
By Mark Blumenthal on September 28, 2009 8:30 AM
| Permalink
| Comments (7)
| TrackBacks (0)
September 26, 2009
By Mark Blumenthal
Troubling new details continue to emerge about Strategic Vision following their reprimand from AAPOR for a lack of methodological disclosure earlier this week. We ought to take great care before making allegations of outright fraud, but there is now enough conflicting information -- including lack of evidence of any physical Strategic Vision office -- that the burden of proof has shifted to Johnson. His obligation is not to AAPOR but to the general public. Transparency is the first crucial ingredient that allows us to determine whether their polls, or any other, deserve our trust.
I want to set aside Nate Silver's trailing digit analysis aside for the moment (beyond my comments yesterday), as I understand that he is working on further analyses that respond to some of the questions raised by his commenters. Instead, I want to focus on the conflicting statements of Strategic Vision CEO David Johnson and other details reported by Politico's Ben Smith and others yesterday.
1) Johnson and AAPOR - Johnson told Smith that "he was refusing to cooperate with AAPOR" because the organization refuses to tell him the identity the person that filed the complaint:
"What we have asked from the very beginning was we would share all the methodology we wanted, - we wanted a copy of who filed the complaint," he said. "If they want transparency there has to be full transparency."
The problem with that statement is the phrase "from the beginning." AAPOR made multiple requests for methodological information from Strategic Vision -- and from 20 other polling organizations that did surveys in four primary states in 2008 -- starting in March 2008. These included two letters sent by Federal Express to the primary Strategic Vision mailing address listed on their web site. But these had nothing to do with any "complaint," and Johnson and Strategic Vision ignored them all.
Johnson's claim that he is "refusing to cooperate" with AAPOR contradicts two statements he made earlier in the week saying that he had cooperated fully. The journal Research reported that Johnson "said the firm had supplied AAPOR with all the information it had requested on 19 June this year." He also told ABC's Gary Langer that "I'm a little confused because we provided them the information on June 19."
AAPOR's press release this week is consistent with the more detailed statement given to ABC's Gary Langer by AAPOR Standards Chair Stephen Blumberg: After receiving a request for information in March 2009, Johnson finally "responded with an explicit refusal to provide the requested information," then after "in response to notification of AAPOR's initial findings of a violation, Mr. Johnson provided some, but not all, of the information requested." At that point, he stopped responding to their queries.
This whole episode is both puzzling and troubling. No, pollsters were not generally as cooperative with the AAPOR investigation last year as some of us had hoped. But Strategic Vision's bizarre stonewalling of AAPOR's requests this year, last year and (and of mine in 2007) were unusual. None of it makes much sense.
2) Where are The Cross-tabs? - Ben Smith writes: "Details of Strategic Vision's polls have long raised flags among pollsters, in part because it refuses -- unlike other pollsters -- to release "cross-tabs" -- the detailed demographic breakdowns of individual polls."
Strategic Vision is not the only pollster that fails to regularly post cross-tabular tables on its web site the way SurveyUSA, PPP and others do. Some hold back such tables for paying subscribers. Others prefer to report subgroup results selectively. However, Strategic Vision is the only pollster that, as far as I know, has refused to release cross-tabulations of its political surveys to anyone, including reputable journalists.
Three years ago (3/31/2006), for example, Johnson promised my colleagues at The Hotline that he would "honor requests for crosstabs and will make them available online in 4/06, when their website is revamped to handle the files." No such files ever appeared on the Strategic Vision website.
A helpful reader also alerts me to requests for cross-tabs made to Johnson by Jim Galloway of the Atlanta Journal Constitution in March 2006 and twice (here and here) in 2007, but finds no evidence that Galloway ever received any of the promised cross-tabs.
3) Where's the Office? - Some alert commenters on FiveThirtyEight discovered something that Ben Smith also reported: "[Strategic Vision's] website, as recently as last month, listed offices in Atlanta, Madison, Seattle, and Tallahassee -- all of which match the locations of UPS stores, rather than actual offices."
And to underline the point, the Atlanta mailing address (2451 Cumberland Parkway SE, Suite 3607) that, as of this hour, remains the only address on the Strategic Vision contact page is also a post office box at a UPS Store. That was the same address to which AAPOR sent its FedEx requests during 2008. I called to the UPS Store to confirm, and Suite 3607 is a mailing address that they maintain.
FiveThirtyEight reader inferno asks a good question: "[D]o they have any sort of actual physical address? i.e., an office?" If there is, its awfully hard to find any record of it.
The annual business registration for Strategic Vision filed with the Georgia Secretary of State indicates that their most recent filing -- which, notably, appears to be "active" but in "noncompliance" -- also lists the Cumberland Parkway address. The only previous address for the firm in a prior filing (2002) lists a suite in a Peachtree Street office tower that, according to a Google search, now appears to be occupied by a law office.
4) What Does It Cost? - Ben Smith gets to the crux of the issue: "Another question is how the firm pays for its polls. Its website lists at least 172 public polls, and at a stated cost of $30,000 a poll, that's an expenditure of more than $5 million -- quite a sum for a small firm."
The "stated" $30,000 cost comes from a 2006 interview Johnson gave to then Hotline polling editor Aoife McCarthy (3/31/2006). She wrote:
So how much does all of this cost? Strategic Vision uses these political polls as marketing tool for the company. Johnson says each poll costs an average of $30K to conduct. Do the math -- 39 polls in '05 and 10 polls so far in '06 at the bargain price of $30K results in nearly $1.5M in just 15 months. That does not include the first year of polling in '04.
Would it really cost Strategic Vision $30,000 to conduct surveys that typically include 800-1000 interviews? I doubt it. Not if they are really the "10-question-per-state-polls" that Johnson claimed to the Hotline in 2004 (roughly the length of most polls on their website). Not if, as Johnson reports in the 2006 Hotline interview that "callers are paid directly by Strategic Vision." I'll save the specific numbers for another post (if anyone is interested), but I have a hard time figuring out how his costs could be much higher than $5 per interview (though that still would amount to nearly a million dollars since 2004).
Why would he cite such a big number? I haven't a clue, but given all the other issues now swirling, it is a question David Johnson needs to answer. In 2004 he told the Hotline that "no client is paying," but he told Smith yesterday that some of their surveys "are piggybacked onto other polls." So which is it?
So my bottom line: I have no idea whether Nate Silver's insinuations of fraud are real, but the burden of proof is shifting. Strategic Vision has to become considerably more transparent about their methods and data, or we will have little choice but to reach an ugly conclusion.
But there is a much bigger problem here for the rest of us. The larger issue is not whether Strategic Vision may be fabricating numbers or whether another less sensational explanation exists for all this obfuscation and contradiction. The problem is that anyone could theoretically make up a set of numbers and -- without a lot more transparency about methods and data than we now typically see -- pass it off as a real poll. The way those of us in the "new media" consume polling releases from every conceivable source, and that certainly includes Pollster.com, makes that possibility all too real. PPP's Tom Jensen has this exactly right:
I could leave PPP, start Tom Jensen Polling, put out a bunch of topline numbers the day before an election that just copied the Pollster, RCP, or Nate Silver predictions and be one of the most accurate pollsters in the country. That would be pretty darn easy and anyone could do it. And that's why public pollsters should hold themselves to a higher standard and also be held to a higher standard by the media.
That's perhaps the most extreme reason why, last month, I argued for a system of scoring pollsters for the quality of their disclosure and posting those scores online, as a matter of routine, alongside polling results. This episode makes the need for such a system more clear and more urgent than ever.
So who's with me?
By Mark Blumenthal on September 26, 2009 2:28 PM
| Permalink
| Comments (6)
| TrackBacks (0)
September 25, 2009
By Mark Blumenthal
The Associated Press reports that the part-time census worker Bill Sparkman was found hanging, gagged and bound with his Census ID taped to his chest but said "investigators have not determined if it was a homicide, suicide or an accident."
Zach Roth confirms that Sparkman had the word "Fed" scrawled on his chest in felt tip pen.
David Johnson answers Ben Smith's questions about Strategic Vision
Tom Jensen offers thoughts on Strategic Vision and crosstabs.
Gallup sees no change in Obama's job approval rating in September.
John Sides reminds us how many Americans tune out from politics.
Anthony Salvanto and Mark Gersh ask if 2010 will be another 1994.
Andrew Gelman sees a potential GOP House takeover in 2010.
Ed Kilgore wonders if a funny thing could happen on the way to the GOP 2010 landslide.
Chris Bowers sees Democrats regaining a foothold on the national generic congressional ballot.
Gary Langer models seniors concerns about Medicare and health reform.
Alan Reifman explores regional differences in health reform attitudes.
Health Care for American Now commissions a poll on the public option in conservative swing House districts.
Resurgent Republic finds support for its focus group findings in the NBC/WSJ poll.
Seth Masket explores the hard-to-disentangle relationship between racial resentment, party identification and opposition to Obama (via Sides).
James Vega explores the use of polling to track "delusional thinking" in politics.
And totally unrelated to polling or statistics: President Obama blames the media for encouraging "outliers in behavior"
By Mark Blumenthal on September 25, 2009 11:31 PM
| Permalink
| Comments (3)
| TrackBacks (0)
By Mark Blumenthal
The Strategic Vision story is getting far more interesting. In the wake of a public reprimand from the American Association for Public Opinion Research (AAPOR) for failing to disclose "essential facts" about his company's methods, and after more than a year of doing his best to avoid public comment, Strategic Vision CEO David Johnson now has much to say and is threatening legal action. Meanwhile, over at FiveThirtyEight, Nate Silver says he has found evidence that "suggests, perhaps strongly, the possibility of fraud" in Strategic Vision's numbers.
On Wednesday, he provided this response to a call from Jim Galloway at the Atlanta Journal Constitution:
Strategic Vision CEO David Johnson said his firm had wanted to appeal the judgment, and said a Sept. 17 hearing had been scheduled - and then canceled by the AAPOR. "We've asked for a copy of the complaint that was filed against us, and who filed it," Johnson said. "How can you respond to something when you don't know who filed the complaint."
Moreover, he added, "We're not a member of their organization. I don't know anything about them."
Johnson also gave Galloway an email sent by AAPOR to Johnson this past June acknowledging receipt of "some of the information requested regarding polls in New Hampshire and Wisconsin" (emphasis added).
Yesterday, James Verrinder of the website Research reports that Johnson now "vows legal action" against AAPOR and some of its members:
Johnson said he "disagreed completely" with the charge levied at his firm by AAPOR and vowed to take legal action against the association. He said the firm had supplied AAPOR with all the information it had requested on 19 June this year, and had electronic proof of what was sent.
Johnson believes a competitor is behind the original complaint to AAPOR, and wants to see the source of the action against his firm. "I find it unusual," he said, "that an organisation that says they are all about transparency won't supply us with details of the complaint. What they were asking for were trade secrets."
He said: "We will be taking legal action. We have spoken with our attorneys and have gotten them the documentation and should know exactly the venue and specific charges that we will be filing against AAPOR specifically and individual members of AAPOR personally."
Johnson alleges that the AAPOR's acted "maliciously" in issuing its ruling. "I think it was timed to coincide with the results of a poll we had out yesterday [on the gubernatorial elections in Georgia]," he said.
Both accounts also include responses from AAPOR President Peter Miller. Miller told Galloway that "AAPOR had sent Johnson notices four times asking him to confirm his attendance at that hearing last week, and finally ended up canceling because of the lack of any response." Miller told Verrinder that it was "completely wrong" that a competitor had filed the complaint and reiterated that Johnson's 2009 response, received after the release of the AAPOR report in April, did not include all of the requested information. In a previous article, also published today, Verrinder had reported that the reply still did not provide requested information about response rates and weighting or estimating procedures.
Now separately, Nate Silver claims to have found evidence of a non-random pattern in the trailing digits of the percentages reported by Strategic Vision in their public polling since 2005, and the implications of that assertion are pretty explosive. Last night, he raised some suggestive questions that mirror some of the unsubstantiated gossip and prodding I've received via email for years from a Democratic activist or two in Georgia. But this morning, as Silver puts it himself, he's making a much more concrete allegation (emphasis his):
Certain statistical properties of the results reported by Strategic Vision, LLC suggest, perhaps strongly, the possibility of fraud, although they certainly do not prove it and further investigation will be required.
In other words, Nate is suggesting that Strategic Vision has been making up its numbers. The analysis he reports this morning is based on Benford's Law, the same principle similar to the principles behind much the analysis of Iran election fraud that we reported this summer [see the clarification below]. The idea is the last digit of numbers with two or more digits should have a uniform distribution. A '1' should occur as often as a '2,' a '3' etc. According to Nate, the pattern for Strategic Vision is far from uniform:
[T]his data is not random. It's not close to random. It's not close to close. Which brings up the other possibility: Strategic Vision is cooking the books. And whoever is doing so is doing a pretty sloppy job. They'd seem to have a strong, unconscious preference for numbers ending in '7', for instance, as opposed to those ending in '6'. They tend to go with round numbers that end in '5' or '0' slightly too often. And they much prefer numbers with high trailing digits like 49 and 38 to those with low ones like 51 and 42.
I haven't really seen anyone approach polling data like this before, and I certainly haven't done so myself. So, we cannot rule out the possibility that there is some mathematical rationale for this that I haven't thought of. But it looks really, really bad. There is a substantial possibility -- far from a certainty -- that much of Strategic Vision's polling over the past several years has been forged.
I recognize the gravity of this claim. I've accused pollsters -- deservedly I think in most cases -- of all and sundry types of incompetence and bias. But that is all garden-variety stuff, as compared against the possibility that a prominent polling firm is making up numbers whole cloth.
I would emphasize, however, that at this stage, all of this represents circumstantial evidence. We are discussing a possibility. If we're keeping score, it's a possibility that I would never have thought to look into if Strategic Vision had been more professional about their disclosure standards. And if we're being frank, it's a possibility that might actually be a probability. But it's only that. A possibility. An hypothesis -- as yet unproven.
Predictably enough, my email box is filling with the same question: What do you think of this? My first reaction is similar to "Mark" (not me) and some other commenters on FiveThirtyEight: The analysis is intriguing, but I would find it far more convincing if he ran comparable statistics for some of the other prolific pollsters in the same contests since 2005 (Rasmussen, SurveyUSA, Quinnipiac, ARG, Zogby, Mason-Dixon, etc). If the Strategic Vision pattern is really different from all the rest, then it would reduce the possibility that the pattern Nate found "is a function of polling in general" (as commenter Matt puts it).
Also, while I stipulate that I am no expert in Benford's Law, my sense from reviewing the analysis of the Iranian election is that its assumptions can get extremely complex. As such, we need to very cautious about jumping to conclusions based on the pattern that Silver is reporting. That said, I will have more about Strategic Vision, AAPOR and the theme of transparency in my National Journal column on Monday.
Update and Clarification: To prove I'm no expert, I initially described Silver's analysis incorrectly. Mark Lindeman is right in the comment below when he says that Nate Silver is expecting a uniform distribution, not a Benford distribution.
I also exchanged email with Walter Mebane, the University of Michigan professor whose has done much work in this area, most recently on fraud in Iran. He reviewed Silver's post and urges caution, saying that some of the comments there (such as those from Mark, Allen and Zach) "cover the kinds of further questions" he would want to ask. Like Zach, Mebane says that with two-digit numbers, we should not expect a uniform distribution of the last digit, especially if it is based on
percentages that have been rounded in a biased manner. Echoing commenter Mark, he says that a "comparison with other polling houses would probably be the most informative and quickest thing to do."
Update 2: For those wondering whether Strategic Vision, LLC has any real clients, a colleague passes along some indisputable proof that they do. The Friedman Foundation for Educational Choice has used Strategic Vision to conduct a series of statewide surveys since 2007 (click on any link showing a state's "Opinion on K-12 Education and School Choice"). I count 14 surveys in all since 2007, the most recent released just last wewek.
Ironically, these reports show that the Friedman Foundation demonstrates a prominent commitment to "methods and transparency." Check Page 2 of the most recent report for Nebraska: "We are committed to sound research and to provide quality information in a transparent and efficient manner." A methodology section found within includes the sort of information -- including response rates and call disposition reports -- that Strategic Vision continues to resist releasing for their political surveys.
By Mark Blumenthal on September 25, 2009 12:25 PM
| Permalink
| Comments (11)
| TrackBacks (0)
September 24, 2009
By Mark Blumenthal
If you have been following the news on cable television or online this morning you have probably already heard about the hanging death of Bill Sparkman, a 51-year-old part time fieldworker for the U.S. Census in Kentucky. According to an Associated Press report, an unnamed law enforcement official said Sparkman's body was found with the word "fed" scrawled on his chest.
Very little about this story has been confirmed by official sources. "Investigators," the story tells us, "are still trying to determine whether the death was a killing or a suicide, and if a killing, whether the motive was related to [Sparkman's] government job or to anti-government sentiment." So we should take care to avoid jumping to conclusions about the circumstances surrounding Mr. Sparkman's death.
Still, let's remember that a Census "field worker" is really a survey interviewer. As noted this morning by the Washington Post's Ed O'Keefe, the Census hires "hundreds of thousands of temporary workers across the country [to] walk door-to-door conducting various demographic surveys." Their work is neither partisan nor political. It is vital to the accurate collection of Census Bureau statistics on the U.S. population and economy.
As my colleague David Hill pointed out a few months ago, accurate Census data has profound importance, not just to the functioning of government, but also to "entrepreneurship, marketing and business planning." He quoted a commercial broker named Howard Carr who once told the Albany Times:
The raw, dry, detailed facts and figures about people that the U.S. government collects every 10 years are the stuff real estate developers live by [and are used by businesses] for everything from determining how many health-conscious products to stock on supermarket shelves to deciding on which side of the street a day-care center should be built.
Last night, SurveyUSA CEO Jay Leve, whose company uses an automated methodology rather than live interviewers, took to the listserv of the American Association of Public Opinion Research with this message: "If Mr. Sparkman's hanging is related to his work for the Census Bureau -- and unclear yet that it is -- this is an attack on each of us, however we collect our data."
Amen to that.
By Mark Blumenthal on September 24, 2009 11:46 AM
| Permalink
| Comments (0)
| TrackBacks (0)
September 23, 2009
By Mark Blumenthal
As the final act of a yearlong investigation of the polling mishaps leading up to the New Hampshire and other primary elections last year, the American Association for Public Opinion Research (AAPOR) today criticized the firm Strategic Vision LLC for refusing to disclose "essential facts" about surveys it conducted prior to the 2008 New Hampshire and Wisconsin primaries. An AAPOR press release described Strategic Vision's nondisclosure as "inconsistent with the association's Code of Professional Ethics and Practices and contrary to basic principles of scientific research."
When AAPOR's special ad hoc committee released its report last April, over a year after the New Hampshire miscues it was created to investigate, it had still not received "minimal disclosure" from 3 of 21 firms that had received requests a year earlier (though Strategic Vision's evasions were notworthy and consistent with my own experience). Today's AAPOR release completes the story:
For more than one year, AAPOR was unable to obtain the following basic information about Strategic Vision LLC's polling in New Hampshire and Wisconsin: who sponsored the survey; who conducted it; a description of the underlying sampling frame; an accounting of how "likely voters" were identified and selected; response rates; and a description of any weighting or estimating procedures used. AAPOR considers the release of this information for public polls to be a minimum requirement for professional behavior among those who conduct public opinion research.
Following Strategic Vision LLC's failure to respond to AAPOR's inquiries, a complaint was filed alleging a violation of the association's Code of Professional Ethics and Practices. The investigation process included two notices of non-compliance to Mr. David Johnson, CEO of Strategic Vision LLC, who explicitly refused to provide the requested information. Later, after receiving notification of the association's initial findings of a violation, Mr. Johnson offered partial but incomplete information. AAPOR never received any information about response rates, weighting, or estimating procedures. The AAPOR Executive Council now concludes that the repeated noncompliance by Strategic Vision LLC was a violation of the AAPOR Code.
AAPOR's action comes with no penalty, since no one associated with Strategic Vision is an AAPOR member.
The release also notes that this action "concludes AAPOR's official evaluation" of the 2008 primary polling mishaps while also noting that Strategic Vision was the "only polling firm" that failed to meet its minimal disclosure standards in response to their committee's requests. That action implies that other firms have provided additional information since the release of the committee's report in April. As of this hour, however, the reports available on the Roper Center web page appear to include no new information on the South Carolina surveys by Clemson University and Ebony/Jet/Lester & Associates beyond what they publicly released in 2008 (both firms had been singled out in the April report, along with Strategic Vision, for failing to respond to committee requests).
So there you have it. Twenty months after announcing its intention to request data related to New Hampshire primary poll, six months after reporting that only 7 of 21 firms had gone beyond the minimal disclosure that AAPOR mandates for public release "in any report of research results," AAPOR today "raises objections" about the response of one firm.
To put it simply: The process of "on demand" disclosure backed by the sort of punitive sanction issued today is not working. As I wrote in August, there may be a better way.
[Interests disclosed: I'm an active AAPOR member and served on the AAPOR's Executive Council from 2006 to 2008].
Update: ABC's Gary Langer has more, including news that "AAPOR also said an updated and final version of its report on the pre-primary polls is now available." An AAPOR spokesperson tells me that Gary is in error and that the report has not been recently updated.
By Mark Blumenthal on September 23, 2009 12:24 PM
| Permalink
| Comments (0)
| TrackBacks (0)
By Mark Blumenthal
As noted in the 'outliers' post, MyDD's Jonathan Singer flagged an odd inconsistency in a recent Rasmussen Reports survey of Minnesota. They asked likely voters to rate Republican Governor Tim Pawlenty on their standard approval scale (strongly approve, somewhat disapprove, somewhat disapprove or strongly disapprove) but then asked about Senator Al Franken using a different scale ( excellent, good, fair or poor).
As Singer points out, the excellent-good-fair-poor scale typically produces lower positive scores. He links to a Pollster.com guest post from Chicago Tribune pollster Nick Panakagis showing that "fair" sounds like a neutral category that appears to attract voters that might otherwise answer "somewhat approve."
"Why," asks Singer, "is Rasmussen using two different metrics -- one, which tends to find higher approval ratings, for the Republican; another, which tends to find lower approval ratings, for the Democrat?"
I put that question to the folks at Rasmussen Reports and received this response from a spokesperson:
It was a mistake that slipped through the cracks. The matter has been addressed internally with all involved.
We work with a local TV station that provides us with local knowledge. In exchange, they get first look at the data and Scott Rasmussen goes on air to discuss the results. In practical terms, this means they suggest questions and topics that are likely to be of interest in their state.
They do not commission the poll, it's a Rasmussen Reports poll and we are ultimately responsible for the questions.
We work with a standard template that includes the President's Job Approval rating and the Governor's. In this case, the station suggested a variety of topics ranging from politics to the Vikings playoff chances and Bret Favre. The station recommended the questions on the Senators and did so with the excellent, good, fair, poor rating. The person preparing the script noted (correctly) that this was an acceptable format we have used before in other surveys. However, they should have noted the inconsistency with the other approval questions and asked all in the same way. The editor reviewing the process also failed to pick up the inconsistency.
Credit to Rasmussen Reports for admitting a mistake, but they should also append this statement to the original analysis and post it separately so that those who saw (and linked to) the original numbers will see their explanation.
By Mark Blumenthal on September 23, 2009 9:27 AM
| Permalink
| Comments (2)
| TrackBacks (0)
By Mark Blumenthal
John Judis reviews the strong relationship between trends in unemployment and presidential approval ratings.
Tom Holbrook says its too early to predict Democratic losses in 2010.
Josh Goodman attributes a closer VA race to delayed reaction to the thesis (via Singiser).
Sam Stein finds another RNC "poll" pushing dubious questions; Michael Scherer has more.
Nate Silver notes Grassley's falling poll numbers.
Jay Cost questions the popular support for health reform.
Kos finds positive reaction to the public option in blue dog districts down on Obama.
Andrew Gelman shares the trait that predicts lame VP nominees.
Lisa Valentine profiles working women and their resistance to Republicans.
Gene Ulm looks at the correlation between consumer confidence and votes for congress.
Jonathan Singer calls out different Rasmussen job approval ratings questions for Pawlenty and Franken.
Tom Jensen thinks Obama is "more or less in the same place he was on election day."
Mark Mellman runs the numbers on Afghanistan.
David Hill says Obama's school speech hurt his education allies and helped his enemies.
Stan Greenberg reviews Lynn Vavreck's, The Message Matters: The Economy and Presidential Campaigns.
Frank Luntz says voters aren't racist, just furious at Obama and Dems.
86% of New Jersey wants Springsteen to "keep rocking as long as he wants."
By Mark Blumenthal on September 23, 2009 7:31 AM
| Permalink
| Comments (0)
| TrackBacks (0)
