January 6, 2009
By Mark Blumenthal
Ever since Election Night this past November, I have been ruminating about a story that I've been meaning to post. Its connection to pollsters and poll methodology is indirect, but seems especially relevant right now as we formally turn the page from the incredible election year 2008 to a new administration and its travails in 2009. So before we get back into our usual routine, I want to share it.
In May 2002, I sat down to draft a questionnaire for a new client, a wealthy businessman who was then considering a challenge to an incumbent U.S. Senator two years hence. It was just six years ago, yet back in the days before my MysteryPollster blog, back when conducting such surveys for Democratic candidates was my full-time occupation. And, as it turned out, this client would be my last "major" race (a designation that, in my mind, falls somewhere between a contest with an uncertain outcome and one with some national significance).
The race that eventually played out was one that, from our campaign's perspective, struck some as conventional (and thus flawed) politics in microcosm. One of the consultants for one of our opponents called our campaign "an exercise in the technology of politics...They do polls and see what people want to hear and then they put ads on the air and run them again and again. There's nothing new about what they're doing." There was truth in this critique, and the ultimate demise of this campaign played a not insignificant role in my evolution from consultant to blogger.
But it was something that happened at the very beginning of that campaign that sticks in my memory now. On that day in May, I started with my usual process when drafting a benchmark "message testing" survey. I took all of my notes from initial discussions with the client, suggestions from other consultants and information gleaned from news clippings and the Internet and synthesized it into a single document listing all of the "messages" -- candidate profiles and arguments about them pro and con -- we hoped to test on the benchmark survey.
I found my "notes for questionnaire" for that project on hard drive yesterday, still there after moving with me through at least three different computers over the last six years. My first cut consisted of roughly 700 words. About half involved the incumbent Republican, about a third concerned our client. I had also typed a two sentence profile of the potential primary opponent, a former Democratic Senator that had been turned out of office a few years before.
Finally, the document also included the names -- just the names --of three potential candidates that most everyone considered extreme long-shots. The paucity of detail on these three spoke to their status as very long shots in this particular Senate race. Anyone looking over my notes right now, however, would immediately notice one now familiar name among the also-rans:
"Barack Obama."
The survey we fielded a few weeks later showed that just 4% of likely general election voters had a favorable impression of Obama, while 5% rated him unfavorably. Nearly four out of five (79%) had never heard of Obama. He did a little better among likely Democratic primary voters -- 11% favorable, 6% unfavorable -- but still ran far behind, winning just 6% of the vote, in a five-candidate primary matchup that also featured former Senator Carol Moseley Braun. Obama did a little better (rising to 12%) when we omitted Braun, but Obama still ran far behind state Comptroller Dan Hynes (with 34%).
Of course, my client, Chicago businessman Blair Hull, started with even less recognition and support (2% of the vote, to be precise), but his candidacy and its ultimate demise was another story altogether. The consultant dissmissed our campaign as "nothing new" was a truly forward looking guy named David Axelrod. But I digress.
It is still hard to believe that a State Senator who seemed like the longest of long shots for the U.S. Senate so recently is about to be sworn in as the 44th President of the United States. I look back at my notes from six years ago, ponder all that has happened since and just shake my head in wonder.
This story teaches many lessons, of course, but the most relevant to readers of this site is that no poll or statistical model could have predicted Obama's assent. Yes, we could see from the "internals" of that first survey that Obama had the potential to be a formidable contender in the 2004 Senate primary, especially after both Braun and incumbent Peter Fitzgerald announced that they would not run. But no survey or model in 2002 could have predicted Obama's 52.8% majority in the seven-candidate 2004 Senate primary or the nearly 70% of the vote he won in the general election, much less all that followed in bid for the White House in 2008. I'm certainly a believer in opinion surveys and statistical models, but they have their limits.
The Obama story from 2004 provides no direct corollary to the ongoing arguments about the "electability" of potential Senate candidates in Illinois, New York or elsewhere, but I do see a warning against leaping to conclusions about a candidate's prospects based only on early favorable ratings or horse-race numbers. "Electability" is an appropriate topic whenever a party selects its nominee (or appoints someone to serve out the term of a departing legislator), but quantifying electability through polling, or through predictive models derived from it, is a shaky enterprise at best. Obama's rise -- in the face of early data that led many to question his electoral potential in both 2004 and 2008 -- is a testament to such challenges (see also this pertinent example from September 2007).
This story also says something bigger about the potential for "change" within the American electoral system. Typically the day-to-day business of politics is mundane and static. We fight the same fights over and over, and little seems to change. Politics seems to be mostly about compromise, mostly "the art of the do-able." Yet every once in awhile, usually in the context of a presidential election, some out-of-the-blue candidacy reshapes our perception of what is possible. It shows us that sometimes, if we're lucky, we get to see history being made in the midst of mere electoral politics.
I look back to my own experience in the 2004 Senate election and realize that there was a silver lining in an unsatisfying campaign that amounted to, admittedly, little more than "an exercise in the technology of politics:" It offered me the opportunity to witness (and, yes, literally chart) from the very beginning one of the most meteoric success stories in American political history from the very beginning. And from this perch at Pollster.com, I got the chance to keep that ringside seat, charting and writing about the most exciting campaign of my lifetime.
So I want to thank Pollster.com's readers for coming along with us for the ride, and especially those who have stayed with us now that things have calmed own a bit. We hope you will stick around and let your friends know as we continue to track public opinion in 2009. You will be seeing a transition in Pollster.com over the next month or so, as we move the 2008 data off the front page and begin to feature charts and data that track the performance of the new administration and the concerns of American voters. But as we do, let's not lose sight of an idea we have tried to stress here from our first post: Using survey data well requires that we understand its limitations.
By Mark Blumenthal on January 6, 2009 4:17 PM
| Permalink
| Comments (1)
| TrackBacks (0)
January 6, 2009
By Eric Dienstfrey
Public Policy Polling (D)
1/3-4/09; 700 registered voters, 3.7% margin of error
Mode: IVR
New York State 2010 Senate
Tested: Attorney Gen. Andrew Cuomo (D), Caroline Kennedy (D), Rep. Peter King (R)
Cuomo 48, King 29
Kennedy 46, King 44
(source)
By Eric Dienstfrey on January 6, 2009 1:25 PM
| Permalink
| Comments (0)
| TrackBacks (0)
January 5, 2009
By Eric Dienstfrey
Public Policy Polling (D)
1/3-4/09; 700 registered voters, 3.7% margin of error
Mode: IVR
New York State
If the choices were Andrew Cuomo and Caroline Kennedy, who would you prefer Governor Paterson appoint to replace Hillary Clinton in the US Senate?
How has your opinion of Caroline Kennedy changed since she started publicly campaigning for appointment to Hillary Clinton's Senate seat?
23% More Favorable
44% Less Favorable
(source)
By Eric Dienstfrey on January 5, 2009 1:56 PM
| Permalink
| Comments (0)
| TrackBacks (0)
January 5, 2009
By Mark Blumenthal
Some odds and ends missed while I was away. Will be back in the swing once I clear out my email...
The Pew Internet & American Life Project releases a survey on voter engagement after the election (via Amy Sullivan).
Jennifer Agiesta finds new optimism among Democrats.
Brenden Nyhan sees no sign of the Obama honeymoon ending.
Gary Langer digs into how veterans voted in 2008.
Tom Jensen shares a tongue in cheek anecdote on response rates.
Chris Bowers considers public opinion among the Palestinians (PS: Get well soon).
One more: John Sides shares an analysis of voter turnout among Virgos.
By Mark Blumenthal on January 5, 2009 12:14 PM
| Permalink
| Comments (0)
| TrackBacks (0)
January 5, 2009
By Eric Dienstfrey
Gallup Poll
2001-2001
Mode: Live Telephone Interviews
National
"Because of these ups and downs, Bush's 49% approval average for his presidency will rank him in the middle of the pack (7th of 11) of post-World War II presidents. His average to-date of 49.4% is similar to Richard Nixon's 49.1% but slightly better than Harry Truman's and Jimmy Carter's historical lows below 46%."
(source)
By Eric Dienstfrey on January 5, 2009 10:52 AM
| Permalink
| Comments (0)
| TrackBacks (0)
December 25, 2008
By Mark Blumenthal
Over these last two years, we are thankful to have had the opportunity to share the most exciting, compelling election of our lifetimes with you from this unique vantage point. The last year in particular has been long and sometimes grueling, so my family and I are looking forward to taking the next week or so off. Eric will be checking in from time to time, but I will be pretty much off the radar until the new year.
It will be a new year full of new polls and new challenges, and we are looking to bringing you new charts and analysis to follow it all.
Until then, from all of us here at Pollster.com, we wish you a Merry Christmas, a Happy Hanukkah or joy in whatever way you celebrate the holiday season. And if I don't get the chance to say it online, a happy New Year too.
By Mark Blumenthal on December 25, 2008 7:24 AM
| Permalink
| Comments (1)
| TrackBacks (0)
December 24, 2008
By Eric Dienstfrey
The latest USA Today/Gallup survey says that while 80% of Americans identify with a Christian faith, 93% indicate that they celebrate Christmas.
According to the most recent ABC News poll, "Americans' Christmas gift wish list has a decidedly more prosaic cast this recession year."
56% of Americans say Gov. Rod Blagojevich has been the "naughtiest" political figure this year, according to a recent CNN / ORC survey; 25% say Gov. Eliot Spitzer, 19% say John Edwards.
A recent Marist poll says that 19% of Americans choose It's a Wonderful Life as their favorite Christmas movie; less than 1% choose Gov. Schwarzenegger's tour de force.
By Eric Dienstfrey on December 24, 2008 4:36 PM
| Permalink
| Comments (0)
| TrackBacks (0)
December 24, 2008
By Mark Blumenthal
Gary Langer offers a requiem for a fallen comrade, the Los Angeles Times poll, dead at age 31.
Mark Murray reports on another debriefing by McCain pollster Bill McInturff.
NPR reviews public opinion on single payer national health insurance.
Jennifer Agiesta looks behind the Post/ABC job loss numbers.
Jon Cohen finds support for the popular vote rather than the electoral college.
The Pew Research Center shows the Internet overtaking newspapers as a news source.
Curtis Gans (PDF) releases the CSAE turnout report (via Hotline On Call)
Swing State Project is crowdsourcing results from the presidential race by congressional district (via Jensen).
John Sides graphs corruption by state -- which state is #1?
David Hill sees little chance for the GOP to leverage the Blagojevich scandal.
Mark Mellman plays the economic anxiety Grinch.
Survey Practice looks at how interviewers view race matching, a British analysis of the American Elections, audio recording in in-person surveys and more.
Aleks Jakulin points to a Slashdot review of a "manga" comic book* about statistics, Andrew Gellman is underwhelmed. *See AySz88's comment.
By Mark Blumenthal on December 24, 2008 1:38 PM
| Permalink
| Comments (1)
| TrackBacks (0)
December 23, 2008
By Mark Blumenthal
In the eight weeks since the elections, I have seen a fair number of self-congratulatory press releases from pollsters boasting of their successes during 2008, but none had quite the audacity of the release put out last week by Investor's Business Daily about the polls they conducted along with the TechnoMetrica Institute of Policy & Politics. "IBD/TIPP Takes Top Honors Again," read the headline, continuing in bold print below:
Now that the '08 tally is official, we note that for the second election in a row, the IBD/TIPP Poll not only came closest to the final margin, but was right on the money -- tantamount to hitting a bullet with a bullet.
My thoughts of these snap judgements of poll "accuracy" over the last two months have had a common theme: We are placing far too much emphasis on "the last poll." Nothing supports that argument as much as the claims in this IBD release. And the story is worth telling in detail, so pull up a chair and let me begin at the beginning.
IBD/TIPP released results of daily tracking from October 13 through the night before the election, Monday, November 4. Their first release was based on interviews conducted over seven days (October 6-12). Their field period narrowed to five days as of the October 16 release and to four days on their second-to-last release on November 2.
As the chart shows below, the IBD/TIPP poll showed a persistent "house effect" favoring McCain. Twenty-two (22) of 23 releases before 11/4 produced an Obama margin that was below our trend estimate. It produced an Obama margin that averaged roughly four percentage points over the course of October, compared to six to eight points for our trend estimate over the same period. A roughly 2-3 point difference on the margin is not large in absolute terms, but it does put the margin reported by the IBD polls at the lower end of the range of those reported by other national polls during October 2008. Other polls produced leads for Obama on the opposite end of that range, but the average margin of roughly seven points during most of October matched the final result almost perfectly.
Without undecideds allocated, the final IBD/TIPP polls showed Obama leading by 5.1 points (47.5 to 42.4%), a result only slightly better for Obama than their average result over the previous four weeks. However, their final projection on Monday afternoon allocated to Obama two-thirds of remaining 6% that were still undecided. That allocation produced an projection that was remarkably close to what the poll averages showed earlier that day. The final IBD/TIPP projection had Obama ahead by 7.2 percentage points (51.5% to 44.3%), exactly the same as our trend estimate as of Monday morning (Obama +7.2 points; 51.5% to 44.3%) and just one tenth of a percent different from the RealClearPolitics average that day (Obama +7.3 points; 51.6% to 44.3%).
Obama's final margin -- as of this writing and based on final results in each state -- turns out to be urned out to be 7.25 percentage points (52.9% to 45.7%), within a whisper of the final IBD/TIPP projection and the result that all other polls, collectively, had been showing for weeks (at the same time, the actual 1.2% support for candidates other than Obama and McCain fell far short of IBD/TIPP's projection of 4.1% -- something not accounted for in their boast of "top honors").
However, the "house effect" that made earlier IBD/TIPP results more favorable to McCain was not lost on those producing the IBD/TIPP analysis. Here is the first line of their first poll release:
In contrast to other polls, which show Obama leading McCain by 4 points (Reuters/C-SPAN/Zogby) to 11 (Newsweek), the IBD/TIPP Tracking Poll debuts today with Obama up just 2 points with 13% (including 25% of independents) undecided.
They came back to this theme repeatedly. On October 22:
Contrary to other polls, some of which show Obama ahead by double digits, the IBD/TIPP Poll shows a sudden tightening of Obama's lead to 3.7 from 6.0.
And again on October 29, the IBD/TIPP analysis pointed out that other polls were "migrating" (briefly, as it turned out) to their closer than average result:
The race tightened again to 3 points Wednesday, a margin IBD/TIPP has shown for six days and to which other polls appear to be migrating. For example, the Rasmussen and Gallup polls, each of which had Obama up 5 points two days ago, now have him at 3.
The relative closeness of the IBD/TIPP polls was also repeatedly noted by supporters of John McCain (all quotes via Nexis transcripts):
- Bill Bennett on CNN, October 22: "Wolf, I don't know if you saw them or James, AP poll and IBD poll which had these races much closer and in some of these states had McCain ahead."
- Hugh Hewitt on CNN, October 23: "I know they're not covered much in the media, but if you look at the Investor's Business Daily poll today, the poll, it's down to 1 point. Yesterday, it was three point seven."
- Sean Hannity on Fox, November 3: "Investors Business Daily, which was the most accurate poll in the last presidential election, now has it down to a two-point race."
- Mary Matalin on CNN, November 4: "[The McCain campaign is] looking at their polls, which are better than the public polls and do match like the IBD poll, all those that are out there but in the target states where they need to close the gap...this is [a] harsh [environment], as you're saying, but certainly not undoable."
Not surprisingly, many conservative bloggers relied on the IBD/TIPP polls to support the same conclusion, namely that the race was closer than other polls made it appear. Ironically, to strengthen that argument, many pointed to IBD/TIPP's boast of being the "most accurate" pollster in 2004, a message that appeared prominently alongside each day's results. "An analysis of Final Certified Results for the 2004 election," they wrote, "showed IBD's polling partner, TIPP, was the most accurate pollster of the campaign season."
TIPP's final projection in 2004, showing a 2.1 percentage point lead, did come closest among the final national polls to the ultimate 2.5 percent margin, although it is worth noting that IBD was one of the few to report results to one decimal place. Four other pollsters showed Bush leading by margins of 2 or 3 percentage points on their final 2004 poll. Since they rounded off their results to the nearest whole number, we cannot say for certain who would have been "closest," although reporting results out to one decimal is largely meaningless for individual polls given margins of error of 3 percentage points or more.
The same can be said this year: Seven pollsters other than IBD/TIPP reported final results or projections that rounded to Obama leads of 7 or 8 percent. Again, IBD/TIPP can claim to be "closest" to the final margin mostly because they chose to report results to one (largely meaningless) decimal place. Other pollsters may have been just as close. Moreover, remember random sampling error. How close a pollster comes to the illusion of "pinpoint accuracy" on any given survey (or even on any two surveys) is still largely a matter of chance.
But the issue raised here that many will wonder about is the allocation of two-thirds of the undecided voters to Obama, especially given the analysis I linked to yesterday by Mickey Blum about the so-called "convergence mystery." She noted that allocations that moved estimates closer to the poll averages helped explain the sharply decreased variance among the final polls that David Moore identified (see also my commentary). So what do we make of the fact that the IBD/TIPP allocation produced a result converged almost perfectly with the poll averages?
First, in their defense, IBD/TIPP is certainly not the only pollster to wait to allocate undecided voters until their final release. Gallup has followed that practice for decades, and five other national survey organizations did the same this year.
Second, Raghavan Mayur, president of TechnoMetrica, explained in a comment posted here yesterday that they used a "Hierarchical Heuristic algorithm" in both 2004 and 2008 that had been developed using their data for the decided. Such algorithms create mutually exclusive clusters of voters based on combinations of variables that, in this case, maximize the difference across those subgroups in terms of support for Obama or McCain. They divide up the decided electorate, creating some subgroups that are very likely to support Obama, some that are very likely to support McCain and others that fall somewhere in between based on what Mayur describes as "dominant demographics" that predict candidate support.
In response to my email query last week, Mayor provided more details on the workings of their algorithm. In this case they created groups of clusters based on combinations of seven variables (race, party identification, age, income, self-reported ideology, 2004 vote and religion). Once they had used those variables to create segmentation based on the decided, they developed allocation rules based on the demographic segments (and, presumably, on the level of support for Obama and McCain in each segment) that they used to allocate each undecided respondent.
So Mayur's explanation is essentially that their allocation was based not on a gut hunch or an arbitrary "rule" based on past experience but rather a statistical model based driven by individual level data. He also sticks doggedly to the initial IBD line that other pollsters had it wrong:
Other commentators questioned why we often had the Obama-McCain race tighter than other polls. The fact is, the race was tight right down to the last week, when undecided voters swung to Obama. The final 7.2-point spread was the widest we showed since our polling started Oct. 13.
Unfortunately, I find no evidence anywhere else to support the contention that two-thirds of undecided voters "broke" to the Democrat in either 2004 or 2008. Not in the exit polls. Not in comparisons between other polls and the final results. Only the IBD/TIPP poll achieved "awesome results" (as Mayur put it in an email), by assuming a 2-to-1 break to the Democrats in 2008 and nearly as much in 2004. More specifically:
- The average or trend estimate of the other national polls (Obama +7.6 on both our final trend estimate and the RCP final average) was less than four tenths of a percent from the actual result.
- Of the seven pollsters that allocated undecided voters for their final projection, IBD/TIPP was the only one that allocated more to Obama than McCain. At least one of those -- the Pew Research Center -- used regression modeling of the decided to inform their allocation, which favored McCain slightly. Charles Franklin and I also did a similar application of a model developed among the decided in early October and found a near even split among the undecided.
- To justify an undecided percentage on his final poll (9%) that was higher than others (our trend estimate showed 5-6% undecided for much of October), Mayur reminds us that according to the exit polls, "the share of electorate deciding in the last weekend in 2004 was 9%, and in 2008 was 7%." True. However, those same exit polls show the same late deciders dividing evenly in 2008 (49% Obama, 48% McCain), and slightly in Kerry's favor (53% to 44%) in 2004.
There is also some confusion about the number of variables used to create the algorithm that allocated the undecideds. When the Wall Street Journal's Carl Bialik asked Mayur about it shortly after the election, Bialik reported that TIPP's allocation "method uses 10 variables, such as the respondent's party, gender and 2004 vote, to predict the 2008 vote." When I asked Mayor to specify the 10 variables, he listed only seven (race, party, age, income, ideology, 2004 vote and religion), explaining that "we also look at gender but do not use it in the model." However, the document he attached to the same email has one listing of classification rules that does not include the age variable, and a listing of computer code that does (Mayur asked that I treat the additional details contained within that document as "strictly 'confidential" so that I would not "give away our 'bread and butter' to other firms").
The point here is that the underlying model used to allocate undecided voters still leaves a lot of room for subjective judgements. How many variables are used to create the segmentation? Which subgroups from that segmentation are used to allocate undecided, and which are not? What is the criteria that determines whether a given subgroup gets allocated to Obama or McCain?
Writing here yesterday, Mayur offered a "moral" of their allocation story:
1. Close to ten percent of the electorate makes decision in the final weekend (Saturday, Sunday, and Monday).
2. They don't break even. Democrats have a clear advantage among them, at least in the past 3 races.
Respectfully, those are the wrong lessons. Here is what we have learned:
- We see a conspicuous disparity between IBD/TIPP's final projection and the results released earlier in October.
- The accuracy of the final poll projections from IBD/TIPP are useful, at best, only in evaluating subsequent final horse race projections from that organization, not the results that come earlier.
- The polling industry, and those of us that follow it, place far too much emphasis on "the last poll" in judging poll accuracy and reliability.
Finally, let us be clear that we are in no position to discern an inappropriate motive on the part of TIPP or Investor's Business Daily. The more important issue here is what we make of the results they produced.
PS: I asked Raghavan Mayur for his comment on my argument that we put too much emphasis on final polls generally, and on the final IBD/TIPP polls specifically, in measuring survey accuracy. His full response appears after the jump.
Continue reading ""Hitting a Bullet with a Bullet" - A Cautionary Tale"
By Mark Blumenthal on December 23, 2008 3:00 PM
| Permalink
| Comments (5)
| TrackBacks (0)
December 23, 2008
By Eric Dienstfrey
CNN / ORC
12/19-21/08; 1,013 adults, 3% margin of error
Mode: Live Telephone Interviews
National
In general, do you think there is too much, too little, or about the right amount of government regulation of the stock market and financial institutions?
22% Too much
59% Too little
18% Right amount
As you may know, a financial advisor named Bernard Madoff has been accused of running an illegal scheme in which many people lost millions of dollars. Do you think Madoff is an isolated case that does not reflect on other financial advisors and institutions, or do you think Madoff's behavior is common among financial advisors and institutions?
23% An isolated case
74% His behavior is common
By Eric Dienstfrey on December 23, 2008 10:41 AM
| Permalink
| Comments (0)
| TrackBacks (0)
December 23, 2008
By Eric Dienstfrey
Quinnipiac University
12/17-21/08; 834 registered voters, 3.4% margin of error
Mode Live Telephone Interviews
(source)
New York State
As you may know, Senator Hillary Clinton has been nominated to be Secretary of State and Governor Paterson will pick her replacement in the Senate. Who do you think Governor Paterson should pick to replace Hillary Clinton in the United States Senate; Caroline Kennedy, Andrew Cuomo, Kirsten Gillibrand or someone else?
33% Kennedy
29% Cuomo
4% Gillibrand
24% Someone else
Do you think Caroline Kennedy is qualified to be a United States Senator or not?
By Eric Dienstfrey on December 23, 2008 9:27 AM
| Permalink
| Comments (0)
| TrackBacks (0)
December 22, 2008
By Eric Dienstfrey
DailyKos.com (D) / Research 2000
12/15-17/08; 600 likely voters, 4% margin of error
(400 likely Republican primary voters, 5% margin of error)
Mode: Live Telephone Interviews
(source)
Alaska 2010 Elections
Governor:
General Election: Palin (R-i) 55%, Knowles (D) 38%
Senate:
GOP Primary: Palin 55%, Murkowski (R-i) 31%
General Election: Palin 53%, Knowles 39%
General Election: Murkowski 49%, Knowles 41%
House:
GOP Primary: Young (R-i) 33%, Parnell (R) 27%, Harris (R) 11%
General Election: Young 49%, Berkowitz (D) 46%
By Eric Dienstfrey on December 22, 2008 11:15 PM
| Permalink
| Comments (1)
| TrackBacks (0)
December 22, 2008
By Eric Dienstfrey
American Research Group
12/16-19/08; 1,100 adults, 3% margin of error
Mode: Live Telephone Interviews
National
Pres. Bush Job Approval: 28% Approve, 67% Disapprove
By Eric Dienstfrey on December 22, 2008 4:53 PM
| Permalink
| Comments (2)
| TrackBacks (0)
December 22, 2008
By Eric Dienstfrey
CNN / ORC
12/19-21/08; 1,013 adults, 3% margin of sampling error
Mode: Live Telephone Interview
National
The federal government will provide some of the major U.S. auto companies with more than thirteen billion dollars in loans in order to prevent them from going into bankruptcy. In exchange for the loans, the government wants the auto companies to produce plans by the end of March that show how they would become viable businesses in the long run. Do you favor or oppose this decision?
If the major U.S. auto companies ask for more money next year, do you think the federal government should give them any additional assistance, or should the government let them go into bankruptcy?
28% Additional assitance
70% Let them go into bankruptcy
How would you rate Dick Cheney in comparison to other vice presidents in U.S. history? Would you say he is the best vice president the U.S. has ever had, very good, good, poor, very poor, or would you say he is the worst vice president the U.S. has ever had?
1% Best
4% Very good
30% Good
28% Poor
13% Very poor
23% Worst
By Eric Dienstfrey on December 22, 2008 4:17 PM
| Permalink
| Comments (1)
| TrackBacks (0)
December 22, 2008
By Eric Dienstfrey
CNN / ORC
12/19-21/08; 1,013 adults, 3% margin of error
Mode: Live Telephone Interviews
National
As you may know, Illinois Governor Rod Blagojevich has been arrested on charges of corruption. Based on what you have heard or read, which of the following statements best describes your view of any recent contact between Blagojevich and top aides to Barack Obama:
12% Some Obama aides did something illegal
36% No Obama aides did anything illegal, but did something unethical
43% No Obama aides did anything seriously wrong
By Eric Dienstfrey on December 22, 2008 1:28 PM
| Permalink
| Comments (1)
| TrackBacks (0)
December 22, 2008
By Eric Dienstfrey
ABC News / Washington Post
12/11-14/08; 1,003 adults, 3% margin of error
Mode: Live Telephone Interviews
(story, results)
National
Apart from dealing with the economy, do you think Obama should or should
not...
... withdraw U.S. forces from Iraq within the next 16 months?
70% Should
26% Should not
... make major changes in the U.S. health care system?
77% Should
20% Should not
... implement policies to try to reduce global warming?
75% Should
20% Should not
... close the U.S. military prison at Guantanamo Bay in Cuba?
40% Should
44% Should not
... expand federal funding of embryonic stem-cell research?
52% Should
42% Should not
By Eric Dienstfrey on December 22, 2008 11:17 AM
| Permalink
| Comments (1)
| TrackBacks (0)
December 22, 2008
By Mark Blumenthal
Survey Practice, AAPOR's online publication, has a new issue out that includes a follow-up to David Moore's provocative piece on what he dubbed the "convergence mystery." Moore observed that the national polls often produced "contradictory estimates and trends" -- and more specifically, a greater variance of results -- that significantly converged over the last few days of the campaign. I posted a follow-up here that showed a similar pattern in state level polling and posed a theory that might explain some of the convergence, and Moore responded with more data and a different theory.
In their new issue, Survey Practice publishes responses from five survey researchers who deserve the title "expert:" Paul J. Lavrakas, Michael Traugott, Micheline Blum, Cliff Zukin and Don Dresser. David Moore follows up with a new article of his own that begins with this helpful summary:
The two major types of explanations offered by our experts for the convergence of polls right before the election are based on 1) changes in the pollsters’ methodology and 2) changes in the certainty of the vote choice.
The first explanation suggests that in the final weeks of the campaign, many pollsters adjust their likely voter models (mentioned by Lavrakas and Blum) or they increase their sample sizes (Dresser).
Lavrakas argues that the adjustment of the voter models, even when done explicitly to make their outcomes more consistent with other polls, should be seen as a positive action rather than as a “suspicious” activity. However viewed, such last minute changes could account for some of the convergence.
Dresser mentions the tendency of many pollsters to substantially increase their sample sizes for their final pre-election polls, to insure as small a margin of sampling error as they can reasonably afford. He specifically mentions Pew, Harris and ABC/Washington Post.
Blum suggests that the outlier polls during the campaign were either a) less likely to poll in the final three days, or b) more likely to allocate their undecided voters, thus bringing them closer to the mean.
Zukin speculates that the convergence has less to do with pollsters’ methods and more to do with the “phenomenon we are measuring.” Specifically, as voters become more certain about their choices, polls will tend to converge toward each other. That sentiment is also found in the explanations by Lavrakas and Blum.
Moore goes on to offer his own thoughts and asks why polls, "conducted at the same time, using virtually the same wording - [are] supposedly more accurate (reliable) when opinion is more crystallized?" For those intrigued by the "mystery," It is all worth reading in full.
Among all of these efforts to explain, I was most intrigued by Blum's analysis, which was the only one of the five to scrutinize the poll data directly. Among the national polls, at least, much of the variance in mid-October can be explained by a small handful of "outlier" results two-weeks out from the election:
Of the six organizations with outlier polls, three reported margins larger than 11 points, and all three consistently showed larger margins in their polls. In the final three days, however, one of these organizations allocated the undecided, giving more to McCain, one did not release a poll, and one had a margin of exactly11 points on its final poll. Of the three organizations releasing four polls with margins smaller than 5 points, one organization (with 2 “outlier” polls) allocated the undecided in the final three days, one did not release a poll in the final three days, and one “converged.” So, of the 6 organizations, 2 organizations (accounting for 3 “outlier” polls) allocated undecided in the final three days in the direction of the previously underestimated candidate, 2 did not release polls in last 3 days, and only 2 “converged.” If we remove the 7 “outlier” polls from the 29 released in the week of 10/21-27, the variance is reduced to only 2.9 points.
Her conclusion:
Basically then, both of the explanations examined, the allocation of the undecided by seven organizations in the final three days and the absence or favorable allocation of a few “outlier” organizations, appear to be major contributors to the “convergence” seen. Apportioning the undecided in the favorable direction and the absence of previous outliers virtually guarantees less variance and the appearance of “convergence. So, perhaps, rather than convergence, what we saw was that much of the earlier variance was due to a few outliers and that the final three days benefited from their absence or favorable apportionment of their undecided vote.
Blum's observation prompted me to take a closer look at the national polls that allocated the undecided. I find seven projections by six organizations (the two pollsters for the GWU/Battleground poll, Republican Ed Goeas and Democrat Celinda Lake, once again produced competing projections with different allocations). The process of allocating undecided should not, by itself, raise any suspicion. Of the seven allocations, three changed the initial margin not at all or within rounding difference of the averages, two moved the margin farther away from the poll averages as of Monday 11/4, and one moved it closer. However, that one move -- by the IBD/TIPP poll -- was important in helping produce the reduction in variance that Blum describes.
By Mark Blumenthal on December 22, 2008 11:14 AM
| Permalink
| Comments (4)
| TrackBacks (0)
December 22, 2008
By Eric Dienstfrey
USA Today / Gallup
12/12-14/08; 1,008 adults, 3% margin of error
Mode: Live Telephone Interviews
(source)
National
Which of the following would you say best describes the current economic situation in the United States?
60% Biggest crisis in lifetime
16% Crisis, but not the worst
18% Major problem
4% Not a major problem
How likely do you think it is that the United States will be in an economic depression within the next two years?
35% Very
39% Somewhat
21% Not too
4% Not at all
By Eric Dienstfrey on December 22, 2008 11:06 AM
| Permalink
| Comments (0)
| TrackBacks (0)
December 19, 2008
By Eric Dienstfrey
Rasmussen Reports
12/17-18/08; 1,000 likely voters, 3% margin of sampling error
Mode: Live Telephone Interviews
(source)
National
Caroline Kennedy, the daughter of President John F. Kennedy, is being considered as Hillary Clinton's replacement in the U.S. Senate now that Mrs. Clinton is becoming secretary of State. Is Caroline Kennedy qualified to serve in the Senate?
Do you have a very favorable, somewhat favorable, somewhat unfavorable or very unfavorable impression of Caroline Kennedy?
67% Very favorable/Somewhat favorable
23% Very unfavorable/Somewhat unfavorable
By Eric Dienstfrey on December 19, 2008 11:25 AM
| Permalink
| Comments (0)
| TrackBacks (0)
December 19, 2008
By Eric Dienstfrey
DailyKos.com (D) / Research 2000
12/15-17/08; 600 likely voters, 4% margin of error
Mode: Live Telephone Interviews
Hawaii
Sen. Daniel Inouye (D-i) 53, Linda Lingle (R) 42
By Eric Dienstfrey on December 19, 2008 10:05 AM
| Permalink
| Comments (0)
| TrackBacks (0)
December 19, 2008
By Mark Blumenthal
Former Clinton pollster Mark Penn has a new Wall Street Journal column named (what else?) "Microtrends" and written "with" E. Kinney Zalesne, co-author of the book of the same name.
Gawker responds with its usual snarky flair but makes a reasonable point, if you can get past the gratuitous nastiness: This week's column spends more time hawking Penn's book than supporting his arguments with data. They point out, for example, that Penn/Zalesne attach this footnote their first use of the term "Impressionable Elites:"
* "Impressionable Elites" is the term we used for educated, affluent people who focus more on personality than issues when it comes to evaluating political decisions. For more, please see pages 131 to 135 in "Microtrends."
There's a stranger footnote. The central contention of the column is that Bernard Madoff's bilking of a lot of very rich people "proves the point" that "elites have become more impressionable -- more removed from everyday problems, more trusting of what they hear, and more likely to adopt unthinking viewpoints based on brand or emotion."
The data that supports this claim?
[O]ur research** shows that the top 1% is heavily swayed by gut and impression, not numbers and facts. They vote more on the basis of personality in campaigns; buy products more on the basis of brands; and invest more on the basis of the tip than on sound logic.
And what research is that? Another footnote tells us:
** A PSB [Penn, Schoen & Berland] poll of 806 telephone interviews among likely 2008 presidential voters, including an oversample of 400 very likely Democratic presidential primary voters.
Two issues here. First, sample size? I will leave it to readers to decide exactly how many interviews such a sample will yield for the "top 1%." Presumably, non-voters fall disproportionately in the bottom 99%, so it may be that the wealthiest or best educated 1% of the population contributed slightly more than 1% of this sample. Still, I still have a hard time seeing how that top 1% subgroup could have yielded much more than n=15, unweighted.
Second, consider the reference to the "oversample" of likely "Democratic presidential primary voters." It certainly sounds like a survey done during or before the Democratic primaries on behalf of Mr. Penn's most well known recent client. It is not hard to imagine such a survey including tests of negative messages about Obama ("numbers and facts") that failed to persuade his very wealthy supporters (for an example of such a message test -- click here, scroll to pp. 11-12). Please tell me that the referenced evidence of elite impressionability depends on more than that. Please.
Update: Reader DS points out that the Microtrends chapter referenced in that first footnote includes results that give a good clue as to the evidence referenced in the second footnote. Specifically, on pp. 132-133, Penn and Zalesne write (the table is my reproduction of the one in the book):
This isn't just my gut. Let's look at the data.
A standard poll question I ask in campaigns is what people consider most important in voting for a candidate: (1) issues, (2) character, or (3) experience. I ask it because I know that all three are important in a leader, and that it can be tough to rank them.
According to a recent poll we did, a large plurality of voters -- 48 percent -- believe that a candidate's stand on the issues is most important, with character a distant second at 32 percent. That preference for issues holds steady whether or not voters have been to college, whether or not they are religious, and across race. Where it does not stay constant, however, is across income. Once voters reach the magic line of $100,000 per year, their priority shifts to character, by a significant margin. As the table below shows, people earning under $100,000 prioritize issues over character by a serious 51 to 30 percent. But once they reach $100,000, the switch, to character over issues, 45 to 37 percent.
That is a 29-point swing. A shift barely ever gets clearer in polling.
So if this is "research" referenced in the footnote, there's a good chance that the data used to make inferences about the "top 1%" was based on the 25% or so of voters that report an income over $100,000. Also, as DS puts it in his email, there isn't any more survey evidence on the book on this point: "no regression analysis, no experiment, just a simple, dumb question."
As it happens, the soon-to-be-extinct LA Times/Bloomberg poll asked a similar question in their September 2008 campaign survey. I have reproduced the results below. Notice two things: First, the Times/Bloomberg question showed 31% choosing "all" or "some" of the categories. The PSB had no such category and reported only 1% as "don't know." If allowing interviewers to take "all of the above" as an answer draws in nearly a third of the respondents, we have a pretty good clue that the underlying attitudes are soft.
Second and more important, the Times/Bloomberg poll showed a clear pattern by party and ideology: Democrats and Liberals were more likely to choose "positions on issues," while Republicans and Conservatives were more likely to choose "experience" and, to a lesser extent, "character." Here's a theory: Voters perceived McCain as strong and Obama as weak on "experience," so McCain supporters were more likely to say they consider experience important and Obama supporters less likely to choose that answer. That result is consistent with my experience that voters rarely answer such questions in a completely abstract way. They tend to answer with high profile candidates in mind, especially when the question comes in the context of a survey that has already asked about those candidates.
So before making too much of the pattern in Penn's cross-tab, I would want to see whether any other variables show similar correlations and whether controlling for other variables (through a regression analysis) would explain the pattern by income.
And of course, as DS points out in his email, there are many sources of political survey data that would allow a more complete test of this thesis. He says, for example, that he examined ANES data "to see if highly educated / high income respondents were more likely to be impressed by character qualities rather than issues," but found no connection.
By Mark Blumenthal on December 19, 2008 10:04 AM
| Permalink
| Comments (1)
| TrackBacks (0)
December 18, 2008
By Mark Blumenthal
The last two days bring news on the issue of cell phones and their impact on political surveys in the form of new reports from the most respected researchers on the subject.
First, yesterday, the Center for Disease Control and Prevention's National Center for Health Statistics released its latest biannual report on the prevalence of households without wireless or standard telephone service (via Bialik). The CDC monitors the cell-phone-only population because it conducts huge ongoing health "surveillance" surveys via telephone, and as such, ask questions about telephone usage on their ongoing, in-person National Health Interview Survey (NHIS). Their latest report, which covers the first six months of the year, shows that 16.1% of adults were reachable only by cell phone, while another 2.1% lacked telephone service of any kind.
As the chart above shows, the latest survey continues an ongoing, near linear upward growth in the cell-phone-only population. It is worth noting that the hint of a plateau in the trend seen between the second half of 2006 and the first half of 2007 was likely the result of a very slight change in question wording that took effect in 2007. Except for the brief near-pause, the trend has been steadily and consistently upward since 2005.
The second news is the latest report from the Pew Research Center on their efforts to survey voters via cell phone during 2008 (summary, full PDF). It is hard to overstate* the influence of the Pew Center's work on cell phones in the political polling industry. At the PAPOR conference in San Francisco last week, Professor Mike Traugott (a past president of AAPOR and chair of AAPOR's Special Committee on 2008 Primary Polling) noted that earlier this year, "the conventional wisdom in the Spring [among pollsters] was that we didn't have to worry about cell phone only people." At the end of the summer, however, "this conventional wisdom changed drastically" when the Pew Center released its report showing that the omission of cell phone only voters could understate Barack Obama's margin over McCain by two to three percentage points." By the fall, many national media surveys included supplemental cell phone samples in their surveys.
The new Pew report confirms that the patterns they saw, "that estimates based only on landline interviews were likely to have a pro-McCain tilt compared with estimates that included cell phone interviews," persisted through their final survey in late October. It was a difference of a point or two on the margins that "while statistically significant, was small in absolute terms -- smaller than the margin of sampling error in most polls." Of possibly greater importance going forward, they also found similar differences in party identification and self-reported ideology, and bigger differences in (not surprisingly) the use of Internet as a news source and social networking sites like Facebook or MySpace. See the report for the details.
Now that the conventional wisdom on cell-phone only households has shifted among national pollsters, from "it doesn't matter" to "we better account for the them," the Pew Report also points us to the next issue for the c.w. to chew over: Simply interviewing by cell phone (and screening for "cell only" respondents) may not be enough. Pollsters will also need to confront the more difficult issue of how to handle and weight the interviews among what some call "cell phone mostlys:"
Unlike most other polling organizations, Pew's election surveys involved a "full dual frame design," in which people reached by cell phone who also have a landline are interviewed, as well as cell-only respondents. In contrast, most pollsters who included cell phones in their election surveys screened their cell samples for cell-only respondents.
The difference between these two approaches can be seen as a tradeoff in methodological challenges. Supplementing a landline sample with cell-only respondents has the advantage of not "double covering" respondents who have both types of phones. This makes combining the samples more straightforward, but assumes that the landline sample is capable of accurately reaching all adults equally. If some adults have landline phones that they rarely or never answer because they favor their cell phones, they will be underrepresented in these surveys. Pew's approach of interviewing all adults in both the landline and cell phone samples ensures that every adult with a telephone is covered by the survey, but raises challenges in combining the data because some adults had a greater chance to participate if they have more than one telephone. Pew's methodology accounts for this double coverage by weighting respondents with both kinds of phones according to their probability of selection and the regularity with which they use each kind of telephone.
The report goes on to present data showing that the slightly different results produced by the two approaches. Duel users reached by cell phone were more likely to support Obama (53%) than those reached by landline phone (46%). The cell-phone-mostlys reached by cell also identified with the Democrats (54%) more often than those interviewed by landline phone (47%).
The report concludes with a section on the "practical considerations" of interviewing by cell phone. As they have found previously, once they offer a monetary incentive to potential cell-phone respondents ($10), the contact and cooperation rates are comparable to what they get on landline phones. So cell phone interviewing can be done. The downside is that cell-phone interviews cost Pew "nearly two-and-a-half times as much as landline interviews" for reasons they
