Interrupting my "vacation" for this special announcement: My NationalJournal.com column, on the potential for a running-mate "bump" and whether we will be able to measure one, is now online.
By Mark Blumenthal on August 18, 2008 5:38 PM | Permalink | Comments (0) | TrackBacks (0)
I will be taking a break this week, although I filed a National Journal column for the week, which should appear in a day or two. Meanwhile, Eric will continue with poll updates and our other contributors should be active this week. See you next week!
By Mark Blumenthal on August 17, 2008 9:49 PM | Permalink | Comments (0) | TrackBacks (0)
"Bounce" or "Bump?" The terminology is up to you, but this is certainly the season to consider the short term changes in polling numbers that frequently result in the wake of national political conventions.
A useful first stop would be the 2004 analysis from Gallup's Jeff Jones. It includes the post convention "bounce" numbers from Gallup back to 1964 that are the primary source of the conclusion that average post convention gain for candidates has been six or seven percentage points.
Last week, Tom Holbrook posted a more a thorough review of the past "before and after" data and it's implications. He looks at the gain for each candidate by taking their "average share of the two-party vote in trial-heat polls conducted six days to two weeks prior to the start of the convention" an subtracting that from the candidate's "share of the two-party vote in polls conducted during the seven days following the close of the convention." He does not list the polls used for each year, but presumably his data looks at much more than the Gallup time series for more recent elections.
Holbrook's post is worth reading in full for the lessons he draws from the considerable variation in past convention bumps, although probably the most important is his caution that "the magnitude of the convention bump is not a great predictor of election outcome." Still, he sees a pattern to the past variation that "should be a useful guide to what to expect" from the conventions and promises to update later this week with a prediction for each candidate.
But before reading two much into the twitches in the daily tracking polls over the next three weeks, please read the latest column from CBS News director of surveys Kathy Frankovic. She reminds us that the gap between the Democratic and Republican conventions is just three days -- much shorter than in past elections -- and will coincide with the Labor Day weekend:
Will we even be able to measure whatever impact the Democratic Convention has on Obama before it’s time to measure the GOP convention’s impact on John McCain? And will we be able to sort out what has caused what?
Will we actually discover a “bounce” or a “bump?”
Probably not. Polling over Labor Day Weekend is always a problem. We confront more than the usual number of people who don’t respond or can’t respond. People are away from their homes, heading back from summer vacation, or preparing their children for the start of the school year. In addition, the focus will shift so quickly from the Democrats to the Republicans that whatever opinions might be expressed over Labor Day Weekend might not last too long.
I thought it would be helpful to look at some past elections, not in terms of the immediate "before and after" averages, but rather where the conventions fit into the longer arc of the trend in the vote preference over the course of the election year. I spent some time gathering past polls from a variety of sources and asked Charles Franklin to create some charts matching the format he used to look at the trends of the 2000 and 2004 elections a few weeks back.
I had hoped to use those charts for a series of posts. Unfortunately, I got delayed, so I will post the charts along with a some very compressed discussion after the jump.
Continue reading "Convention "Bumps" in Context"
By Mark Blumenthal on August 17, 2008 4:49 PM | Permalink | Comments (1)
In the first two installments of this online dialogue, I asked a question we have heard from readers about why we choose the results for "likely voters" (LVs) over "registered voters" (RVs) when pollsters release both. Charles answered and explained our rationale for our "fixed rule" for these situations (this is the gist):
That rule for election horse races is "take the sample that is most likely to vote" as determined by the pollster that conducted the survey. If the pollster was content to just survey adults, then so be it. That was their call. If they were content with registered voters, again use that. But if they offer more than one result, use the one that is intended to best represent the electorate. That is likely voters, when available.
Despite my own doubts, I'm convinced by the rule for this reason: I can't come up with a better one. Yes, we would arbitrarily choose RVs over LVs until some specified date, but that would leave us still plotting numbers from pollsters that only release LV samples. And on which date do we suddenly start using the LV numbers? After the conventions? After October 1? What makes sense to me about our rule, is that in almost all cases (see the prior posts for examples) it defers to the judgement of the pollster.
Several readers posed good questions in the comments on the last post. Let me tackle a few. Amit ("Systematic Error") asked about how likely voters are constructed and whether we might be able to plot results by "a family of LV screens (say, LV_soft, LV_medium, LV_hard)" and allow readers to judge the effect.
I wrote quite a bit back in 2004 about how likely voter screens are created, and a shorter version focusing on the Gallup model two weeks ago. One big obstacle to Amit's suggestion is that few pollsters provide enough information about how they model likely voters (and how that modeling changes over the course of the election cycle) to allow for such a categorization.
"Independent" raised a related issue:
Looking at the plot, it appears that Likely Voters show the highest variability as a function of time, while Registered Voters show the least. Is there some reason why LVs should be more volatile than RVs? If not, shouldn't one suspect that the higher variability of the LV votes is an artifact of the LV screening process?
The best explanation comes from a 2004 analysis (subs. req.) in Public Opinion Quarterly by Robert Erikson, Costas Panagopoulos and Christopher Wlezien. They found that the classic 7-question Gallup model "exaggerates" reported volatility in ways that are "not due to actual voter shifts in preference but rather to changes in the composition of Gallup's likely voter pool." I also summarized their findings in a blog post four years ago.
Finally, let me toss one new question back to Charles that many readers have raised in recent weeks. The two daily tracking surveys -- the Gallup Daily and the Rasmussen Reports automated survey -- contribute disproportionately to our national chart. For example, we have logged 51 national surveys since July 1, and more than half of those points on the chart (27) are either Gallup Daily or Rasmussen tracking surveys. Are we giving too much weight to the trackers? And what would the trend look like if we removed those surveys?
By Mark Blumenthal on August 13, 2008 2:26 PM | Permalink | Comments (3) | TrackBacks (0)
I want to add one thought the chorus of commentary on Josh Green's Atlantic Monthly article on the Hillary Clinton campaign, based on a remarkable collection of email and memoranda he obtained from sources within the campaign. It concerns the first sentence in an April 25 email from newly installed pollster Geoff Garin to the Clinton high command:
Attached is the filled in questionnaire from the North Carolina survey.
Those ten words probably seem utterly mundane to the ordinary reader, even to the ordinary campaign consultant. Pollsters share results with their clients. It's a basic part of the job. Notice also that Garin sent his email at 7:25 a.m. on a Friday morning. The timing and content imply that he was sharing the most critical "top line" results of a tracking survey that had completed the night before.** Thus, this email shows us Garin passing along results as soon as he has them for review by other decision makers. Further analysis and internal discussion no doubt followed.
What makes Garin's ordinary act so remarkable is that Mark Penn, the original Clinton pollster and "chief strategist" rarely delivered a "filled in questionnaires" to the Clinton campaign's senior decision makers. I know this because I heard the story a few months ago from a Clinton staffer with first-hand knowledge of what Penn provided to the campaign (who agreed to share the story on condition of anonymity). My source said that Penn would routinely brief strategy sessions without providing the complete results of the poll in advance. Instead, he would present whatever results best made his case (as exemplified by the the smattering of numbers that appear in the Penn memoranda that accompany Green's article).
Perhaps Hillary and Bill Clinton received the full data, but the senior staff and consultants did not. Amazingly, I am told, Penn also initially refused to share the full cross-tabular reports (the reams of tables like this one showing results to every question by every subgroup of interest), as is also standard practice among campaign pollsters. It was not until relatively late in the campaign at the insistence of then campaign manager Patti Solis Doyle that Penn relented, sharing a hard copy of the cross-tabs on condition that Solis Doyle keep it locked up in a file cabinet in her office.
One can understand the temptation that a "chief strategist" might have to control the flow of data. If you are convinced you have the right strategy, and you make the final decision, why give others a tool to question your judgment?
The problem with that approach should be obvious. It poisons the environment within which functional campaigns privately hash out disagreements and reach consensus about strategy. The pollsters job in this process is to put the data on the table, to provide analysis and guidance about that data, but also to let other senior staffers examine and question it. When the pollster wears two hats -- pollster and "chief strategist" -- greater conflict, questioning of motive and campaign "dysfunction" are inevitable.
**One reason I'm confident that this email followed within hours after completion of calling is that one of the respondents later blogged about his experience (discussed here). The respondent reported having been called a night or two before Garin sent his email.
By Mark Blumenthal on August 12, 2008 4:25 PM | Permalink | Comments (7) | TrackBacks (0)
Since adding the maps and upgrading this site, we have received a number of good questions about how the charts and trend lines work and why we choose to include the poll results that we do. I want to answer a few of those questions this week before we call get swept up in the conventions and the final stretch of the fall campaign.
Our approach to charting and aggregating poll data follows the lead and philosophy of our co-founder Charles Franklin. And while I am tempted to describe that approach as well entrenched, the reality is that in many ways it has and will continue to evolve.
Since launching this site nearly two years ago, Franklin and I have continued to discuss (and occasionally debate) some of the technical issues offline. Most of the time we agree, but I tend to propose ways to change or tinker with our approach, and Franklin usually succeeds in convincing me to stay the course.
In considering some of issues that came up more recently, I thought it might be helpful to take this dialogue online. Hopefully, we can both answer some of the questions readers have asked and also seek further input on those issues we have not completely resolved.
So with that introduction out of the way, here is the first question for Franklin:
Over the last few weeks, in commenting on the "likely voter" subgroups reported by Gallup and other national pollsters, I have essentially recommended that we focus on the more stable population of registered voters (RV) now, and leave the "likely voter" (LV) models for October (see especially here, here, here and here). Yet as many readers have noticed, when national surveys publish numbers for both likely and registered voters, our practice has been to use the "likely voter" numbers for our charts and tables.
Why?
Like the other sites that aggregate polling results from different sources, we face the challenge of how to best choose among many polls that are not strictly comparable to each other. Even if we examine data from one pollster at a time, we will still see methodological changes: Many national pollsters will shift at some point from reporting results from registered voters to "likely voters." Some will shift from one likely voter "model" to another, or will tinker with the mechanics of their model, often without providing any explanation or notice of the change. And no two pollsters are exactly alike in terms of either the mechanics they use or the timing of the changes they make.
As such, two principles guide our practices for selecting results for the charts and tables: First, we want to defer to each pollster's judgement about the most appropriate methodology (be it sample, questionnaire design or the most appropriate method to select the probable electorate). Second, we want a simple, objective set of rules to follow in deciding which numbers to plot on the chart.
In that spirit, when pollsters release results for more than one population of potential voters, our rule is to use the most restrictive. So we give preference to results among "likely" voters over registered voters and to registered voters over results among all adults. In almost all cases, the rule is consistent with the underlying philosophy: The numbers for the more restrictive populations are usually the ones that the pollsters themselves (or their media clients) choose to emphasize.
But there have been some notable exceptions recently, of which, last month's ABC News/Washington Post poll provided the most glaring example. ABC News put out a report and filled-in questionnaire with two sets of results: They showed Barack Obama leading John McCain by eight points (50% to 42%) among registered voters, but by only three points (49% to 46%) among likely voters. Following our standard procedure, we included the likely voter numbers in our chart.
However, ABC News emphasized the eight-point registered voters numbers in the headline of their online story ("Obama Leads McCain by Eight But Doubts Loom"). Within the text, they first reported the registered vote numbers and then used the likely voter results to argue that "turnout makes a difference." The 8-point lead also made the headline of the Washington Post story, but they did not report the likely voter results at all, either in the text of the story on in their version of the filled-in questionnaire.
So in this case, the news organizations that sponsored the poll clearly indicated that the RV numbers deserved greater emphasis, yet we followed our rule and included the LV numbers in our charts.
Charles, in cases like these, should we find a way make an exception? And why not just report on "registered" voters until after the conventions?
Update: Franklin answers in Part II.
By Mark Blumenthal on August 11, 2008 5:59 PM | Permalink | Comments (3) | TrackBacks (0)
The punditry is crackling this morning over remarks by Howard Wolfson, Hillary Clinton's campaign communications director, over what might have happened had John Edwards' been forced out of the presidential race last year: "I believe we would have won Iowa, and Clinton today would therefore have been the nominee," Wolfson told ABC News.
Washington Post polling director Jon Cohen did the logical thing and checked relevant survey data from Iowa:
It is a pure hypothetical, of course, and the entire dynamics of the contest would have been different without Edwards. But the public data do not bolster the notion that Clinton would have won.
In the networks' Iowa entrance poll, 43 percent of those who went to a caucus to support Edwards said Obama was their second choice, far fewer, 24 percent said they would support Clinton if their top choice did not garner enough votes at that location. The remainder of Edwards' backers said they would be uncommitted under such a scenario, offered no second choice or said they preferred someone else.
Nor was Clinton the obvious second choice among Edwards supporters in Post-ABC pre-election Iowa caucus polls in July, November or December. In July, for their alternate pick, Iowans split 32 percent for Obama to 30 percent for Clinton. In November, Obama led 43 to 26 percent as backup pick, and he had a slight 37 to 30 percent edge in December.
Nate Silver echoes that last point point and notes that, looking at the trend lines in late January, "Barack Obama appeared to get the lion's share of Edwards supporters once Edwards dropped from the race."
By Mark Blumenthal on August 11, 2008 12:15 PM | Permalink | Comments (6) | TrackBacks (0)
Kathy Frankovic reminds us that despite "record levels of participation and interest...voters may be doing other things in July and August."
Mark Mellman sees the din of poll analysis drowning out an opportunity to educate.
Lydia Saad explains why the election is not just about Obama and why McCain is keeping it close.
John Sides reminds us that the forecasting models predict a close election.
Nate Silver examines whether Barack Obama has been "underachieving."
Jennifer Agiesta compares low-wage workers interviewed in cell-phone-only households with those from land-line telephone households.
Frank Newport says religious affiliation remains an important predictor of vote choice
Todd Domke reviews the factors that will make for unpredictable polls in 2008.
Andrew Gellman shows that from 1948 to 1992, the popular vote showed little "bias" in it relationship with electoral votes (and links to a paper finding much the same in 2000 and 2004).
Tom Jensen identifies the "tipping point" for Obama in North Carolina African American turnout.
Craig Garthwaite and Tim Moore [pdf] say Oprah Winfrey was responsible for "an additional 1,000,000 votes" for Barack Obama (via John Sides).
David Hill chides GOP consultants for second guessing the McCain campaign.
Jay Cost dives deep into the Pennsylvania numbers.
Josh Goodman has a polling wish list.The New Yorker's Bruce McCall has the latest from the all-important New Orleans Times-Picayune/Bravo/Popular Mechanics poll..and more!
And from our regular contributors...
Charles Franklin compares polling trends in 2000, 2004 and 2008
Margie Omero finds Barack Obama's support from women anything but "lackluster."
Steve Lombardo says the race is closer than you think.
Kristen Soltis makes a case for weighting by party.
David Moore and guest contributor Nick Panagakis continue to debate how to best measure "undecided."
By Mark Blumenthal on August 8, 2008 5:33 PM | Permalink | Comments (1) | TrackBacks (0)
Here are a few short but relevant updates to topics covered earlier in the week that I did not want to get lost at the bottom of older posts:
One thing we overlooked in the Time poll: The self-identified "registered" voters included an even smaller percentage of 18-to-29-year-olds (9% - see QF1) than the "likely voters" in the USA Today/Gallup survey (10%), and six points fewer than the self-ID'd registered voters in the Gallup survey (15%).
One possible reason the uncommitted voters haven’t changed much: they’re paying much less attention to the campaign in the last few weeks.
When asked in mid-July how much attention they’d been paying to the 2008 campaign, generally, 45% said they’d paid a lot and just 14% said not much or none. When asked in this poll how much attention they’d been paying in the last few weeks, only 18% reported paying a lot lately.
By Mark Blumenthal on August 8, 2008 1:04 PM | Permalink | Comments (4) | TrackBacks (0)
Are all "likely voter" models created equally? Not at all.
Case in point, the comment left yesterday by George Mason University political science Professor Michael McDonald about the latest Time/SRBI poll:
Continuing my war on likely voter models...
Here we have 808 "Registered Likely Voters." Q1 reports 100% of the sample is registered and Q2 reports 90% are "definitely" going to vote and 10% "probably." I guess this means that registered likely voters must have to respond affirmatively to being registered and "definitely" or "probably" to voting. This is different from Gallup, which requires likely voters to have a past history of voting and to express an interest in the campaign. There is no indication of weighting in this survey, so who knows what it going on there.
If I am correct, then this two-question likely voter model seems less biased against young voters and less volatile due to changing interest. This may explain the stability since June in this poll compared with the USAToday/Gallup poll.
Mike's theory seemed plausible, so I sent an email to Mark Schulman, CEO of Abt SRBI, the firm that conducts the Time poll. Here is his full response:
Mike, the Time sample is indeed weighted based upon the entire cross-section sample, as are most election surveys. We retain demographics for the entire sample, registered or not, and weight the entire cross-section sample on the usual Census demographic variables. The 100% you cite is the total of self-reported registered voters who are then asked about likelihood to vote. It does not include unregistered screen outs, who skip straight to the weighting demographics. I see that this can cause confusion. I'm glad that you requested this clarification.
You are correct in that we are not currently using past vote in our model. My objective in the pre-convention polling is to be fairly inclusive in the voter model until after the nominating conventions, when the campaigning starts in earnest. We're likely being a bit too inclusive with the light voter screen, but this still improves upon reporting based upon registered voters. Research on models which include "interest in campaign" and related questions finds variability in the composition of the likely voter profile during early campaign period, leading to some volatility in the estimate. This volatility is reduced as the election approaches.
We always tighten the model a notch after the nominating conventions. To be perfectly honest, I don't claim to have all the answers at this point on which approach we will use to tighten the model. I'm concerned about the likely influx of new voters, young voters, newly registered voters, newly activated voters. In 2004, we had an increase in turnout, even with an incumbent whose job rating was still just below 50% at that time. I don't have a fix at the moment on what to expect in 2008. Our plan is to consult with several leading experts in turnout models later this month and then make some decisions on which approach to take on our turnout model and targets. We're not wedded to any one approach. FYI, for internal purposes, we do break out our horse-race data by likelihood to vote to gauge the impact of smaller vs. larger turnouts.
I do wish to emphasize that we should not strictly abide by past turnout percentages reported by the U.S. Census. Our landline telephone universe is smaller than the Census CPS universe because of undercoverage. Therefore, our target turnout number will be higher than Census turnout trend data would suggest.
Thank you again for requesting this clarification.
If all of this detail confuses you, here is the short version: The Gallup Likely voter model, as applied to the last two USA Today/Gallup polls, uses self-reports of past voting and interest in the election to help identify "likely voters" (in addition to questions about registration and intent to vote). In surveys conducted before the conventions, the Time/SRBI poll does not -- it uses only questions about registration and intent to vote.
Update: Although the Time/SRBI poll uses a simplified likely voter model that should produce less volatility, their sample of registered voters managed to include an even smaller percentage of 18-to-29-year-olds (9% - see QF1) than the "likely voters" in the USA Today/Gallup survey (10%) discussed earlier, and six points fewer than the self-ID'd registered voters in the Gallup survey (15%).
By Mark Blumenthal on August 7, 2008 2:02 PM | Permalink | Comments (2) | TrackBacks (0)
My NationalJournal.com column for the week is now online. It revisits the nearly two-week old USA Today Gallup poll that showed a big difference between registered voters and those selected as "likely voters" with a focus on the age of the likely voter pool.
After you read the column, the following data may be of interest. First, notice that while the most recent, conducted in late July, showed a net shift of seven points between registered and likely voters, no such gap existed in the poll conducted just a month before. In mid-June, Obama led by six percentage points among both registered and likely voters.
What makes that difference interesting is the additional data generously provided by Jeff Jones of the Gallup organization showing how respondents in different age groups answered the four questions used to identify likely voters. As noted in the column, younger voters tend to score lower on all four questions. Notice that the percentage of 18-29-year-olds who said they had given "quite a lot of thought" to the election plummeted from June (60%) to July (45%). Similarly, the percentage who rated their chances of voting as a 9 or 10 on a 1-10 scale dropped ten points (from 69% to 59%).
Thoughts anyone?
Update: Nate Silver has additional thoughts. Note that the method he describes as "the most logical way to handle" the likely voter problem is, in essence, the way the CBS/New York Times poll will model likely voters in October. Their most recent release provides results for registered voters, but not likely voters.
Also, see the related comments we just posted from Time/SRBI pollster Mark Schulman.
By Mark Blumenthal on August 7, 2008 12:19 PM | Permalink | Comments (4) | TrackBacks (0)
Bear with me. The post that follows links to three seemingly unrelated items that will hopefully add up to a coherent point about our tendency to over-analyze day to day "change" in polls on the presidential race.
The first is Ellen Gamerman's recent Wall Street Journal feature on "whether people are telling the truth" to pollsters. She reviews the steps well known political pollsters are taking to check for the "Bradley Effect" (sometimes also called the Bradley-Wilder effect), "the idea that some white voters are reluctant to say they support a white candidate over a black candidate." The piece notes that both CBS and ABC News will be checking whether results vary with the race of the interviewer have an impact on vote preferences.
The article also includes a useful review of the work being done by academic survey researchers on whether respondents will be more honest on "self administered" surveys (those without a live interviewer). Don't miss the interactive graphic featuring audio commentary from the researchers on examples of other ways that respondents are sometimes less than honest on surveys. While there are some intriguing new findings (see especially those on "good TV" and "M&M's"), the bulk of the research on this subject warns us to watch out for circumstances where respondents tell us "what they expect [the interviewers] want to hear" (as Tim Byers, a researcher at the Colorado School of Public Health, puts it).
That possibility leads me to a second finding from last week's "News Interest Index" survey from the Pew Research Center. The result that caught my eye received no mention in their analysis, largely because it involved a question they have been tracking on a weekly basis since January that showed no meaningful change last week:
[In the past week] Did you follow news about candidates for the 2008 presidential election very closely, fairly closely, not too closely or not at all closely?
30% very closely
34% fairly closely
21% not too closely
15% not closely at all
<1% don't know/refused
So, taking these results at face value, we know that less than a third of Americans are paying "very close" attention to the presidential race. More than a third (36%) say they are following the campaign "not too closely" or "not closely at all." Now consider the 34% who say "fairly closely" in light of what survey researchers tend to take for granted: Respondents sometimes tell us things they think we want to hear. In the context of a survey about about how much attention people are paying to the news, some respondents may be exaggerating their attentiveness to news. I would take the "fairly closely" result with a grain of salt.
Next consider the results from the second and third questions asked on the same survey. Fifty-nine percent (59%) said they heard nothing about Barack Obama in the previous week that made them either more or less favorable to Obama, and 62% said they heard nothing about John McCain that changed their view of him.
These data paint a clear picture for me: Most Americans are paying far less attention to news about the campaign than most journalists, pundits and readers of this site. If we assume that all Americans are following the campaign as a jury follows a trial, we are in error.
Finally, consider something ABC's polling director Gary Langer wrote earlier today:
We too often expect knee-jerk reactions to events of the day; rarely, in fact, do we see them. With few exceptions public opinion proceeds, instead, by a process known as considered judgment: People obtain information as it develops, evaluate it, let it accumulate to the point that it warrants reconsideration of existing attitudes, and at that point re-evaluate and either maintain or change their views.
Attitudes, this means, are far less flighty or reactive to individual events than is commonly assumed; for the most part they are, actually, rational. Obama's trip, like everything else he's doing - and ditto for John McCain - are therefore about building a case, not about changing daily numbers (which, at this stage, are fundamentally silly).
Combine Langer's description with the fact that many Americans are "obtaining" information about the candidates at glacial pace, and we should be surprised to see much meaningful change in the polling numbers right now, especially those measuring vote preference.
Update: In the comments, Along makes a good point:
As Mr. Franklin posted on Monday, the polls in 2004 and 2000 began to shift quite a bit right around now--the 100 days out mark. In 1992 Bill Clinton also started rising significantly in mid-July, after the convention. Kerry's summer swoon started just after his convention, Gore's summer rise started just before his. So are you saying the summer movement in those races was not "meaningful change," or that it was, and we should simply be waiting till the conventions grab our attention to see meaningful movement this year?
I meant mostly the latter. The conventions were earlier in previous years. While convention "bumps" may not endure, voters do pay more attention and the information they receive during conventions is important. Nate Silver made a similar observation last night:
Pundits -- including yours truly -- generally exaggerate the speed with which political news reaches a saturation depth in the American electorate. There are a few exceptions -- debates, conventions, and major victories in the primaries can have measurable effects almost immediately, and certainly within the first 48-72 hours. So can DEFCON-2 level controversies like Jeremiah Wright. But most of the things we write about here, or the National Review talks about, or Keith Olbermann talks about, take a long time to penetrate the electorate if they do so at all.
By Mark Blumenthal on August 5, 2008 4:23 PM | Permalink | Comments (6) | TrackBacks (0)
How is Barack Obama doing among low income white voters? Quite well, says the headline and lead of a front page story in today's Washington Post. But before leaping to conclusions, we might want to take a closer look at the complete survey questionnaire.
Under the headline "Obama leads, Pessimism Reigns Among Key Group," The Washington Post tells us that Barack Obama "holds a 2 to 1 edge" over John McCain "among the nation's low-wage workers." The Post, in partnership with the Henry J. Kaiser Family Foundation and Harvard University, interviewed 1,350 randomly selected adults under 65 earning $27,000 a year or less and working at least 30 hours a week. Obama's margin was 56% to 27% among all adult respondents,, slightly more (58% to 28%) among registered voters. But the result getting much attention today came in the second paragraph (emphasis added):
Obama's advantage is attributable largely to overwhelming support from two traditional Democratic constituencies: African Americans and Hispanics. But even among white workers -- a group of voters that has been targeted by both parties as a key to victory in November -- Obama leads McCain by 10 percentage points, 47 percent to 37 percent, and has the advantage as the more empathetic candidate.
"Now this," writes TPM's Greg Sargent, "should put the "Obama's working class whites problem" meme to rest." Perhaps.
Taken at face value, Obama's margins do look strong, even stronger than what John Kerry received four years ago among similar voters according to exit polling. While I cannot precisely replicate the universe sampled by Post/Kaiser/Harvard study, the respondent level exit poll data from 2004 available from the Roper archive get us pretty close. I tabulated results for voters under 65 with incomes under $30,000 a year who said they were employed full time. Those voters supported Kerry by a nearly twenty point margin (59% to 40%), while the white voters in the subgroup divided almost evenly, 50% for Kerry to 49% for Bush.
So a survey showing Obama leading by 10 point among low income white voters would certainly represent an improvement.
But take a closer look at the complete questionnaire that -- to their credit -- the Post published online. The presidential vote preference question (#36) comes (by my count) 59 items and roughly 15 minutes into the interview. Before asking about presidential vote preference the survey probed respondents about their personal financial situation, the state of the American economy, their priorities for the things "the government might do to try to improve people's financial situation." They suggested seven different things "you or someone in your family done in the past year to make ends meet," and asked if any applied.
They also asked respondents if their "personal financial situation" has improved or declined since "George W. Bush took office in 2001" (48% said it had declined, 11% said it improved). And finally, immediately before asking the the presidential vote preference question, they asked:
During the past year, have you or has someone in your family had your overtime or regular hours cut back at work, or not?
During the past year, have you or has someone in your family been laid off or lost your job, or not?
And then they asked which candidate they would be most likely to support. Do we think that priming respondents for 15 minutes about the state of the economy and their own personal financial insecurities would have no impact on their vote preference?
Don't get me wrong. The researchers that designed this study are among the best in the field. The survey itself represents an extraordinary and unparalleled effort to "take a close look" the the lives of low wage workers "and try to understand how they are faring amidst all the economic changes around them," as Washington Post economics correspondent Michael Fletcher puts it in a companion video analysis. Among other aspects of their rigorous methodology, the pollsters used used cell phone interviewing to get an adequate representation of low wage workers living without land line phones.
However, if the pollsters wanted to measure where the presidential vote preferences stand now among low income workers, they should have asked the vote preference question up front. The results they obtained tell us something about how low wage workers might react to a campaign framed entirely on economic and pocketbook issues (a finding that presents an obvious strategy for Obama). As such, its measurement of vote preferences is hypothetical and possibly misleading.
Update: University of Wisconsin-Milwaukee political science professor Tom Holbrook was thinking along the same lines.
By Mark Blumenthal on August 4, 2008 2:58 PM | Permalink | Comments (2) | TrackBacks (0)
Andrew Gellman solves the "nonpuzzle" of close election polls.
Chris Cillizza notes an "attacking unfairly" gap in this week's CNN poll (Pollster reader Gary Kilbride caught it too).
Mark Mellman sees significant changes in Barack Obama's base of support, as compared to Kerry in 2004 and Gore in 2000.
Nate Silver sees similar patterns in the Gallup data.
Jennifer Agiesta delves deeper into how increased black turnout might have affected the 2004 outcome.
Allan and Sheri Rivlin think John McCain needs to articulate a credible economic plan.
Tom Jensen points out the online recording of an automated PPP interview call.
SurveyUSA rounds up their recent polling on the economy.
The Associated Press has questions and answers for polling skeptics.
Late update:
Kathy Frankovic says the economic issues remain dominant, as they were in 1992
By Mark Blumenthal on August 1, 2008 5:49 PM | Permalink | Comments (2) | TrackBacks (0)
"Numbers Guy" Carl Bialik devotes his Wall Street Journal column and a companion blog post today to the subject of the automated "interactive voice response" polling that has become such a staple of the current campaign. Both are well worth reading in full.
Bialik managed to interview most of the major players in the political IVR field, and had a reaction from our partner Charles Franklin, summing up our own philosophy regarding the automated polls (that use a recorded voice rather than a live interviewer, and ask respondents to answer questions by pressing keys on their touch-tone phones):
The automated-polling method, says Charles Franklin, professor of political science at the University of Wisconsin and co-developer of the poll-tracking site Pollster.com, "can prove itself through performance or it can fail through poor performance, but we shouldn't rule it out a priori."
The column notes that IVR pollster SurveyUSA ranks second most accurate among all pollsters during the 2008 primaries in the ratings compiled by Nate Silver and that IVR polling was indistinguishable during the primaries in terms of how the final poll compared to the election result:
Their accuracy record in the primaries -- such as it was -- was roughly equivalent to the live-interviewer surveys. Each missed the final margin by an average of about seven points in these races, according to Nate Silver, the Obama supporter who runs the election-math site fivethirtyeight.com.
Franklin did our own compilation of polls conducted during the final week of the 2006 (for a paper presented at the AAPOR conference last year) and reached essentially the same conclusion.
The article also indicates some cracks may be forming in the intense skepticism that the survey research establishment has long held for IVR surveys. Bialik notes that a polling textbook (The Voters Guide to Election Polls ) authored by Paul J. Lavrakas and Michael Traugott, "refers to these surveys as Computerized Response Automated Polls -- insulting acronym intended." But at the end of the column, Lavarakas indicates a willingness to consider the methodology:
Accepting responses by touch tones may have a particular advantage this election, says Mr. Lavrakas, former chief methodologist at Nielsen Media Research, because it may extract more-honest responses from white respondents about their intent to vote for Sen. Obama. "Ultimately the proof is in the pudding, and those firms that use IVR for pre-election polling and do so with an accurate track record should not be dismissed," he says.
Again, this is good stuff. Word reading in full.
By Mark Blumenthal on August 1, 2008 11:44 AM | Permalink | Comments (1) | TrackBacks (0)
My NationalJournal.com column for the week is now online. This week, I delve a bit deeper into the ongoing debate about what outcome we can expect in November based on the polls of July.
By Mark Blumenthal on July 31, 2008 12:59 PM | Permalink | Comments (5) | TrackBacks (0)
Not sure how we missed this, but Gallup has put up detailed demographic cross-tabulations based on the Gallup Daily tracking (that's the survey showing a four point lead for Barack Obama as of yesterday, not the USA Today Gallup poll that applied the "likely voter" screen and has been a source of controversy this week).
Here are descriptions and links just received via email from Gallup:
Gallup now has available online extensive data breaking out support for the presidential candidates within demographic, partisan, ideological, and regional subgroups.
The data are updated each Wednesday -- based on more than 6000 Gallup Poll Daily interviews conducted each week – and available here: http://www.gallup.com/poll/election2008.aspx. Importantly, the analyses can now be trended back through the week of June 9-15, showing the pattern of support within subgroup across time.
Candidate Support by Gender
Candidate Support by Gender Among Whites
Candidate Support by Age
Candidate Support by Region
Candidate Support by Race
Candidate Support by Education
Candidate Support by Education Among Whites
Candidate Support by Political Party
Candidate Support by Political Party Among Whites
Candidate Support by Political Party and Ideology
Candidate Support by Political Party and Ideology Among Whites
Candidate Support by Church Attendance Among Whites
Candidate Support by Marital Status
Candidate Support by Marital Status and Gender
Candidate Support by "Red," "Purple," and "Blue" States
By Mark Blumenthal on July 31, 2008 9:59 AM | Permalink | Comments (10) | TrackBacks (0)
Quinnipiac University
7/23-29/08
Mode: Live Telephone Interviews
Florida: (1,248 LV, 2.8%)
Obama 46, McCain 44 (June: Obama 47, McCain 43)
Ohio: (1,229 LV, 2.8%)
Obama 46, McCain 44 (June: Obama 48, McCain 42)
Pennsylvania: (1,317 LV, 2.7%)
Obama 49, McCain 42 (June: Obama 52, McCain 40
By Mark Blumenthal on July 31, 2008 8:33 AM | Permalink | Comments (20)
The most important and useful campaign related data you will read today was released this morning by the Wisconsin Advertising Project. It tells us precisely what the two presidential candidates and political party organizations have been spending on television advertising in each state from June 3 to July 26. Here are the two key (forgive me) "money" paragraphs:
The McCain ad effort is more narrowly focused with intense attention being paid to four states -- Michigan, Ohio, Pennsylvania, and Wisconsin. McCain is out-advertising the Democratic nominee in these four states where the RNC has also entered the fray. That said, in seven other battleground states where both campaigns are up (Colorado, Iowa, Minnesota, Missouri, New Mexico, Nevada, and West Virginia) the McCain campaign is also out-advertising the Obama campaign.
Despite being out-advertised in nearly all states where both candidates are airing ads, Obama continues to advertise in states that have recently been unfavorable to Democratic presidential candidates. To date, Senator Obama is airing ads in 37 markets where McCain has not aired a single ad, while McCain is advertising in only two markets where Obama is not. Although Florida was the pivotal state in the 2000 presidential election, John McCain has not aired a single ad there since June 3rd. Senator Obama has aired over 7,000 ads in Florida since becoming the presumptive Democratic nominee and has spent more money in Florida than in any other state. Other states where only Obama’s paid advertising message is being heard are Alaska, Georgia, Indiana, Montana, and North Carolina.
The full pdf report has tables with specific data on each state and much more. Simply put, you ought to read recent poll data in the context of what candidates (and party committees) have been spending in each state. The numbers provided by the Wisconsin project are an absolutely invaluable tool.
The data were collected by TNS Media Intelligence/CMAG. I wrote more about CMAG and how it collects advertising data last October.
Update: More in today's New York Times and some observations from Nate Silver.
(H/T Ben Smith).
By Mark Blumenthal on July 30, 2008 11:40 AM | Permalink | Comments (22) | TrackBacks (0)
Ever had a day where everything seemed to fall on your desk at once? Today, for me, has been one of those days, although just one is obvious, and that involves those seemingly contradictory numbers on the presidential race from the Gallup organization. As blogged yesterday, the most important differences are the result of Gallup's well known (and often controversial) "likely voter" model.
If you are a long time reader and remember my old blog, Mystery Pollster, you will remember that over the final weeks of the 2004 campaign, I did a seven-part review of how pollsters select likely voters, including an explanation of the Gallup model and a review of criticism of it. Most of the issues reviewed then are relevant now, but in the context of this latest controversy, let's consider (a) why pollsters try to identify "likely voters," (b) the approach Gallup and USA Today took this week and (c) some thoughts about what it all means about the state of the race.
Why Screen for Likely Voters?
Four years ago, according the website maintained by Professor (and frequent Pollster commenter) Michael McDonald, 122 million Americans cast a ballot for president, which amounts to a turnout of 60% of the eligible adults in the United States (a significant increase from 54% in 2000).
A pre-election survey of all adults that made no effort to identify likely voters would have included the 40% who did not vote, and -- as should be obvious -- those extra interviews create the potential for a error the results since non-voters might have different preferences than actual voters. So all pollsters care about trying to identify the likely electorate.
But there is a big problem: Simply asking respondents whether they plan to vote does not work. Many more Americans will report they are likely to vote, or will claim they have voted in the past, than actually do. Consider the following results from the final national survey conducted by the Pew Research Center
However, when the Pew Research Center conducted their final survey before Election Day 2004 (interviewing 2,804 adults, October 27-30), they found the following:
So screening for just self-identified registered voters is a good idea, but would still include roughly 20% non-voters. And simple questions about past voting or vote intent would largely overstate the size of the electorate. As such, the Pew Center and most other media pollsters used various indirect techniques (with some success) to screen for or otherwise "model" the likely electorate.
How Does Gallup Do it Now?
As explained in an article posted yesterday by Gallup's Frank Newport, the USA Today/Gallup has been using a three question scale to likely voters on the nine surveys they have conducted so far this year that asked a presidential vote question. Actually, that should be four questions, as they first ask adults if they are registered to vote, than ask:
1. How much thought have you given to the upcoming election for president -- quite a lot, or only a little? (quite a lot or volunteer "some" = 1 point)
2. How often would you say you vote -- always, nearly always, part of the time, or seldom? (always or nearly always = 1 point)
3. Do you, yourself, plan to vote in the presidential election this November, or not? ("yes" = 1 point)
They award points as noted above to those who give responses that have shown to correlate strongly, in past elections, with actual turnout. How do they know? They have conducted studies in past elections where they checked the voter registration rolls to see which respondents actually voted and which did not (how long ago? - I'm not sure if Gallup has disclosed that).
When they scored the three questions, they found that 56% of adults scored a perfect 3 on the 1-3 scale, answering all three questions as a highly likely voter would. The next category -- those scoring at least 2 out of 3 -- amounted to another 17% of adults, which would add up to 73%. But Gallup wanted their likely voter tabulations to "model" a turnout of 60% of adults, so they weighted down the "2s" to (those getting 2 out of 3 points) to a little less than one third of their original value.
I am leaving out a few details (involving Gallup's standard demographic weighting) but that is the gist of it: Likely voters are the 3s on their likely voter scale plus the 2s weighted down to roughly a third of their original value.
Gallup will use more or less the same procedure in the surveys they conduct in September and October, except that they will add four more questions to the scale (involving past voting behavior, knowledge of their polling place and a ten point scale to rate vote intent.
What are the problems here? First, as should be obvious, this is not the most precise method of identifying a true likely voter. If you are not yet registered, you are not included. If you registered this year for the first time and respond honestly that you have never voted before, your preferences are weighted down by a factor of 2 as compared to other voters or thrown out altogether if you say you are not paying much attention to the campaign.
Second, as Robert Erikson and his colleagues reviewed in the pages of Public Opinion Quarterly four years ago, the classic 7-question Gallup model "exaggerates" reported volatility in ways that are "not due to actual voter shifts in preference but rather to changes in the composition of Gallup's likely voter pool" (also summarized here).
Third, as Mike McDonald points out in a comment earlier today, a higher than ever turnout will challenge these models and their assumptions. Other surveys continue to show a huge Democratic advantage on measures of supporter "enthusiasm" for the two candidates. Those measures have not previously been included in the Gallup-style model, but they may be important this year.
Fourth, and this is the really important one, no one knows how accurate this technique is in terms of predicting turnout in November based on an application to survey data gathered in July. We have a lot of evidence that the Gallup-style "cutoff model," clunky as it may seem, does make surveys more accurate when applied to data collected the week before the election. But I have yet to see any comparable evidence regarding data collected in July.
So I tend to agree with Gallup's Frank Newport when he told Jill Lawrence yesterday that "'registered voters are much more important at the moment,' because Election Day is still 100 days away." For now, the poll of self-identified registered may be too broad a representation of the likely electorate, but at least they allow for a consistent measurement. Looking at vote preference among typically higher turnout subgroups is useful, analytically, but may or may not improve our conception of where the race stands.
So What Do These Results Say About Where the Race Stands?
First, to put the question as several readers did in emails over the last 24 hours? So which poll or approach is the most accurate right now? Listen closely now: We. Don't. Know.
If the election were being held today, past evidence would argue for placing more trust in the Gallup "likely voter" model than in the preferences of registered voters. But the election is not today, and I am not convinced that any pollster has a monopoly on wisdom when it comes to predicting turnout 100 days out.
As Brian Schaffner (our new contributor) reminds us, if we look at all the recent polls, and not just one, we can stills say with considerable confidence that Barack Obama is ahead. The precise margin probably depends on what assumptions one makes about turnout, which is more art than science at this point. However, as progressive blogger Chris Bowers has been pointing out lately, it is far better to be ahead than behind.
Having said that, we should not discount that two recent polls -- USAToday Gallup and ABC/Washington Post -- show McCain doing better when the classic Gallup "likely voter" model is applied. What is truly interesting about that finding is that the opposite was true on six of seven surveys that Gallup conducted from January to May: Obama did better slightly better among "likely voters" (defined as they were above) than among registered voters.
I have a theory (that someone at Gallup can probably test empirically): What changed is that the Democratic primaries ended. From February to June, Republicans who usually vote had a perfectly good reason to say they were paying "only a little attention" to the presidential campaign. All of the news what about the Obama Clinton race. Now that the media has started to focus on the McCain-Obama contest, Republicans have greater reason to be engaged. At least, that is something worth checking.
Also, finally, consider something from the perspective of a no-longer-practicing campaign pollster: Campaigns matter. So I am less concerned at this stage about "projections" that predict the outcome than in understanding what each campaign needs to accomplish to win. If the "likely voter" pattern evident in the recent USA Today/Gallup and Washington Post/ABC polls is accurate, it tells what the Obama campaign needs to do to win this election: They need to mobilize Americans that are ready to support Obama but that do not typically vote. That comes through loud and clear.
**PS - My conclusions above raise an obvious question: If registered voters are a better subgroup to watch, why does Pollster.com use the likely voter numbers on our tables and charts? I will blog on that highly pertinent question next, I promise.
[Typo
