November 10, 2009
By Guest Pollster
Adam Geller is the CEO of National Research, Inc. and conducted polling for Chris Christie's campaign in New Jersey this year.
I'd like to contribute a few thoughts on the performance of the public polls during the recently concluded New Jersey Gubernatorial race. On this topic, I bring a unique perspective, as the pollster for the Christie campaign, and I'd like to offer my thoughts not as any type of authority, but rather to contribute to an important professional discussion.
I should mention that, for what it's worth, some observers may have been surprised by the results on November 3rd, but neither Governor Elect Christie nor his advisers were surprised.
Before the cement hardens and ink dries on the post election wrap up, let me offer the following five thoughts:
- The automated polls were more accurate than the live interview public polls, due in part to the methodology of the live interview polls.
From polls that were in the field for an entire week (Quinnipiac) or even longer (FDU), to polls that oversampled Democrats (Democracy Corps, among several others) to polls that asked every single name in the ballot (Suffolk), an essential reason for the poor performance of the live interview polls had less to do with the fact that a live person was administering the poll and more to do with methodological issues.
- The partisan spread in the polls ought to be reported up front.
Some public pollsters make it difficult to determine how many Republicans, Democrats and unaffiliated voters they interviewed. Why not just put it into the toplines? Reporters and bloggers should demand this before they report on the results. Not to pick on Quinnipiac, but they had Corzine and Christie winning about the same amount of their own partisans, and they had Christie winning Independents by 15 percentage points, and yet they STILL had Christie trailing overall by 5 points. Quinnipiac did not publish their partisan spread, but then an astute blogger was able to ascertain the fact that there were, in fact, too many Democrats in the sample. Other polls, notably Democracy Corps, regularly produced samples with too many Democrats (though, in their parlance, some of these were "Independent - Lean Democrat"). That their sample was loaded up with Democrats had the obvious effect on their results. Whether this was intentional or not, I would leave to others to speculate.
- In general, RDD methodology is a bad choice in New Jersey, if the goal is predictive accuracy.
In New Jersey, there are many undeclared voters (commonly but mistakenly referred to as Independents). These undeclared voters identify themselves as Republicans or Democrats - even though they are not registered that way. In our polls, we frequently showed a Democrat registration advantage that matched their actual registration advantage - but when it came to partisan ID, the spread was more like a six point Democrat advantage. By using a voter list, we knew how a respondent was registered - and by seeing how they ID'ed themselves, we gained insight into the relative behavioral trends of undeclared voters and even registered Democrats who were self identifying as Independents. Public pollsters who dialed RDD missed this. Partisan identification in New Jersey is not enough, if the goal is to "get it right."
- The public polls oversampled NON voters.
Again, this is a function of RDD versus voter list dialing. It is easy for someone to tell a pollster they are "very likely" to vote. With no vote history and no other nuanced questions, the poll taker has little choice but to trust the respondent. Pollsters who use voter lists have the benefit on knowing exactly how many general elections a respondent may have voted in over the past five years, or when they registered. By asking several types of motivation questions, the pollster can construct turnout models that will have a better predictive capacity. The public polls did not seem to do this.
To this end, we had heard all about the "surge strategy" that the Corzine campaign was going to employ. This refers to targeting "one time Obama voters" and driving them out in force on election day. With voter lists, we were easily able to incorporate some "surge targets" into our sample. After running our turnout models, we saw no evidence that the surge voters would be game changers.
- The Daggett effect was overstated in the public polls.
Conventional wisdom holds that Independent candidates underperform on election day. But the reality is, many analysts could have easily predicted Daggett's collapse, based not on history, but on simple a simple derivative crosstab: for example, voters who were certain to vote for Daggett AND had a very favorable opinion of him. They could have asked a "blind ballot" where none of the candidate choices were read. We did these things - and we estimated Daggett's true level of support to be around 6%.
None of this is meant to pick on the "live interview" public pollsters. For the most part, these polls are conducted and analyzed by seasoned research professionals. But in non-Presidential years, RDD methodology can lead to inaccurate results, which can then lead to inaccurate analysis. It is tough to conclude that the automated polls are somehow superior to live interview polls, given the methodological issues I've outlined.
What does it mean for next year? At the very least, journalists, bloggers and reporters need to ask more questions about the methodology and construction of the poll sample. They need to understand the partisan spread, and the extent to which it conforms to reality. They need to know how long the survey was in the field. They also need to beware of polls being released that are designed to manipulate opinion rather than manage it. They need to ask if certain polls are being constructed to reflect what is happening, or if they are being constructed to reflect what the poll sponsor would LIKE to happen. The public polls add to the dialogue, and given their ever increasing contributing role, we all ought to be more demanding when reporting their results.
By Guest Pollster on November 10, 2009 1:30 PM
| Permalink
| Comments (6)
| TrackBacks (0)
October 27, 2009
By Guest Pollster
Humphrey Taylor is chairman of the Harris Poll at Harris Interactive, which conducts surveys on the internet.
These comments are prompted by the paper Comparing the Accuracy of RDD Telephone Surveys and Internet Surveys Conducted with Non-Probability Samples by Yeager, Krosnick, et al, and by Mark Blumenthal's two excellent articles in the National Journal reviewing their paper.
The paper's conclusions were based on a comparison between six "benchmarks" and the findings of the various polls they examined. They assumed that the benchmarks were perfectly accurate, and that any differences between the polls and the benchmarks were "errors." I believe that this is not the case and that some of the benchmarks were inaccurate because of the social desirability bias that is often found in surveys where respondents are interviewed, by telephone or in-person, by live interviewers.
Social desirability bias occurs where respondents are not comfortable telling interviewers the truth because they are embarrassed to do so, or where their behavior or attitudes may be seen as unethical, immoral, anti-social or illegal.
Our online surveys have always found substantially more people than our telephone surveys who tell us they are gay, lesbian or bisexual (by a 3-to-1 margin). Our online surveys also find fewer people who claim to give money to charity, clean their teeth, believe in God, go to religious services, exercise regularly, abstain from alcohol, or drive under the speed limit.
Furthermore, in-person surveys by the Census Bureau report substantially more people claiming to have voted in elections than actually voted. If there is a better explanation than social desirability bias, I haven't heard it.
This conclusion - that surveys with live interviewers underreport "socially undesirable" behavior is supported by the data used by Yeager et al.
Our online survey, used by Yeager, found more smokers and more people having had 12 drinks in a life time than either the benchmark surveys conducted by government agencies or the RDD sample (and our own telephone surveys). Our online survey found that (to the nearest whole number) 28 percent were smokers compared to 26 percent in the RDD sample and 22 percent in the benchmark survey. Our online survey found only eight percent who had not had 12 drinks in their lifetime compared to 15 percent in the RDD sample and 23 percent in the benchmark survey.
Another government study, the NHANES study reported that 24.9 percent of adults said they were smokers but that blood tests showed that an additional 4.5 percent had smoked in the previous 24 hours but had not reported it when asked by an interviewer. The resulting NHANES estimate of 29 percent is closer to our estimate of 28 percent than to Knowledge Network's 26 percent or the RDD sample's 24 percent.
Two of the six benchmarks used by Yeager et al come from government sources where one would not expect to find any social desirability bias. In both cases, the Harris Interactive data were slightly closer to the benchmark data than were the findings of the RDD telephone survey. Our surveys found 28 percent of adults with passports compared to 30% for the RDD sample and the 23 percent in benchmark. Our survey found 92 percent having a driver's license compared to 93 percent in the RDD sample and the 89 percent benchmark.
In addition to the presence or absence of live interviewers there is one other reason why our online polls may have less social desirability bias than most telephone and in-person surveys. Our panel members have agreed in advance to be surveyed, which suggests that they trust us with confidential information, and are therefore more likely to tell the truth.
All this evidence suggests that the Harris Interactive data used by Yeager et al is generally more accurate than the RDD sample and that some of the so-called benchmarks probably overstate socially desirable behaviors because they were obtained in surveys with interviewers.
By Guest Pollster on October 27, 2009 3:09 PM
| Permalink
| Comments (0)
| TrackBacks (0)
October 8, 2009
By Guest Pollster
This guest contribution comes from Michael McDonald, an Associate Professor of Government and Politics in the Department of Public and International Affairs at George Mason University and a Non-Resident Senior Fellow at the Brookings Institution.
Saturday Night Live's sketch mocking Obama prompted CNN to run a story stating that the 'SNL' Obama sketch marks end of [Obama's] honeymoon. Actually, SNL is not leading public opinion here. Polling suggests that Obama's honeymoon ended in early August. Since then, Obama's job approval rating has remained essentially flat.
If you are an Obama supporter, you might ask how this is possible, since an Oct. 1-5 AP-GfK survey shows a resurging six percentage point increase in support for Obama since their Sept. 3-8 survey. Or, if you oppose Obama, you might point to the slight downward trend in Obama's job approval among all polling firms from early September clearly evident on Pollster.com.
What is going on here is that Pollster.com's trend line behaves fine when there are lots of polls to average together, but it does not work as well when two daily tracking polls are averaged together with more sporatic national polling. The two daily tracking polls - Gallup and Rasmussen - consistently find lower Obama job approval ratings than other polling firms. In addition to these two daily tracking polls, there are approximately bi-monthly internet polls from YouGov/Polimetrix and Zogby that also consistently show lower Obama job approval numbers compared to other polls.
These so-called "house effects" whereby different pollsters consistently report different numbers is well-known. I do not want to get sidetracked into speculation about why these polls have lower numbers, since we really cannot know what the true population value is for Obama's job approval rating.
What is interesting is what happens when these polls are disaggregated into two types (1) the tracking and internet polls and (2) all other polls.
To examine the first type of polls, let's use Pollster.com's filter tool to include all internet polls and the two daily tracking polls.
According to this trend estimate, Obama's job approval rating leveled out in early August at about 50 percent, and may be slightly increasing since.
To examine the second type of polls, let's use Pollster.com's filter tool to exclude all internet polls and the two daily tracking polls.
According to this trend estimate, Obama's job approval rating leveled out in early August at about 53 percent.
Seen in this light, Obama's job approval rating has remained steady since early August, and it is here that Obama's honeymoon likely came to an end. Most pollsters took a vacation during August, except those conducting the first type of polls, which show lower Obama job approval than the second type. The bump up in Obama's job approval at the beginning of September is an artifact of the increased number of the second type of polls conducted when Obama delivered his health care speech to Congress. Subsequently, the absence of the second type of polls allows the first type of polls to again dominate the trend line, thereby giving the appearence that Obama's approval is now decreasing from the (non-existent) short-term early-September rally. The different mixes of the first and second types of polls are confounding the trend line and incorrectly coloring perceptions of the direction of Obama's job approval rating. Indeed, if you squint closely at Pollster.com's trend line for all pollsters, you'll see a long-term periodicty that apparently fluctuates along with the mix of the first and second types of polls.
[Editor's Note: So that Professor McDonald's commentary will always match the graphics, we replaced the embedded, interactive version of charts with screenshots, although you can click the link above each chart to see the most recently updated version with the filtered polls he selected].
By Guest Pollster on October 8, 2009 4:38 PM
| Permalink
| Comments (0)
| TrackBacks (0)
September 10, 2009
By Guest Pollster
Robert Y. Shapiro is a professor of political science at Columbia University who specializes in public opinion, policymaking, political leadership, and mass media. He is a member of the board of directors of the Roper Center for Public Opinion Research.
The polling and pundit world is now looking to see if President Obama's
speech will rally public support for his health care reform plan. In
addition to looking at the stream of polls that will now follow, I direct your attention, hot off the presses, to the latest issue of the journal Political Communication. A timely article by Brandon Rottinghaus provides a broader political science view on presidential efforts to influence public opinion. What we know from George Edwards' book, On Deaf Ears: The Limits of the Bully Pulpit (Yale, 2003), is that it is difficult for presidents to succeed at influencing public opinion. However, Rottinghaus's article provides evidence for why Obama correctly chose to take his best shot in a nationally televised speech.
The article
uses "a comprehensive data set spanning 1953 to 2001," to examine
"several strategic communications tactics through which the presidents
might influence temporary opinion movements." Specifically, it finds
that "presidential use of nationally televised addresses is the most
consistently effective strategy to enhance presidential leadership, but
the effect is lessened for later serving presidents." In contrast,
other strategies such as those involving domestic travel do not have
positive effects and "televised interactions"--press conferences and the like - tend to have negative effects. While some may not be surprised with these findings, it is good to have empirical evidence to wrestle with.
But getting to the point, how will this now play out for Obama? My sense is that Obama's speech will come out on or above average in impact, though there is a question of what its half-life will be. What I see as most important, however, is not the new polls that we will soon see (if they are not out already). Putting Rottinghaus' article aside, what will count most is not what the public thinks at this moment, but rather the extent to which Democratic leaders unite around Obama's plan (which may well be close to Baucus'?); it is this elite consensus that will enable any positive effect of the speech to last or even widen. This assumes that the consensus will be more salient and striking than any continued Republican opposition.
Echoing the famous political scientist, V.O. Key, what matters more than the immediate polls is political leadership more broadly. The speech itself is the start of what could be a stronger consensual message than we have seen to date from Democratic and potentially other political leaders. The relevant public opinion research comes from Richard Brody's book on presidential leadership, (Assessing the President: The Media, Elite Opinion, and Public Support. Stanford, 1991), John Zaller's seminal book on public opinion (The Nature and Origins of Mass Opinion, Cambridge, 1992), and what Ben Page and I examined (The Rational Public. Chicago, 1992).
Larry Jacobs and I (Politicians Don't Pander, Chicago, 2000) looked at the President Clinton's 1993-94 health care reform effort from this perspective. What happened there was the Democratic leaders never supported any Clinton plan, and this, along with the strong Republican leadership opposition caused the public to become apprehensive and turn against health care reform. This happened much earlier in the legislative process than what occurring now, as the Clinton plan got to Congress later in Clinton's first term. In contrast, we are at that same juncture now --- however, earlier in Obama's first term but later in the legislative process, as there are now actual bills that have made it through congressional committees. Clinton never made it that far. The Democrats now have a better chance than Clinton did, since at this moment they are poised to unite around a president's plan. But if they don't do that quickly, then it's 1994 all over again. If by all appearances they come together, they can prevent public support from tapering off and very likely increase it.
In the end, Obama may have timed his entry into the fight just right--it's earlier than when Clinton entered the actual legislative fray in 1994--and this may have been the only way he could have gotten a major health care reform bill through. Given the financial crisis, the stimulus bill, and the two wars, he may well have been stopped in his tracks earlier on--without the health care reform bills making it through multiple committees as they have. He needed to enter the fight when he could rally congressional support in both houses, with drafted legislation in hand and already substantially debated. Of course we will never know since as we can't replay history. For now, the main point is don't just watch the polls-watch the leaders. The public will not just be responding to Obama but to the extent to which he has liberal, blue dog, and any (albeit unlikely) Republican leadership support.
By Guest Pollster on September 10, 2009 10:09 PM
| Permalink
| Comments (1)
| TrackBacks (0)
By Guest Pollster
Today's Guest Pollster article comes from Thomas Riehle, a Partner of RT Strategies.
Technological capabilities can become temptations to conduct research studies that add nothing to our knowledge of public opinion, just because we can. Get thee behind me, Satan!
For example, it would be no problem, technologically, to display squiggly lines with the moment-by-moment reactions of a panel of viewers to the blathering of the talking heads on news show panels. The Onion demonstrates what a mess that would be, in a parody entitled "New Live Poll Allows Pundits to Pander to Viewers in Real Time."
What would happen if we let the talking heads see whether viewers at home agreed or disagreed with what they were saying, "using the Insta-Poll Tracker on our web site"? The talking heads would become self-conscious about the direction of their own squiggly line and start tailoring their statements...word by word...to make the squiggly line go up.
Insta-polls like September 9th's CNN/Opinion Research Corporation poll of adults who watched President Barack Obama's address to Congress may have a similar effect on poll respondents. Mark Blumenthal correctly points out the age-old problem of such polls--the partisan make-up. Last night, the audience for this address was heavily weighted with Obama supporters rallying to watch their leader, supplemented with a few civic-minded Americans who would watch any Presidential address, regardless of their own partisanship. Of the 427 adults in this study, all of them interviewed September 5-8 in advance of the speech, and all of whom indicated both an intention to watch the speech and a willingness to be re-interviewed after the speech, 18% were Republicans, 45% Democrats. These kinds of post-speech poll samples always skew heavily in favor of the speaker. Pollster.com's report on this poll last night squeezes out what knowledge can be gleaned by comparing the "bump" among this group of speech watchers to the bump registered among similarly situated groups of speech watchers in the past.
The problem with this kind of insta-poll may be exacerbated when the study is designed, as this one was, to compare the pre-speech responses of speech watchers to opinions after the speech. In the pre-speech survey, I would guess that respondents would strive to express their opinions as forthrightly as possible, as most survey respondents do. In the follow-up poll after the speech, however, I am afraid respondents would be like the Onion's self-conscious pundits. They'd be aware that they are about to become as much a part of the story as South Carolina Republican Rep. Joe Wilson who heckled the President. They'd tailor their answers to make their leader look good. Drawing much of a conclusion from their answers would not be any fairer than judging the entire Republican caucus by the boorishness of a few Members.
By Guest Pollster on September 10, 2009 3:44 PM
| Permalink
| Comments (0)
| TrackBacks (0)
September 6, 2009
By Guest Pollster
Douglas Rivers is president and CEO of YouGov/Polimetrix and a professor of political science and senior fellow at Stanford University's Hoover Institution. Full disclosure: YouGov/Polimetrix is the owner and principal sponsor of Pollster.com.
I woke up on Tuesday morning to find several emails pointing me to Gary Langer's blog posting, which quoted extensively from a supposedly new
paper by Jon Krosnick. These data and results appeared previously in a paper, "Web Survey Methodologies: A Comparison of Survey Accuracy," Krosnick coauthored with me and presented at AAPOR in 2005. The "new" paper has added some standard error calculations, some late arriving data, and a new set of weights, but the biggest changes in this version are a different list of authors and conclusions.
The 2005 study compared estimates from identical questionnaires fielded to a random digit dial (RDD) sample by telephone, an Internet-based probability sample, and a set of opt-in panels. Of these, Internet probability sample had the smallest average absolute error, followed closely by the RDD telephone survey, and the opt-in Internet panels were around 2% worse. In his presentation of our paper at AAPOR in 2005, Krosnick described the results of all the surveys, both probability and non-probability, as being "broadly similar." My own interpretation of the 2004 data, similar to James Murphy's comment on AAPORnet, was that although the opt-in samples were worse than the two probability samples, the differences were small enough--and the cost advantage large enough--to merit further investigation. Even if it were impossible to eliminate the extra 2% of error from opt-in samples, they could still be a better choice for many purposes than an RDD sample that cost several times as much.
Krosnick now concludes that "Non-probability sample surveys done via the Internet were always less accurate, on average, than probability sample surveys" and, tendentiously, criticizes "some firms that sell such data" who "sometimes say they have developed effective, proprietary methods" to correct selection bias in opt-in panels.
In fact, the data provide little support for Krosnick's argument. The samples from the opt-in panels were, as we noted in 2005, unrepresentative on basic demographics such as race and education because the vendors failed to balance their samples on these variables, while the two probability samples were balanced on race, education, and other demographics. This is not a result of probability sampling, but of non-probabilistic response adjustments. It is too late to re-collect the data, but the solution (invite more minorities and lower educated respondents) doesn't involve rocket science.
Instead, Krosnick tries to fix the problem by weighting, and concludes that weighting doesn't work. A more careful analysis indicates, however, that despite the large sample imbalances in the opt-in samples, weighting appears to remove most or all selection bias in these samples. Because the samples were poorly selected, heavy weighting is needed and this results in estimates with large variances, but no apparent bias. In fact, if we combine the opt-in samples, we can obtain an estimate with equal accuracy to the two probability samples.
First, consider the RDD telephone sample. The data were collected by SRBI, which used advance letters, up to 12 call attempts, $10 incentives for non-respondents, and a field period of almost five months. Nonetheless, the unweighted sample was significantly different from the population on ten of the 19 benchmarks. RDD samples, like this one, consistently underrepresent male, minority, young, and low-education respondents. These biases are reasonably well understood and, for the most part, can be removed by weighting the sample to match Census demographics.
Next, consider the Probability Sample Internet Survey, conducted by Knowledge Networks (KN). The unweighted sample does not exhibit the skews typical of RDD. How is this possible, since the KN panel is also recruited using RDD? Buried in a footnote is an explanation of how KN managed to hit the primary demographic targets more closely than SRBI (which had a much better response rate). The answer is that "The probability of selection was also adjusted to eliminate discrepancies between the full panel and the population in terms of sex, race, age, education, and Census region (as gauged by comparison with the Current Population Survey). Therefore, no additional weighting was needed to correct for unequal probabilities of selection during the recruitment phase of
building the panel." That is, the selection probabilities that are
supposedly so important to probability sampling were not used because they would have generated an unrepresentative sample!
The opt-in panels, for the most part, were not balanced on race and education. Only one of the opt-in samples, Non-Probability Sample Internet Survey #6 actually used a race quota. Another, the odd Non-Probability Internet Sample #7, claims to have sent invitations proportionally by race and ended up with 46% of the sample white, despite a 51% response rate. (This survey will be excluded from subsequent comparisons.) Non-probability Sample Internet Survey #1 involved large over-samples of African Americans and Hispanics. I could find no explanation of how Krosnick dealt with the oversamples in the 2009 paper, but it should either match exactly (if the conventional stratified estimator is used) or be far off (if the data are not weighted). In fact, the proportion of whites and Hispanics is off by 1% to 2%.
The selection of a subsample of panelists for a study is critical to the accuracy of opt-in samples. Regardless of how the panel was recruited, the combination of nonresponse or self-selection at the initial stage along with subsequent panel attrition, will tend to make the panel unrepresentative. In 2004, we instructed the panel vendors to use their normal procedures to produce a sample representative of U.S. adults. The practice then (and perhaps now for some vendors) was to use a limited set of quotas. If you didn't ask most opt-in panels to use race or education quotas, they wouldn't use them.
Even without correcting these obvious imbalances, the opt-in samples
provided what most people would consider usable estimates for most of
the measures. For example, the percentage married (unweighted) was between 53.7% and 61.5% vs. a benchmark of 56.5%). The percentage who worked last week (unweighted) was between 53.6% and 63.1% (vs. a benchmark of 60.8%). The percentage with 3 bedrooms (unweighted) was between 41.2% and 46.1% (vs. a
benchmark of 43.4%). The percentage with two vehicles (unweighted) was
between 40.1% and 46.9% (vs. a benchmark of 41.5%). Home ownership
(unweighted) was between 64.8% and 72.8% (vs. a benchmark of 72.5%).
Has one drink on average (unweighted) was between 33.8% and 40.2% (vs.
a benchmark of 37.7%). The KN sample and phone samples were better,
but the difference was much less than I expected. (Before doing this
study, I thought the opt-in samples would all look like Non-
probability Sample Internet Survey #7.)
The 2009 paper attempts to correct these imbalances by weighting, but the weighted results do not show what Krosnick claims. He uses raking (also called "rim weighting") to compute a set of weights that range from .03 to 70, which he then trims at 5. The fact that the raking model wants to weight a cell at 70 is a sign that something has gone wrong and can't be cured by arbitrarily trimming the weight. If there really are cells underrepresented by a factor of 70, then trimming causes severe bias for variables correlated with the weight and not trimming causes the estimates to have large variances. In either case, the effect is to increase the mean absolute error of estimates.
The fact that the trimmed and untrimmed weights have about the same average absolute error does not mean that weighting is unable to remove self-selection bias from the sample. The mean absolute error is a measure of accuracy. It is driven by two factors: bias (the difference between the expected value of the estimate and what it is trying to estimate) and variance (the variation in an estimate around its expected value from sample to sample). The usual complaint about self-selected samples is that you can never know whether they will be biased or the size of the bias. Inaccuracy due to sampling variation can be reduced by just taking a larger sample. Bias, on the other hand, doesn't decrease when the sample size is increased.
Obviously, uneweighted estimates from these opt-in samples will be biased because the vendors ignored race and education when selecting respondents. This wouldn't have been difficult to fix, but it wasn't done. Apparently very large weights are needed to correct demographic imbalances in these samples, but the large weights give estimates with large variances and, hence, a high level of inaccuracy. If one tries to control the variance, as Krosnick does, by trimming the weights, then the variance is reduced at the expense of increased bias. The result, again, is inaccuracy. We are asking the weighting to do too much.
A simple calculation shows that all of Krosnick's results are consistent with the weighting removing all of the bias from the opt-in samples. One way to combat increased variability is to combine the six opt-in samples. Without returning to the original data, a simple expedient is to just average the estimates. Since the samples are independent and of the same size, the average of 6 means or proportions should have a variance about 1/6 as large as the single sample variances. The variance is approximately equal to the square of the mean absolute error which, after weighting, was about 5 for the opt-in samples, implying a variance of about 25. If there is no bias after weighting, then the variance of the average of the estimates should be 25/6 or approximately 4, implying a mean absolute error of about 2%.
How does this prediction pan out? If we average each of the weighted estimates and compute the error for each item using the difference between the average estimate and the benchmark, the mean absolute error for the opt-in samples is 1.4% -- almost identical to the mean absolute error for each of the weighted probability samples. That is, the amount of error reduction that comes from averaging the estimates is about what would be predicted if the all bias could have been removed by weighting. Thus, the combination of these six opt-in samples gives an estimate with about the same accuracy as a fairly expensive probability sample (which also required weighting, though not as much).
There is no reason, however, why you should need six opt-in samples to achieve the same accuracy as a single probability sample of the same size. If the samples were selected appropriately, then we could avoid the need for massive weighting. It is still an open question what variables should be used to select samples from opt-in panels or what the method of selection should be. In the past few years, we have accumulated quite a bit of data on the effectiveness of these methods, so there is no need to focus on a set of poorly selected samples from 2004.
Probability sampling is a great invention, but rhetoric has overtaken reality here. Both of the probability samples in this study had large amounts of nonresponse, so that the real selection probability--i.e., the probability of being selected by the surveyor and the respondent choosing to participate--is not known. Usually a fairly simple nonresponse model is adequate, but the accuracy of the estimates depends on the validity of the model, as it does for non-probability samples. Nonresponse is a form of self-selection. All of us who work with non-probability samples should spend our efforts trying to improve the modeling and methods for dealing with the problem, instead of pretending it doesn't exist.
By Guest Pollster on September 6, 2009 8:14 PM
| Permalink
| Comments (3)
| TrackBacks (0)
August 27, 2009
By Guest Pollster
Prof. Alan Reifman teaches social science research methodology at Texas Tech University, and is compiling the results of public opinion polls on the specifics of health care reform at his blog, Health Care Polls.
There's been a lot of discussion of how seniors, who already are on Medicare, appear to be the least supportive age group of President Obama and the Democrats' plans for enacting health care reform. Seemingly at the center of seniors' concerns is the idea of cutting federal support for a program called Medicare Advantage. According to a Los Angeles Times article:
Although scaling back payments would have no effect on a sizable majority of Medicare users, it would create an opening for opponents to make the blanket allegation that the president wants to cut back on Medicare benefits -- as some Republicans are already starting to say.
Also, of course, seniors were more likely to vote for John McCain in last year's presidential election than were younger voters, who went overwhelmingly for Obama.
The diagram below (which you may click on to enlarge) compares different age groups' attitudes toward health care reform in four recent polls. Compiling these percentages was not as easy as I thought it might be, for a variety of reasons. First, only some pollsters make a public release of cross-tabulations between demographic characteristics and health care-related attitudes (other pollsters reserve such cross-tabs for paid subscribers). Second, age cross-tabs on a common attitude item were not always available. My plan was to use general favor/oppose items toward Obama and the Democrats' reform plan, but such an item was not always available so I had to substitute other types of items, as described below. Third, different pollsters use different cut-points to create their age groups. There's always a youngest age group, for example, but some pollsters bracket it from 18-29 whereas others use 18-34; similar discrepancies exist for other age groups, as well.
Having said all this, the pattern of seniors showing the least support for Obama/Democratic reform plans is clear and well replicated. For any given color of bar (purple, light blue, green, or orange; each representing a different pollster and question), the shortest height is with the seniors.
One other thing to notice is that two polls, ABC/Washington Post and The Economist/YouGuv, only reported on a 30-64 broad middle-age group rather than having two groups like other pollsters; whether groups in the lower and upper halves of the 30-64 age range were combined because they did not differ much in their responses, or the pollsters never broke 30-64 year-olds into smaller subsets, I don't know. For these two polls, I have taken the percentage on the respective attitude measures attributed to 30-64 year-olds and plotted them twice (linked by a light-blue or green horizontal line), where a 30s-40s group and a 50s-60s group would ordinarily go. Now that these "housekeeping" matters are out of the way, here are the question wordings used:
Survey USA (Aug. 19): “Now I am going to tell you more about the health care plan that President Obama supports and please tell me whether you would favor or oppose it. The plan requires that health insurance companies cover people with pre-existing medical conditions. It also requires all but the smallest employers to provide health coverage for their employees, or pay a percentage of their payroll to help fund coverage for the uninsured. Families and individuals with lower- and middle-incomes would receive tax credits to help them afford insurance coverage. Some of the funding for this plan would come from raising taxes on wealthier Americans. Do you favor or oppose this plan?”
ABC/Washington Post (Aug. 13-17): “Reform’s supported by 58 percent of adults under age 30, but 44 percent of 30- to 64-year-olds and just 34 percent of seniors, apparently concerned about its potential impact on Medicare” (this quote comes from an article and does not depict the actual survey item).
Economist-You Gov (Aug. 16-18): “If President Obama and Congress pass a health care reform plan, do you think you personally would receive better or worse care than you receive now?" (% Saying Better).
Kaiser Family Foundation (Aug. 4-11): “Do you think you and your family would be better off or worse off if the president and Congress passed health care reform, or don’t you think it would make much difference?” (% Saying Better).
The four polls above were not the only ones that made some type of age-related comparison. Others did, as well, but their age groupings and/or survey items appeared non-comparable in some way to the four polls whose results I plotted. Two additional polls are as follows:
A Harris Interactive poll used what I think are the most interesting age-group descriptors (shown in Table 2 of the linked document): "Echo Boomers (18-32), Gen. X (33-44), Baby Boomers (45-63), Matures (64+)." Harris plotted the percentage of respondents in each age group who rated Obama's job performance in various issue domains as "fair" or "poor." On health care, higher percentages of Matures (71%) and Gen. X (69%) gave Obama these unflattering ratings than did Echo and Baby Boomers (each 62%). Along with some of the figures from other polls plotted above, this finding from Harris shows a non-linear trend (i.e., support does not decline in perfect progression from the youngest to the oldest voters).
Finally, a Penn, Schoen, & Berland poll released in conjunction with AARP reported only comparisons between respondents younger than 50 and 50-plus. A section of this poll's report entitled "Specific Policy Proposals" (on pages 6-7) is perhaps the most worthy of attention. On most of the items, the younger respondents are more favorably inclined, but on others, there is little or no difference.
(Cross-posted to Health Care Polls)
By Guest Pollster on August 27, 2009 1:10 PM
| Permalink
| Comments (0)
| TrackBacks (0)
August 11, 2009
By Guest Pollster
Prof. Alan Reifman teaches social science research methodology at Texas Tech University, and has begun compiling the results of public opinion polls on the specifics of health care reform at his new blog, Health Care Polls.
Perhaps the most contentious issue among congressional negotiators and interest groups in Washington, DC (and elsewhere) is the so-called public option. The idea is that the government would create a new health-insurance program (modeled to one degree or another on Medicare, the government insurance program for seniors) that people could join. Proponents argue that, by having it compete with private insurers, the public option would help control costs. Opponents, on the other hand, see the public option as yet another government intrusion into an area they feel should be left to the private market.
Where does the public seem to stand? Not surprisingly, the public option has been widely polled, and we shall focus exclusively on it today. As seen in the diagram below (which you can click on to enlarge), levels of support for the public option vary widely according to different polls, despite the relative consistency of question wording (all the survey items refer in some fashion to the public option being a government health-insurance program that would compete with private insurance companies). The predominant trend, I would say, is that a majority of respondents supports a public option, with five of the eight polls showing between 52-66 percent in favor.
Still, though, two other polls show support in the mid-40s and one poll (Rasmussen) has support way down at 35%. What to make of this? Let's start with Rasmussen. Whereas Rasmussen's presidential-election polling has tended to be highly accurate (relative to the actual results), other types of polls from this outfit appear to have had a Republican slant. Here are some examples:
*Whereas most polls tended to have George W. Bush's job-approval ratings during the waning months of his administration in the low-30s or even the 20s, Rasmussen consistently had it around 35%.
*Whereas virtually every pollster other than Rasmussen has shown a majority of voters to prefer the Democrats (at this early point) in next year's U.S. House elections, Rasmussen has been showing the Republicans in the lead (albeit with large percentages undecided).
Polling analysts refer to systematic differences in the results (on the same basic issue) between different survey firms (or survey "houses") as house effects. These may stem from different firms' practices regarding question-wording, sample weighting, etc. On health care reform and other issues, it looks to me as though Rasmussen has a substantial house effect.
There's one other aspect of the public-option polling I'd like to point out. As can be seen in the diagram above, I have highlighted in red the words "option" and "offering" in the wording of some of the survey items. It appears that wordings stressing the voluntariness of the public option (i.e., that it is an "option," or something "offered" to the consumer) tend to elicit higher support than wordings that don't highlight voluntariness as much. This is just a hunch. If anyone has other explanations for the large variation in support between the polls, please share them in the comments section.
(Cross-posted to Health Care Polls)
By Guest Pollster on August 11, 2009 1:51 PM
| Permalink
| Comments (2)
| TrackBacks (0)
July 30, 2009
By Guest Pollster
Today's guest pollster contribution comes from Brendan Nyhan, a political scientist and Robert Wood Johnson Foundation Scholar in Health Policy Research at the University of Michigan. This entry is cross-posted at his blog, Brendan-Nyhan.com.
Matthew Yglesias calls the public "ill-informed and hypocritical" based on a New York Times poll that found "Most Americans continue to want the federal government to focus on reducing the budget deficit rather than spending money to stimulate the national economy... [y]et at the same time, most oppose some proposed solution for decreasing it."
The problem, however, is that the available evidence doesn't support Yglesias's conclusion (which is encouraged by the way the poll is framed in the Times). When you look at the raw poll results (PDF), you'll see that the public prefers reducing the deficit to stimulating the economy 58%-35%, but 53% oppose cuts in public services and 56% oppose higher taxes. Those numbers may seem "ill-informed and hypocritical," but the problem is that we're dealing with aggregate data (this is what is known as an ecological inference problem). We can't draw any strong conclusions about the proportion of individual members of the public who have incoherent preferences about deficit reduction without access to the raw data. Ideally, we would break out the members of the public who advocate deficit reduction over stimulus and see how many of them oppose both higher taxes and reduced services. That's the quantity of interest, but it's unfortunately not available to us at this point.
Update 7/30 12:12 PM: Yglesias has generously updated his post to note that you "can't infer very much about individual preferences from this aggregate data."
By Guest Pollster on July 30, 2009 1:58 PM
| Permalink
| Comments (0)
| TrackBacks (0)
July 27, 2009
By Guest Pollster
Today's guest pollster contribution comes from Brendan Nyhan, a political scientist and Robert Wood Johnson Foundation Scholar in Health Policy Research at the University of Michigan. This entry is cross-posted at his blog, Brendan-Nyhan.com.
Just to briefly elaborate on the point I made last week, here are comparable plots of President Obama's overall job approval and approval of his handling of health care:
As you can see, what's happening on health care is a leading indicator of the end of Obama's honeymoon period. As we return to our normal, highly polarized political climate, most Republicans and Republican-leaning independents will disapprove of a Democratic president's performance in office and his handling of high-salience issues, especially in a bad economy. As a result, Obama's numbers will inevitably decline across the board -- this reality shouldn't be surprising to anyone who works in or reports on politics.
Going forward, we should focus on more important questions. First, how much will Obama's approval numbers decline? Given the state of the economy, it wouldn't be surprising to see him in the low- to mid-40s by the end of the year. Second, what is the distribution of opinion on Obama's handling of health care? Aggregate public opinion on the issue is less relevant than how it's playing in the states of key senators whose votes will determine the fate of the legislation in Congress.
By Guest Pollster on July 27, 2009 11:23 AM
| Permalink
| Comments (0)
| TrackBacks (0)
July 24, 2009
By Guest Pollster
Today's guest pollster contribution comes from Brendan Nyhan, a political scientist and Robert Wood Johnson Foundation Scholar in Health Policy Research at the University of Michigan. This entry is cross-posted at his blog, Brendan-Nyhan.com.
The Washington Post is reporting that a new ABC/WP poll shows a major decline in Sarah Palin's favorability ratings. Her favorables have dropped from a peak of 58% after the GOP convention in September to 40% now, while her unfavorables have surged from a low of 28% to 53% now. Her 40/53 favorable-unfavorable ratio puts her into Hillary/Bush/Cheney territory as one of the most polarizing figures in American politics -- quite an achievement for someone who was a completely unknown less than a year ago.
It's almost impossible to imagine Palin getting the GOP nomination in 2012 at this point (though Intrade still puts the probability at 16%). With numbers like that, her general election prospects are dim, and the Post poll shows growing doubts about her among Republicans as well:
Republicans and GOP-leaning independents continue to rank Palin among the top three contenders in the run-up to 2012, however, with 70 percent of Republicans viewing her in a positive light in the new poll. But her support within the GOP has deteriorated from its pre-election levels, including a sharp drop in the number holding "strongly favorable" impressions of her.
And while Palin's most avid following is still among white evangelical Protestants, a core GOP constituency, and conservatives, far fewer in these groups have "strongly favorable" opinions of her than did so last fall.
...Perhaps more vexing for Palin's national political aspirations, however, is that 57 percent of Americans say she does not understand complex issues, while 37 percent think she does, a nine-percentage-point drop from a poll conducted in September just before her debate with now-Vice President Biden. The biggest decline on the question came among Republicans, nearly four in 10 of whom now say she does not understand complex issues. That figure is 70 percent among Democrats and 58 percent among independents.
Her favorability numbers also stack up extremely poorly against the rest of the expected 2012 field, as this graph illustrates:
The candidates are ordered left to right by their favorable-unfavorable ratio in the most recent poll on Pollingreport.com. As you can see, Palin's numbers are even worse than Newt Gingrich (!) -- the other highly polarizing candidate -- and she has less room to change her image because so many Americans already have an impression of her. By contrast, Romney, Huckabee, Jindal, and Pawlenty start the race without that sort of baggage and are therefore much more likely to make a serious run for the nomination.
To be sure, it's not impossible to come back from numbers like Palin's. Hillary Clinton overcame numbers that were nearly as bad and almost won the Democratic presidential nomination, but she did so with a great deal of hard work and discipline -- qualities that Palin appears to lack. Runner's World photo spreads, feuds with David Letterman, and useless policy op-eds are not going to turn her image around anytime soon.
By Guest Pollster on July 24, 2009 10:25 AM
| Permalink
| Comments (0)
| TrackBacks (0)
June 4, 2009
By Guest Pollster
Patrick Murray is the founding director of the Monmouth University Polling Institute and maintains a blog known as Real Numbers and Other Musings.
There are a couple of pieces of accepted wisdom when it comes to contested primary elections versus general elections: 1) turnout has a bigger impact on the ultimate margin of victory in primaries and 2) primaries are more difficult to poll (see point #1).
The voters who show up for primaries come disproportionately from either end of the ideological spectrum. Even in states with closed primaries (i.e. one has to pre-register with a party to vote in its primary), there is still a particular art for determining which groups of voters should be included in the likely voter sample.
Voters' likelihood to turnout generally correlates with their ideological inclination. Last year's Democratic presidential nomination provides a good illustration of this. Lower turnout caucus states saw a bigger proportion of higher educated liberal activists participate in the process. These same voters also showed up in the primary states, but they were joined by a good number of less educated, blue-collar Democrats. Result: Obama basically swept the caucus states, while Hillary Clinton held her own in the primaries. Texas, which held both a primary and a caucus that were won by different candidates, is a stark illustration of this turnout effect.
The same is true for Republican primaries. Lower turnout means a larger proportion of the electorate will be staunchly conservative in their views. As turnout increases, it's moderates who are joining the fray, thus diminishing the conservative voting bloc's overall power. And with the GOP being in its present ideologically-splintered state, small changes in turnout can have a real impact in primaries cast as battles between the party's ideological factions.
To some extent, we saw this play out in New Jersey's recent gubernatorial primary where the two leading candidates were seen as representing different wings of the Republican party. Former mayor Steve Lonegan cast himself as the keeper of the conservative flame, while former U.S. Attorney Chris Christie claimed to adhere to core conservative principles (e.g. anti-abortion), but presented himself as a more centrist option. New Jersey's Republican voters agreed - a plurality of 47% described Christie as politically moderate while a majority of 56% tagged Lonegan as a conservative.
The Monmouth University/Gannett New Jersey Poll released a poll nearly two weeks before the June 2 primary showing Christie with an 18 point lead over Lonegan - 50% to 32%. New Jersey has a semi-open primary - meaning both Republicans and "unaffiliated" voters are permitted to vote (although unaffiliateds have their registration changed to Republican if they do vote). So, technically about 3.5 million out of New Jersey's more than 5 million registered voters were eligible to vote in the recent GOP primary. But in the last two contested gubernatorial primaries only between 300,000 and 350,000 voters were actually cast.
So, how do you design a sampling frame for that? First, it's worth noting that state voter statistics show that extremely few unaffiliated voters ever show up for a primary - certainly not enough to impact a poll's estimates. So we are left with about one million registered Republicans, of whom still only one-third will vote. That is, of course, IF turnout is typical (more on that below).
Our poll for this primary used a listed sample of registered Republican voters who were known to have voted in recent primaries. It was further screened and weighted to determine the propensity of voting in this particular election (based on a combination of known past voting frequency and self-professed likelihood to vote this year). In the end, our model assumed a turnout of about 300,000 GOP voters, based on turnout in the past two gubernatorial primaries.
However, turnout in other recent GOP gubernatorial primaries in New Jersey have gone as low as 200,000 - that was in 1997 when incumbent Christie Whitman went unchallenged. Turnout in contested U.S. Senate primaries is also generally around the 200,000 level. On the other hand, turnout has been much higher than 300,000 as well. It even surpassed 400,000 as recently as 1981.
The GOP primary saw higher than average turnout in 1993 - another year when a trio of Republicans were vying to take on an unpopular Democratic incumbent. So, it was fair to speculate that Governor Jon Corzine's weak position in the polls would give GOP voters extra incentive to turn out in the expectation of scoring a rare general election win. On the other hand, perhaps the state's Republicans have become so demoralized by their poor standing nationally and 12-year statewide electoral drought that turnout could be lower than the 300,000 used for our poll estimate.
Because we had information on actual primary voting history for each voter in our sample - i.e. rather than needing to rely on notoriously unreliable self-reports - it was possible to re-model the data from two weeks ago with alternative turnout estimates. If the GOP primary turnout model was set well above 430,000 - a 40-year record turnout for a non-presidential race - the Christie margin in our poll grew to 23 points. Alternatively, if the turnout model was pushed down to about 200,000 - a typical U.S. Senate race level - the gap shrank to 13 points. In other words, adjusting the primary poll's turnout estimate from 5% to 12% of eligible voters could swing the results by 10 points!
Why? The analysis showed that "strong" conservatives comprise about half of New Jersey's 200,000 "core" GOP turnout - and this group was largely for Lonegan. But when we widened the turnout estimate, more and more moderates entered the mix. As a result, Chris Christie gained one point on the margin for approximately every 25,000 extra voters who "turned out."
On primary day, Christie ended up beating Lonegan by a respectable 13 point margin - 55% to 42% - on a 330,000 voter turnout. Based on the model above, if Republicans had been a lot less enthusiastic, Lonegan may have been able to narrow this gap to 8 points. On the other hand, record level turnout would have given Christie a 16 or 17 point win.
By Guest Pollster on June 4, 2009 1:01 PM
| Permalink
| Comments (0)
| TrackBacks (0)
June 3, 2009
By Guest Pollster
Alan I. Abramowitz is the Alben W. Barkley Professor of Political Science at Emory University in Atlanta, Georgia. He is also a frequent contributer to Larry Sabato's Crystal Ball.
On May 15th, the Gallup Poll reported what they described as a significant shift in Americans' attitudes on the issue of abortion. For the first time since Gallup began asking the question in 1995, more respondents described themselves as "pro-life" than "pro-choice" on the issue of abortion. The proportion of Americans describing themselves as "pro-choice" fell from 50% in May of 2008 to 42% in May of 2009 while the proportion describing themselves as "pro-life" increased from 44% to 51%. To back up this conclusion, Gallup cited a recent Pew Poll that showed a decline from 54% to 46% in the proportion of Americans who wanted abortion legal in all or most cases and an increase from 41% to 44% in the proportion who wanted abortion legal in only a few or no cases.
While the results of these two polls appear to show a shift in public opinion on abortion, Gallup neglected to report an important fact about the Pew results that might have undercut this claim. Pew has asked the same question on at least seven occasions since early 2007 with results ranging from a 45-50 split in February/March of 2007 to a 57-37 split in June of 2008. Taken together, these results show no clear trend. The 2009 results could reflect a real change, or they could just be random noise.
Gallup also made no mention of a CNN poll in late April of this year that showed a 49-44 advantage for the "pro-choice" label over the "pro-life" label. CNN has asked the "pro-life" vs. "pro-choice" question three times since 2007 with results ranging from a 45-50 split in June of 2007 to a 53-44 split in August of 2008 to the recent 49-44 split. Again, no clear trend is evident in these results.
And now a new AP poll appears to show continued stability in public attitudes on the issue of abortion. This poll, conducted between May 28 and June 1, found that 51% of Americans want abortion legal in all or most cases vs. 45% who want abortion illegal in all or most cases. These results can be compared with two polls conducted last year. An NBC/Wall Street Journal Poll in early September found 49% of Americans wanted abortion legal always or most of the time while 49% wanted it illegal with no exceptions or only a few exceptions. And a Washington Post/ABC Poll in August found that 54% of Americans wanted abortion legal in all or most cases while 44% wanted it illegal in all or most cases.
The Washington Post/ABC Poll has actually asked this question 23 times between June of 1996 and August of 2008. In these 23 polls, support for keeping abortion legal in all or most cases has ranged from 49% to 59%. Interestingly, the highest and lowest levels of support for legal abortion were found in two polls conducted only a few months apart in 2001.
The safest conclusion one can draw from these results is that at this point the evidence for a significant shift in public attitudes toward abortion is far from conclusive.
By Guest Pollster on June 3, 2009 3:57 PM
| Permalink
| Comments (0)
| TrackBacks (0)
May 6, 2009
By Guest Pollster
[J. Ann Selzer is the president of Selzer & Company and conducts the Des Moines Register's Iowa Poll.]
Can you trust your data when response rates are low? And, in this age of the ubiquitous internet, do we make too much out of its inability to employ random sampling? We asked and answered those questions in a study we conducted a few years ago, commissioned by the Newspaper Association of America. Given recent online discussions of data quality, I revisited this study.
In April and May of 2002, five surveys-asking the same questions-were conducted in the same market. The only difference was the data collection method used to contact and gather responses from participants. This rare look at what role data collection methodology plays in the quality of data yields some fascinating results. Our goal for each study was to draw a sample that matched the market, to complete interviews with at least 800 respondents for each separate study, and to gather demographics to gauge against the Census.
Method of contact. Our five methods of contact were:
- Traditional random digit dial (RDD) phone (landline sample);
- Traditional mail;
- Mail panel, contracting with a leading vendor to send questionnaires to a sample of their database of previously screened individuals who agree to participate in regular surveys, with a small incentive;
- Internet panel, contracting with a leading vendor to send an e-mail invitation to a web survey to a sample of online users who agree to participate in regular surveys, with a small incentive; and
- In-paper clip-out survey, with postage paid.
The market. We selected Columbus, Ohio as our market. It was sufficiently large that the panel providers could assure us we would end up with 800 completed surveys, yet it is perceived to be small enough that mid-sized markets would feel the findings would fit their situation.
Analysis. To compare datasets, we devised an intuitive method of analysis. For each of six demographic variables-age, sex, race, children in the household, income, and education-we compared the distribution to the 2000 Census, taking the absolute value of the difference between the data set and the Census. For example, our phone study yielded 39% males and the Census documents 48%, so, the absolute value of the difference is nine points. We calculated this score for each segment within each demographic, added the scores, then divided by the number of segments to control for the fact that some demographics have more segments than others (for example, age has six segments, education has three). We then summed the standardized scores for each method and those raw scores give us a comparison allowing us to judge the rank order of methods according to how well each fits the market. Warren Mitofsky improved our approach for this analysis.
Problem with the internet panel. I'll just note that both panel vendors were told the nature of the project-that we were doing the same study using different data collection methods to assess the quality of the data. I said we wanted a final respondent pool that matched the market. They would send reminders after two days. Participants would get points toward rewards, including a monthly sweepstakes. The internet panel firm e-mailed 7,291 questionnaires; after 850 completed responses were obtained, they made the survey unavailable to others who had been invited. Because the responses to the first 850 completed surveys were so far out of alignment with the Census, we opted to implement age quotas post-hoc, to systematically substitute some in the 45-54 age group (which were too plentiful) with respondents in other age groups (which were underrepresented) with additional invitations to the survey. We reported out both findings-those before and after the adjustment.
Results. Unweighted, the RDD phone contact method was best; the in-paper clip-out survey was worst.
Weighting just for age and sex improved all data collection methods. Most notable is traditional mail, which comes close to competing with traditional phone contact after weighting for age and sex. The in-paper survey showed the greatest improvement because the respondent pool was strongly skewed by older women. One in four respondents to that survey were women age 65 and older (26%). The median age was 61 (meaning, just to be clear, half were older).
Other data. This study was commissioned by the newspaper industry, so it was natural to look at readership data. Scarborough is to newspapers what Nielsen is to television, and we had their data from the market for comparison. Partly because of the skew toward higher income and especially in higher educational attainment in the internet panel, that method produced stronger readership numbers-higher than the Scarborough numbers and higher than any other data collection method. This was one more check on whether a panel can replicate a random sample, and casts suspicion on whether a panel can ever sufficiently control for all relevant factors to deliver a picture of the actual marketplace.
Concluding thoughts. I have to wonder how this study might change if replicated today. The rapid growth in cell-phone only households probably changes the game somewhat. Panel providers probably do more sophisticated sampling and weighting than was done in these studies. Our mail panel vendor indicated they typically balance their sample draw, though their database in Columbus, Ohio, was just on the low end of being viable for this study, so we're confident less rather than more pre-screening was done. We did not talk with the online vendor about how they would draw a sample from their database, though we repeatedly said we wanted the final respondent pool to reflect the market. It is our sense little was done to pre-screen the panel or to send out invitations using replicates to try to keep the sample balanced. Nor did they appear to have judged the dataset against the criteria we requested before forwarding it to us; it did not look like the Columbus market. We specified we did not want weighting on the back end because we were wanted to compare the raw data to the Census. Had they weighted across a number of demographics, they certainly could have better matched the Census. And, maybe that is their routine now. But, I wonder how the readership questions might have turned out, for example. The Census provides excellent benchmarks for some variables, but not all. Without probability sampling, I always wonder if the attitudes gathered in from panels do, in fact, represent the full marketplace.
Epilogue. Of course it would be a good idea to replicate this study given recent changes in cell phone use. The non-profit group that commissioned this study just announced it is laying off half its staff, so they are unlikely to lead this quest.
By Guest Pollster on May 6, 2009 11:40 AM
| Permalink
| Comments (0)
| TrackBacks (0)
March 16, 2009
By Guest Pollster
Sheri Rivlin and Allan Rivlin are the Co-Editors of CenteredPolitics.com. Allan Rivlin is a Partner at Hart Research Associates. In 1993 Allan Rivlin was a Special Assistant in the U.S. Department of Health and Human Services.
Remember 1993? Snoop Dogg was on the radio. Grunge ruled the world of fashion, and one of the top movies was "Groundhog Day" where Bill Murray had to relive the same day over again until he figured out just what he had to offer the world and finally got it right.
A charismatic young Democrat had just been elected President promising, among other things, to reform a broken health care system. Public opinion seemed to be behind him but the effort ultimately failed and a more careful reading of public opinion in those early months of the Clinton Administration reveal some of the fault lines that eventually sank the effort. Not only did reform fail to make it out of either house of Congress, but in the 1994 election voters ratified the decision and punished Democrats who supported reform rather than the Republicans who had defeated the plan.
Now a new Democrat has taken office promising healthcare reform. The question becomes; has enough changed in public opinion to offer hope that the outcome will be different this time around? A thorough review of the available polling then and now is less than encouraging for supporters of comprehensive health care reform (a category that includes the authors who should be understood to be supporters of comprehensive reform albeit sobering ones.)
Where common questions can be found in polls leading up to health reform 1993 and 2009, the public is currently less attuned to the issue, expresses less dissatisfaction with the status quo, and offers lower levels of support for the general prospect of reform. But an even greater challenge for reformers is the fact that the basic contours of public opinion that undercut the previous effort continue to be true today - perhaps even more so.
Just as in 1993, it would be easy to read current polls as highly encouraging. Many of these measures appear quite strong, it is just that they are not as strong as comparable numbers in surveys taken before the start of the 1993 effort when many pollsters, including those advising the White House were fooled into believing they had a clear mandate for major change.
Now: A 2008 Harris Interactive survey finds 29% saying so much is wrong with the current health care system that it needs to be completely rebuilt, and an additional 53% says that while there are some good aspects the system needs fundamental changes. That adds up to 82% calling for fundamental change. Just 13% say the system works pretty well and only needs minor changes.
Then: The problem is, these results were typical, though a little stronger in the period before the failed effort. As early as 1991, the same pollsters (then Lou Harris and Associates, the word "Interactive" as we know it today had not yet been coined) using the same question recorded 42% saying so much is wrong with the current health care system that it needs to be completely rebuilt, and an additional 50% said that while there are some good aspects the system needs fundamental changes - for a total of 92% calling for fundamental change and just 6% said the system worked well and only needed minor changes.
Now: A 2008 Harvard School of Public Health survey found a 55% majority in support of "national health insurance" with 35% opposed. While this is unlikely to be a phrase that this round of reformers will find useful or descriptive of their proposals, the term that was in common use in 1993 does allow for an apples to apples comparison.
Then: The same researchers using the same phrase in 1993 found 63% supporting "national health insurance" and just 26% were opposed.
Then as now the real problems facing health care reformers were structural and clearly visible in the polls. As the nation reached near consensus that there was a problem, there was never any such agreement on the specific solution. While many people agreed then as they do now that it is wrong that so many Americans are either uninsured or underinsured, the priority then, as now, for most people was on finding ways to lower their own health insurance cost. Then as now most people had health insurance that they judged to be pretty good.
Then: In 1993 a 77% majority told Martilla and Kiley that they were at least somewhat satisfied with their own health care coverage.
Now: For comparison, 82% expressed a similar level of satisfaction with their own insurance in a 2007 Greenberg Quinlan Rosner Poll.
Then: A 1993 Gallup Poll asked people about their priorities for reform and 38% said they wanted health insurance that included all Americans. The bare majority, 51% wanted to control costs, and 10% volunteered that they want reform that did both.
Now: The comparison here is a little less direct, but in 2008 the Harvard School of Public and the Kaiser Family Foundation found similar results with 45% saying they want to make health care insurance more affordable and 22% saying their goal for reform would be to expand insurance to the uninsured.
Then: An NBC News Wall Street Journal Poll in March 1993 found 66% agreeing with the statement "I would be willing to pay higher taxes so that everyone can have health insurance." Just 30% were opposed. A Martilla and Kiley poll found a similar result but in a clear sign of the problems that would emerge, among their 65% willing to pay higher taxes, just 25% said in a follow up question that they would be willing to pay as much as $50 more a month, 40% said they would pay $30, and the majority 62% were only willing got go as high as $10 per month more in order to give coverage to everyone.
Now: In the most recent NBC News Wall Street Journal Poll conducted February 26 to March 1, 2009 the public is now split with just 49% agreeing with the statement "I would be willing to pay higher taxes so that everyone can have health insurance" and nearly as many 45% do not agree.
Does all of this mean that the Obama plan is doomed before it has even begun? Of course not, but putting the apparently positive number from many of today's poll questions in the context of even more positive numbers from polls taken before the previous failed effort should serve to underscore the difficulty of the challenge ahead.
It is clear that the new team will benefit from lessons learned in the earlier health care reform effort. Reflecting a hard won understanding that most Americans are fairly satisfied with their current coverage, the first words out of any Administration spokesperson, including President Obama, on the subject of health care reform is that if you like what you have now you will be able to keep it. Also reflecting the priorities expressed in public opinion polls today (and back then), far greater emphasis is now being placed on cost containment than on extending coverage.
The real question will of course come in the details of the proposal. If Obama can come up with a plan that extends coverage to more Americans without a major increase in the burdens it places on the individuals and businesses who pay for it, then it will be difficult for those who want to see this effort fail to generate much public opposition. Naturally this is a tall order, but we would not want to be among the legions of commentators who have had to swallow their doubts that Barack Obama can achieve the difficult.
The only thing we will predict is that there will be a lot of articles written looking at statistics like some of the ones mentioned here (in fact they are likely to grow stronger as the heat is turned up on the issue) to make the case that this time around the public strongly supports reform. We hope this little bit of context will help keep these articles in perspective.
The authors wish to thank Julia Kurnik for Research Assistance and Robert Blendon of the Harvard School of Public Health for invaluable assistance. Would anyone try to write this article without first calling Bob Blendon?
By Guest Pollster on March 16, 2009 12:39 PM
| Permalink
| Comments (0)
| TrackBacks (0)
February 25, 2009
By Guest Pollster
[This Guest Pollster contribution comes from Philip Gould, who served as a polling and strategy adviser to the British Labour Party for general elections held from 1987 until 2005.
Editor's note: Gould was a central figure in the dispute between pollsters Stan Greenberg and Mark Penn that we have covered this week, as he was responsible for managing the services that each provided to the Labour Party. He submitted his comments to Pollster.com in an effort to help clarify and resolve some of the issues raised here this week.
Since I emphasized the question of whether Penn delivered complete marginals and cross-tabulations, I want to promote the following paragraphs that come toward then end of Gould's memo:
After a poll Stan normally presented a filled in questionnaire, a full banner book containing complete cross tabs.
Mark had a different approach. Following a poll he quickly made available a full and extensive polling report. This went immediate to the whole campaign. This was not an inconsiderable document. I have one in front of me now: it is 18 pages long; it contains historic voting and favourability data; it closely examines 12 targeting groups ranging from rural lower class Conservatives to union households; it uses seven different batteries to examine campaign issues. It analyses responses to the news and key policy areas. And of course it contains numerous message batteries: in all well over 100 questions were asked and recorded. All of these were analysed by voting preferences, and sometimes by demographic categories.
These reports were extensive and useful documents, far in excess of a normal filled in campaign questionnaire. They did not constitute a full banner book and did not contain 'full marginal's' in the manner favoured by Stan Greenberg, but what Penn did supply was both exhaustive and useful, and certainly met the regular needs of the campaign. As one senior campaign official with responsibility for polling in 2005 has said: 'Mark Penn 'could quite fairly argue that the memos were intended for an audience that had no time or interest in delving into every corner of the data. I don't think that in any way illegitimises the findings or his advice'. On a personal note Mark Penn invariably supplied any additional cross tab or targeting data that I required, and I presume the same is true of others. Two pollsters, two approaches.
Gould's piece covers far more ground than this narrow excerpt. It is well worth reading in full.
-- Mark Blumenthal]
I am aware that intercession in the Greenberg/Penn polling war can precipitate what has probably never happened before: uniting Stan and Mark in the face of a common enemy (i.e. me). But with all the risks it entails I will press on. From the start I must declare an interest- I suspect I am one of the very few people around who can claim that they like and respect both Greenberg and Penn (I can already feel them starting to unite against me!). I worked with Stan for well over ten years and believe him to be an outstanding pollster and strategist. I worked with Mark for a much shorter time, and came to greatly appreciate his skills too, different from Stan's certainly, but considerable for all that. It is in that spirit that I write this piece.
There are so many issues here, of methodology, strategy, personality and of course memory that getting to the truth of what actually happened in the UK election campaign of 2005 is probably impossible, but I will try at least to clear away some of the fog. Not by focusing on the smaller, although I accept crucial disagreements between the two pollsters, but by trying to paint a bigger picture, and using where possible contemporary sources, notes written at the time, my rather sketchy diary, and in particular a lecture I made to the LSE on the campaign in 2006 which pretty accurately sums up what I believe about the campaign.
[Continue reading after the jump]
Continue reading "Gould: Greenberg versus Penn, Continued"
By Guest Pollster on February 25, 2009 10:51 AM
| Permalink
| Comments (0)
| TrackBacks (0)
February 23, 2009
By Guest Pollster
This guest pollster contribution from Stan Greenberg is part of Pollster.com's week-long series on his new book, Dispatches from the War Room and responds to comments from Mark Penn in Mark Blumenthal's post earlier today. Greenberg is chairman and CEO of Greenberg Quinlan Rosner.
To avoid this discussion descending into an ugly mud-wrestling match between two squabbling pollsters, I will only take up issues where the "facts" are indisputable and where we learn something about Tony Blair and political leadership and about differing approaches to polling and strategy.
What this exchange reveals even more clearly than the book itself are the limits of building a strategy from a coterie of target groups, rather than from the leader's vision or party's mission for the times. It underscores the need for frankness about what is holding voters back and the need to challenge leaders with blunt truths. It underscores the need for transparency and methodological rigor.
Penn's basic argument is straightforward. He took over the campaign's polling in July 2004 about nine months before the election when Blair was at a low point, working under Philip Gould, Blair's long time advisor for research and media. Greenberg was pushed out and was in no position to judge the character of Penn's work, as he was "not in the loop." Seems straightforward enough.
When I first learned in December of Penn's involvement and in January of our dividing the polling, I was convinced that Gould had played just such a role and I wrote about it. I was wrong. Philip was hurt by the accusation that he had concealed Penn's involvement and wrote me with detailed diary entrees that show he only learned of it in September and resisted Penn's involvement until the end of the year, when he decided to "make the best of it."
Penn's premature rush to anoint himself as Blair's pollster obscures Blair's effort to examine competing solutions to the problems he faced. In May, Blair had reached a low point in the polls, dragged down by Iraq, the "hyping" of pre-war intelligence and Abu Ghraib. He was very despondent, seriously considering not running again and consulted widely, including with President Clinton and Senator Clinton who urged him to run and to use Penn.
Penn offered his own path back for Blair, aided by huge surveys and "clustering work" that coughed up "school gate mums" as a key target. Because Labour got its highest marks on the economy, his message started there, but Penn's emphasis was on policies that appeal to the groups that can grow Blair's coalition. Penn's imprint was immediately evident in Blair's September conference speech when he spoke of the stresses of the need for "more choice for mums at home and at work." Blair's policy offer was grounded in this clustering and coalition building.
At the very same time, we were commissioned by Phillip to do a special research project and I reported in July with a very different approach to the problem - centered on New Labour's central mission. For the first time in a long time, respondents shifted to Labour on hearing of Blair's commitment to "a better life for hardworking families," though only when Blair expressed his own frustration with the state of public service reform and offered some learning by showing independence from Bush on climate change. Iraq was the elephant in the room. Finding a way to acknowledge it, even indirectly, allowed people to come back to Blair's project.
In the September party conference speech, Blair was eloquent about "hardworking families," but just could not get himself to be reflective on Iraq - perhaps with Penn's support. That was the learning voters needed if they were to come back.
I respect Blair for rejecting my advice and deciding to go with Penn who did not push him to address the Iraq question and who offered a way to make electoral gains. The mistake was not firing me and leaving both of us in the campaign.
In fact, I have all of Penn's memos - about a two-inch pile on my desk at the moment, available for inspection by Mr. Blumenthal. Philip's note to me confirms he shared all of them during the course of the campaign, as did many of my friends "in the loop."
The whole concept of "in the loop" betrays a lack of transparency and openness in Penn's approach to campaigns - painfully evident in the Blair campaign, perhaps a precursor to Hillary Clinton's presidential run two years later.
Pollsters as a rule share the results for all their questions and hypotheses, even the ones that didn't pan out. In the Blair campaign, Penn provided a memo with large tables including only the questions he wanted to report; he did not provide a standard book of demographic cross-tabulations. Read Penn's words carefully, "The campaign received all of the agendas, marginals, as requested without reservation." In short, he provided breakouts only when asked, in effect keeping his own client and campaign team "out of the loop."
The surveys were methodologically sloppy and included biased tests, though it is important to underscore here that Philip Gould came to value Penn's research and rejects my characterization of it in the book.
Specifically:
1) Penn failed to incorporate professional learning from Britain. Penn national polling - not some errant tracking program - showed Labour with landslide leads of 8 or 9 points for the entire six weeks prior to the election being called. Penn discovered just 27 days before the election what every pollster in Britain has knows: you have to weight to offset the "shy Tories" - Conservatives reluctant to be interviewed. In an instant, the Tories gained 6 points in Penn's polls.
2) Penn's fixed targeting let real targets slip away. With Penn focused on "mums," the campaign regularly rolled out initiatives on breast cancer screening and childhood obesity. But voters in the key marginal seats were older and among those most likely to return to Labour, two-thirds had no children at home and found this campaign irrelevant.
3) Penn exaggerated the reliability of findings. Penn conducted a valuable weekly open-ended Internet panel of undecided voters. When the sample dropped to 100, so did the reporting of sample size that produced a testy email exchange that restored it. Still, Penn reported this as a "Survey of Undecided Swing Voters" and reported the full percentage results over 18 pages, including results for men and women, with about 50 cases each.
4) Penn created biased tests. Two weeks before the election, Penn declared that "our policy approach remains stronger than the Tories," but the Labour statement was more than twice as long, with more rhetorical flourishes and covering a much broader range of policies with greater specificity (which I'm happy to share). Even with this biased test, the Conservative's statement ran 6 points ahead of its vote. An unbiased test might have revealed potential Tory gains.
To inform the decision of whether to close positively or negatively, Penn constructed a sensible experiment where half the respondents were read positive statements about Labour's progress and half read attacks on the Conservatives' record and plans, and then respondents were asked to vote again. But this was not meant to be a fair test. The negative statements were 50 percent longer by word count and helped foreclose an uplifting close.
Penn describes the 2005 third-term as "historic" but in the campaign everyone was disappointed with the result, what the media called Labour's "drastically reduced majority," produced by a disengaged electorate and historically low turnout. Many factors contributed to the result, but among them were Penn's research, not to mention having two polling teams with different theories on how to win.
By Guest Pollster on February 23, 2009 12:33 PM
| Permalink
| Comments (2)
| TrackBacks (0)
February 21, 2009
By Guest Pollster
This guest pollster contribution from Stan Greenberg is part of Pollster.com's week-long series on his new book, Dispatches from the War Room. Greenberg is chairman and CEO of Greenberg Quinlan Rosner.
Brian Schaffner focuses on the role of pollsters in identifying groups and thus empowering them -- making their opinions relevant to political leaders. I am very conscious of the role and as you correctly point out, I put the spotlight on Macomb County's "Reagan Democrats" and after this election, moved the spotlight next door to upscale suburban Oakland County.
What groups "matter" in my work is not some blind search of the data to find interesting and distinctive groups.
In the period when the wall between my academic and political lives was starting to crumble, I was very taken by E. E. Schattshneider's argument that whomever decides what the fights about likely wins. Successful political leaders and campaigns control the subject, define the choice and choose the fight. Drawing that line decides what issues are important and critically, who gets engaged and who loses interest. In 1992 Clinton made the election about change and the economy stupid and President Bush failed to make it about trust and experience. This year, Obama made it about change and Hillary Clinton tried unsuccessfully to make it about experience, but when she shifted to the economy and the middle class, she put the spotlight on white working class voters who rallied to her.
"Reagan Democrats" derived from the political project that tried to put the middle class back at the center of a renewed Democratic Party -- but the groups emerged from the project. In the book, I argue for the strength of these five leaders because they made politics purposeful.
Related to this point is David Moore's important discussion of "intensity" of beliefs and and the ability of leaders to get people to change their views on an issue and follow them. Whether a leader touches people, understands the times and poses a choice that impacts their lives impacts both which issues get highlighted and how intense are reactions.
I fully agree that mapping intensity will give you a much better view of public thinking and how issues are likely to break. But what is interesting about my Jerusalem example is that people held intense views (which I measured and monitored closely) when they rejected the idea of dividing Jerusalem, but shifted their views nonetheless once the public debate forced them to think about all the possibilities. This is a life and death and emotional issue and voters followed it very closely but Ehud Barak, like earlier Israeli leaders, was able to move the deliberation to a longer-term framework for preserving a Jewish state.
Focusing on intensity will help pollsters know which opinions really matter and difficult to move, and I did a lot of simulation in my polls to see how dynamic are opinions. But I'm still in awe of how much opinions shifted on such a central issue in such a short period and still learning from the fact.
By Guest Pollster on February 21, 2009 3:06 PM
| Permalink
| Comments (0)
| TrackBacks (0)
February 20, 2009
By Guest Pollster
This guest pollster contribution from Stan Greenberg is part of Pollster.com's week-long series on his new book, Dispatches from the War Room. Greenberg is chairman and CEO of Greenberg Quinlan Rosner.
Charles Franklin rightly begins his comments by putting up my quote on page 58 that "the endgame in presidential campaigns brings out all sorts of irrationalities, starting with the media polls. Many are criminally bad." One of the problems in writing a book and a memoir is living with your words and thoughts, particularly when as unnuanced as those.
In retrospect, I might have been more nuanced. First, I made the comment in the context of the Clinton presidential campaign when the statement was clearly true, as described in the book. Second, it reflects my experience during the final weeks in campaigns in Britain and Israel and in Latin America, even very recently. But because of sites like Pollster.com, there is more transparency and exposure of shoddy methods, and despite strong budget pressures, the national media organizations in the US produced very credible polling programs in this last election. But as recently as 2004, there were stark examples of volatile polls without political weighting conducted by Gallup and aired on CNN, along with commentary on how fickle were the voters. The challenge will be what happens with media polls, as there is more upheaval in the industry and need for more costly multi-modal methodologies and greater use of IVR.
This is a very different matter when one goes down to the state and congressional level and when you are in lower turnout elections and primaries. The media polls, as well as polls conducted by universities and institutes, are often out of line with the campaign surveys, as they are less likely to screen or filter for likely voters, factor-in historic turnout patterns and consider use of exit polls, as well as CPS. That one in four state polls in 2008 were conducted one day suggests we are dealing with a genuine issue.
I Amen, Franklin's Amen. The biggest problem is the reporting, not the polls themselves. It is the "outlier" poll -- not the boring average that gets headlines. But it is even worse in the war rooms I'm writing about that are poised to explode in the closing week of the campaign. It is the errant poll, not the average, that sets off the sparks in the war room and gets the attention of the candidate.
By Guest Pollster on February 20, 2009 8:40 PM
| Permalink
| Comments (1)
| TrackBacks (0)
By Guest Pollster
This guest pollster contribution from Stan Greenberg is part of Pollster.com's week-long series on his new book, Dispatches from the War Room. Greenberg is chairman and CEO of Greenberg Quinlan Rosner.
I want to come back in my next post to the first half of Charles Franklin's piece where he raises legitimate issues about my characterization of media polls. But before that I want to develop a point he makes -- about order of questions -- because I actually have some new information on the subject and it is hard to find anyone interested in such issues.
When it comes to the vote, I have spent a lot of time assessing how to get people most comfortable with answering and to minimize the number of false undecided. In my experience, the closer the question to the start of the survey, the larger the undecided. There is a price in possible bias in introducing prior questions but if those questions reflect the broad political context in which the vote choice is being made, you can risk that. So, we will usually have a right direction/wrong track question, most important problem (either open-end or closed), a favorability or thermometer battery on political leaders, parties and organizations -- broadly distributed and balanced. Each election is a test of whether that structure produces unbiased estimates.
Our national presidential results for Democracy Corps or by Greenberg Quinlan Rosner closely resemble the national media polls, with a somewhat smaller undecided (though I haven't done the comparison on this issue).
Presidential approval is different. People are comfortable answering the question and at presidential level, not a lot of undecided once in office for a period. When I polled in the White House for President Clinton, we started by asking the job approval after the thermometer/favorability battery and right before the vote. With President Clinton, I discovered that asked at that point, his approval rating was higher than reported by other organizations. So, we conducted an experiment where half the respondents heard the job approval at the front of the survey and half heard it right after the favorability battery about Clinton and other political leaders. The implicit comparison led people to rate Clinton higher. Since job approval was the indicator most important to us, we moved up to the front in the survey.
We continued that practice during Bush's term and Democracy Corps' approval for Bush was higher than the norm -- and often cited by the White House. Bush's job approval, unlike Clinton's, likely fell if considered alongside other leaders or if the survey dealt with Iraq, the economy, health care or any other topic where is approval was probably even lower. I know this is obvious, but order really matters.
I agree that openness is the best route to sorting these issues -- including publishing the full surveys wherever possible.
By Guest Pollster on February 20, 2009 6:35 PM
| Permalink
| Comments (1)
| TrackBacks (0)
By Guest Pollster
This guest pollster contribution from Stan Greenberg is part of Pollster.com's week-long series on his new book, Dispatches from the War Room. Greenberg is chairman and CEO of Greenberg Quinlan Rosner.
Kristen starts with a key piece of the role of pollster -- keeping elites and elected officials "in-touch" with people; a reality check. Lost in all the talk about politicians with their fingers to the wind is how hard it is for the voter to get heard amidst the lobbyists, experts, bureaucrats, donors and more. If there is a problem to solve, that's not a bad one to address.
I really share Kristen's frustration with the search for the "silver bullet" -- particularly a word or phrase -- when the "silver bullet" is really have a theory of the race, knowing why you are running, defining the choice in a way that really impacts people's lives. Look what happened when Obama's "change" encountered Clinton's "experience" and coalition of small groups. Clinton lost ground because Obama had the force of what was happening in society and the economy with him. Understanding your times and having a mission puts you in a more powerful electoral position.
I agree that voters not voters not grounded in the current ideological polarization. That's why it hasn't worked to label Clinton a "liberal" in 1996 or Obama in 2008. But that has led to some to say that voters aren't moved by big ideas and political political forces and alignments and they have rushed to advance a bunch of small policies. I think 2008 showed that America is a country moved by big currents and open to big ideas about how to address our problems. I think Obama and McCain debated big philosophical issues and those mattered in how people became engaged and voted.
By Guest Pollster on February 20, 2009 10:33 AM
| Permalink
| Comments (0)
| TrackBacks (0)
February 19, 2009
By Guest Pollster
This guest pollster contribution from Stan Greenberg is part of Pollster.com's week-long series on his new book, Dispatches from the War Room. Greenberg is chairman and CEO of Greenberg Quinlan Rosner.
I
think Steve Lombardo makes an important point. Some elections and some
periods invite big choices and political projects, while many do not.
When I raised the issue, I was not so much suggesting what is the best
choice in a particular campaign. Obviously, you will do what works and
use the opportunity of the moment to win. But I was making a
self-reflective judgment about my work over a number of decades, and
the parallel emergence of polling that was self-consciously tactical,
some ideologically centered and some post-ideological. There is
evidence cited in the book that more recent generations of political
consultants (p. 423) give greater weight to the thrill of the contest
rather than partisan or ideological goals, compared to earlier
generations. I don't think that was just a consequence of diminishing
issues in the late 1990s -- as many of them advanced this approach
earlier -- e.g., the 1996 Clinton campaign -- and later.
I do think any campaign -- even ones in less tumultuous times and a
lower place in the ticket -- will seek to pose a choice that draws on
the issue and partisan environment, the candidate's goals and project,
and that poses a defining choice with the opponent. That is strategic
and possibly purposeful, regardless of how small bore. But
collectively, pollsters and political consultants elevate the political
discourse in the country as they are part of that process. But where
they are opportunistically jumping on tactics or "swing groups"
detached from that kind of process, they may well be diminishing the
quality of political discourse.
By Guest Pollster on February 19, 2009 9:55 AM
| Permalink
| Comments (0)
| TrackBacks (0)
February 17, 2009
By Guest Pollster
Today's guest pollster contribution comes from Stan Greenberg, chairman and CEO of Greenberg Quinlan Rosner. He will be discussing his new book, Dispatches from the War Room all week on Pollster.com.
I very much appreciate Pollster.com hosting this discussion of the book and Mark's introductory note that raises central issues.
Let me underline one point that is at the heart of the book and then address the two key questions raised by Mark.
I started the book with columnist Joe Klein's assertion that the polling-media industrial complex diminishes politics, leaders; it makes them less bold and more risk averse. I wasn't sure he wasn't right, if you can excuse the double negative.
And as you can see, I'm pretty critical of trends in polling and critical of some of my own choices, which we can discuss.
But, I come out of this believing that strong political leaders build a special bond with people, rather than flying in the face of it. Strong leadership is not defying the public, but engaging with it -- using support to get things done; mobilizing the public, educating the public on challenges and goals and working to shift opinion. I look at the example of Lincoln and Franklin Roosevelt who were both intensely solicitous of public opinion. Engaging with the public was a precondition for boldness. That contrasts with Bush and Cheney who thought they were strong because they pursued bold policies, never guided by polls and focus groups, but I think we can look now at the consequences. President Obama's special bond with people is part of his leadership but he will struggle like these leaders to keep people with him and enhance his chances of success. That makes for stronger and more democratic leadership and produce greater civic engagement. I'm not making a partisan point -- only the case for strong leadership that is solicitous of public opinion.
Mark raised two issues.
First was tactics. Yes, all campaigns are tactical but divorced from a political project, it becomes just a game and our techniques risk diminishing politics, as Klein suggested.
Example. Reassurance. Bill Clinton reassured voters by his commitment to "end welfare as we know it," support for death penalty, and commitment to cut middle class taxes. That is tactical. But with voters more comfortable about Clinton's values and how he would use government, they now were much more supportive of his bigger agenda for investing to create sustainable growth, investing in people and education, and allowing all to have health insurance. The reassurance built support for the main project, thus a strategy.
Tony Blair reassured by promising not to raise taxes and to not increase the budget over the next two years -- and that allowed voters to support him so that would invest in public services, particularly health care and education.
But when Dick Morris advised President Clinton, he relished "stealing" the Republicans' issues -- like welfare reform -- to make the Republicans irrelevant, not to advance Clinton's larger vision. Here it becomes a game and diminishes politics.
And then there is Jerusalem. My conclusion from that is that pollsters need to be unbelievably careful about assuming current attitudes are static or can't be changed. Political pollsters should not be focused on depicting current thinking, but instead, on searching for the underlying dynamism. Even strongly, deeply emotional positions can give way -- if leaders with authority in certain areas are committed their educative roles.
Remember, we also said in Bolivia that proceeding with the export of natural gas would lead to violent opposition and that voter opinions could not be moved on this deeply emotional issue. The president there was determined to make this bold move and ultimately was forced from office, as violence grew in the country.
So, Jerusalem is a lesson but so is La Paz.
[Follow the complete series here].
By Guest Pollster on February 17, 2009 10:42 PM
| Permalink
| Comments (0)
| TrackBacks (0)
January 28, 2009
By Guest Pollster
Today's guest pollster contribution comes from Humphrey Taylor, who has served as chairman of The Harris Poll, a service of Harris Interactive, since 1994.
In the late 1970s, Louis Harris & Associates (one of the two firms that merged to form Harris Interactive in 1996) was commissioned to conduct a survey of American attitudes to allowing oil companies to drill for oil in wilderness areas that were off-limits. Unfortunately neither Lou Harris nor any of his senior colleagues knew anything about this survey until we read the results in the media -which showed strong public support for opening up wilderness areas for drilling, quite contrary to the findings of other polls by Harris and other firms.
When we looked at the questionnaire it was immediately obvious that this was a particularly egregious example of a "hired gun poll" designed to get the answers the client wanted, a survey designed to mislead rather than to inform policy makers about public opinion.
Lou Harris, to his great credit, publicly disowned the survey and said that the findings did not reflect public opinion. Soon afterwards, he received a phone call from an irate Senator Stevens of Alaska, who was apparently close to our clients, trying unsuccessfully to persuade Lou to back down. And the clients never paid the bill.
Were I a lobbyist with a strong point of view, I (or any competent pollster) could without much difficulty write a series of questions which might be technically acceptable, but which - taken together or separately - would mislead the media and many legislators into believing that public support for my client's position was much stronger than it really is. And here I'm not talking about bad samples or manipulating data to get the answers I want, only about the design of the questionnaire itself.
Over the years I have read many published surveys that fitted the description "hired gun polls." They are often easy for pollsters to spot because the results are surprising and are strikingly different from the results from other polls. Furthermore, they provide powerful arguments that can be used by those who paid for the survey to lobby government, to influence elected officials and to generate favorable publicity. However, it is much harder for most people, including policy makers and influentials, who are not themselves pollsters to recognize, and discount, those "hired gun polls." And this can be made more difficult when the funding source - which may be a company, a trade association, a public relations firm, an NGO or an advocacy group - is hidden behind a supposedly independent third party.
This would not matter if polls had no influence, but I strongly believe that polls sometimes influence the political agenda, policy makers, regulators and the way the media cover issues. At the risk of sounding pompous, I believe all pollsters whose polls are released to the media and the public have a moral obligation that they inform and do not mislead. Unfortunately, hired gun polls are designed to mislead.
In the polling community, some polling firms are seen as particularly bad practitioners of hired gun polling. Some will defend their right to ask whatever questions they and their clients want. Most major polling firms probably try to avoid hired gun work, but in an imperfect world it is not always easy to spot them early enough to prevent them.
Those who are not pollsters may be puzzled by this whole issue. They may believe that the public opinion is what it is and that as long as a representative sample is surveyed the replies will reflect public opinion. Pollsters know better. Different questions on the same issue can produce very different and apparently contradictory results, and questions asked earlier in a survey can have a big impact on questions that are asked later. It is not difficult to write questionnaires that greatly increase the number of people who give a particular response. Fortunately the reputation of our organizations can be hurt if we are seen as to be doing this, and our business may suffer. But, sadly, some firms that have a track record of doing hired gun polls are still in business.
Some pollsters' clients may be puzzled. Once in Latin America, I asked a presidential candidate why he had published polls showing him in the lead, when no other polls did so (he was soundly defeated). His immediate and refreshingly honest reply, "If I pay for a poll, I should be able to get the answers I want and am paying for." He is not unique.
To address this problem Harris Interactive has a set of guidelines and procedures that I summarize below. I would be delighted to hear that other survey firms have similar rules. And I would be thrilled if the leading polling firms could agree on a code of ethics based on similar principles.
We pollsters need to put our house in order so that we do not have to defend polling that is morally and ethically indefensible.
Hired gun polls damage the credibility and reputation of polling and pollsters and, I am glad to say, can - when spotted and criticized - be damaging to the firms that do them and to their clients. A much more serious effect is that they can mislead and misinform policy makers, opinion leaders and the media.
The leading polling organizations have a code but it is a code of disclosure, not a code of ethics. It is necessary but not sufficient. Many years ago they came together to found the National Council on Public Polls (NCPP) and agreed to a code of disclosure that describes what they must publish whenever they release a new poll (they must describe the universe, the methodology, the fieldwork dates and the relevant questions, and who commissioned the research). But complying with the NCPP code of disclosure does not do much to help the readers to recognize a hired gun poll, that is intended to mislead rather than to inform.
Maybe the time has come for pollsters to agree on a code of ethics that would inhibit them from conducting surveys that they know are designed to get the answers the clients want. This is not an argument against conducting and publishing any polls for interest groups and advocacy organizations. They fund many valuable and useful surveys. Rather it is a plea that pollsters do everything they can to ensure that such polls are sufficiently comprehensive and are fair and balanced (although those words seem to have taken on a new meaning) that they do not mislead.
Note: This document provides a summary if the ground rules that bind us and our clients who commission surveys for public release.
By Guest Pollster on January 28, 2009 1:24 PM
| Permalink
| Comments (2)
| TrackBacks (0)
November 13, 2008
By Guest Pollster
Today's Guest Pollster contribution comes from Peter Holm, a Ph.D candidate in Political Science at the
University of Wisconsin Madison. His research focuses on the military
and political attitudes.
Last month, the Military Times newspapers released their quadrennial election survey showing that 68% of currently serving military respondents favored John McCain for president, as compared to only 23% for Barack Obama. Examining the results, Duke professor of political science Peter Feaver noted that "A lot of people thought that eight years of frustration with the Bush administration was going to undermine [the conservative Republican leanings of the military as an institution]. This evidence suggests that it hasn't undermined it as much as they thought, at least not yet."
In one respect, Feaver is clearly right: the military remains significantly more conservative and more Republican than the public generally. But let's take a closer look at the evidence to see whether there has actually been some moderation in military Republicanism over the course of the Bush administration.
Both Times reporter Brendan McGarry, in his piece reporting the poll results, and West Point professor Jason Dempsey, here on Pollster, explained that the Military Times survey cannot be regarded as representative of the military population as a whole. The papers' readers are whiter, older, more likely to be male, more senior in rank, and more highly educated, for example, than are the armed forces at large. Further, the survey (like all the Times' annual surveys) used non-random sampling of this already unrepresentative group, simply allowing any subscriber or former subscriber to respond to an emailed questionnaire. Among active-duty personnel, for example, junior enlisted soldiers (E-1 through E-4) comprise about 22% of the force, but only 6% of the Times 2008 election survey respondents came from the junior enlisted ranks. So how can we use these data to make inferences about political attitudes among the military generally?
One good way to do this is to look at trends over time, as Dempsey demonstrated. A second approach, which I use here, is to weight the military survey data to bring the sample into line with the demographic characteristics of the force as a whole. This is as simple as the weighting we see in most national opinion surveys every day. When the sample doesn't conform to the expected distribution of sexes, races, and ages among registered (or likely) voters, for example, most organizations use post-stratification or raking procedures to "weight up" the responses from underrepresented groups.
I constructed weights for the Times annual surveys of active-duty servicemembers going back to 2003 using race, sex, rank, age, education, and branch as raking variables. I used Department of Defense personnel data from 2005 to construct these weights, as full data on all the demographic variables I included are not available for each year individually. In any case, the demographic profile of the armed forces changes quite slowly; the biggest change over the past eight years has probably been the aging of the active force as recruitment and retention pressures have pushed the services to raise the age limits for entering and leaving the force. Unfortunately, the 2008 survey focused almost exclusively on McCain-Obama comparisons and did not ask respondents to identify with a party or place themselves on an ideological spectrum, so the data from that survey cannot be included here. (The 2008 annual survey, distinct from the election survey, will ask these questions, as annual surveys in previous years have.)
Presidential Approval and Party Identification
Let's look first at President Bush's approval ratings among active-duty servicemembers. Figure 1 shows that, in fact, Bush's support inside the military has declined significantly, while his disapproval rate has increased. In 2003, 62% approved of his job performance and 17% disapproved. In 2007, those numbers were 44% and 36%, respectively. The largest shift came during the year 2006, when civil violence in Iraq reached its peak and disapproval shot up from 22% to 39%.
Figure 1. Source: Military Times annual surveys, 2003-2007;
weighting done by the author using Department of Defense personnel data.
Has this dissatisfaction with the president translated into declining Republican identification among military personnel? Yes, but less strongly. As Figure 2 shows, the gap between Republican and Democratic identifiers closed significantly during the first two years of the second term - in fact, the Democratic deficit was more than cut in half from 37 to 16 points between 2004 and 2006. In 2007, though, this trend moderated. Republican identification rebounded and Democratic identification receded as violence in Iraq abated, the president appeared to have developed a more coherent and successful strategy for managing the conflict, and Republicans attacked congressional Democrats for wanting to "pull the rug out" from under the troops in their attempts to force a timetable for withdrawal into war funding bills. At the end of 2007, the weighted results found Republicans making up 44% of the military population, as compared to 17% calling themselves Democrats.
Figure 2. Source: Military Times annual surveys, 2003-2007;
weighting done by the author using Department of Defense personnel data.
The Military Vote in 2008
How did these trends extend to military voting behavior in 2008? Let's consider two potential data sources. First, we can look at the Military Times 2008 election survey, weighted to correct for the demographic distortions in the subscriber base. Among active-duty respondents, the weighted results show a slightly closer contest than was originally reported: 60% favored McCain, 29% favored Obama, and 11% were undecided or favored someone else. Obama still trailed by a 2:1 ratio, but these same respondents reported that in 2004, they voted nearly 4:1 in favor of George Bush over John Kerry. Although the military vote almost certainly still favors Republicans for president, the gap has clearly narrowed.
It is important to note that the demographic weighting done here does not render the Military Times poll data fully representative of the active-duty population as a whole. It does make the sample demographically representative, at least along the dimensions I have included, but there is no way to tell with the data available what other factors may exist that are a) correlated with political attitudes; b) correlated with a person's propensity to subscribe to the Times newspapers or to respond to the survey; and c) that are not accounted for by the demographic characteristics included. It is the case, for example, that military donations reported to the FEC ran just about even between Obama and McCain, a result that suggests these survey findings, even after being weighted, may still underestimate Obama's support within the armed services. Of course, it could also indicate that Obama's supporters are simply more willing to give money but still constitute a distinct minority. We will have to await the results of better-designed surveys of the military population to find out.
In the meantime, one place to gauge how Obama actually did in relation to previous Democrats among military voters is in the election returns from military bastions around the country. I looked at returns in 29 of the 30 counties with the highest proportion of residents serving in the military as measured in the 2000 census. (Alaska does not report its returns by county, so one county is excluded from the analysis.) Figure 3 shows how Obama did in relation to Kerry's 2004 performance in these 29 counties, where the military percentage of the adult population ranges from 9.8% (in Hardin County, Kentucky) to 63.3% (in Chattahoochee County, Georgia).
Figure 3. Source: state election agencies and cnn.com.
The 45-degree line represents the divide between counties where Obama outperformed Kerry (in the upper left) and where he underperformed against Kerry (in the lower right). In all but one of the top 29 military counties, Obama's share of the two-party vote was higher than Kerry's, and by an average of over 6 points. Nationwide, Obama outperformed Kerry by 4.4 percentage points (Kerry received 48.8% of the two-party vote; Obama got 53.2%). This means that in the military bastions of the country, the counties most dominated by military personnel, Obama not only gained over Kerry's 2004 totals, but he improved on them in these areas even more than he did in the country as a whole. Although Obama still fell well shy of a majority in these counties (achieving 42.6% of the two-party vote, on average), the strong performance of a relatively young and inexperienced Democrat running against a decorated Republican war hero is highly suggestive that military allegiance to the Grand Old Party is on the wane.
By Guest Pollster on November 13, 2008 3:02 PM
| Permalink
| Comments (6)
| TrackBacks (0)
November 7, 2008
By Guest Pollster
Today's Guest Pollster contribution comes from Mark DiCamillo, director of The Field Poll in California.
Having put to rest the so-called Bradley effect in this year's presidential election, we are now seeing numerous references to a so-called "Bradley effect" regarding the California vote on Proposition 8, the same sex marriage ban. The Bradley effect in the California gubernatorial election, even back in 1982 was minimal (at most 2 pts out of the 8 point error in the pre-election polls). It was a convenient theory for people to use when describing the fallibility of the pre-election polls conducted in California in that year, but a closer examination would find most of the polling errors were not due to factors relating to racial bias.
While the notion that social desirability effects could have played a role on a controversial social issue like same-sex marriage, tit s theory without any real evidence, whereas an alternate explanation of the deviation between the pre-election polls and the election outcome is far more compelling and is supported by the data.
First, a quick review of the pre-election polling done by the two leading public opinion polls in California, The Field Poll and the Public Policy Institute of California. They show the following trend:
- July Field: No = +9
- August PPIC: No = +14
- Early Sept Field: No = +14 or +17 (depending on wording)
- Mid-Sept PPIC: No = +14
- Mid October PPIC: NO = +8
- Late October Field: No= +5
- Election outcome: Yes = +4
These data show the No side ahead by double- digit margins throughout most of the pre-television campaign stages. However, as the TV advertising hit in mid to late September, the Yes campaign ads proved to be more effective, and the polls showing the No side advantage slipping.
The movement continued into and through the final weekend of the election when the churches and various religious groups made a concerted effort to win over the support of their congregations. The evidence shows that they were successful.
When comparing the findings from The Field Poll's final pre-election survey of likely voters (n-966) to the Edison Media Research exit poll in California, the biggest differences relate to the turnout and preferences of frequent church-goers and Catholics. The Field Poll, completed one week before the election, had Catholics voting at about their registered voter population size (24% of the electorate) with voting preferences similar to those of the overall electorate, with 44% on the Yes side. However the network exit poll shows that they accounted for 30% of the CA electorate and had 64% of them voting Yes. Regular churchgoers showed a similar movement toward the Yes side. The pre-election Field Poll showed 72% of these voters voting Yes, while the exit poll showed that 84% of them voted Yes.
The same kind of phenomenon occurred when the first same-sex marriage ban was voted in California in the March 2000 election (Prop. 22), although because of the size of its victory( 61% Yes vs. 39% No) it didn't matter much back then. In that year The Field Poll's final pre-election poll, also completed about one week prior to the election, had 50% of Catholics on the Yes side, and accounting for 24% of the vote. Yet, the network exit poll conducted that year by Voter News Service showed them to account for 26% of the electorate with 62% voting Yes.
My take is that polling on issues like same-sex marriage that have a direct bearing on religious doctrine can be affected in a big way in the final weekend by last minute appeals by the clergy and religious organizations.
By Guest Pollster on November 7, 2008 11:54 AM
| Permalink
| Comments (10)
| TrackBacks (0)
November 4, 2008
By Guest Pollster
This Guest Pollster contribution comes from Samuel Popkin, Professor of Political Science at the University of California, San Diego, and Visiting Scholar at the Russell Sage Foundation, and Douglas Rivers, Professor of Political Science at Stanford University, a Senior Fellow at the Hoover Institution, and CEO of YouGov/Polimetrix.
John McCain was dealt a bad hand of cards and then played it poorly. His party has been in power eight years, and even in good times, third terms are never easy. On one hand, there is governing fatigue from a combination of satisfied, complacent partisans, and, on the other, sniping from embittered disappointed former supporters. If we add to the typical eighth year blues an unpopular war, soaring gas prices and the worst financial crisis in a generation, the odds would be against any candidate from the incumbent party.
It is nonetheless surprising just how Senator McCain managed to lose this election. Running against an unknown, inexperienced candidate with a foreign name and dark skin, the McCain strategy was obvious from day one: contrast the security and confidence Americans could have in a known, heroic and inspirational leader with the gamble of an inexperienced, risky choice. With a nation at war, as FDR said, you shouldn't change horses in the middle of the stream. Or as Gerald Ford said every day when making up nearly all of a thirty point deficit against Jimmy Carter in 1976, "Trust means saying what you mean and meaning what you say."
The odds were against the McCain campaign , but this is not how it should have played out. The campaign's primary strategy was to overcome the Bush-Cheney stigma by emphasizing McCain's leadership, heroic sacrifice for his country and willingness to take on both right and left to do what is right for America.
The big surprise of 2008 is that John McCain made voters nervous about himself and not about Barack Obama. Reporters and scholars will be studying the brilliance of Barack Obama and the genius of his campaign staff for many years. We focus on the declines in public regard for Senator McCain as an essential part of the Obama victory, not to denigrate Obama or his campaign. A better Republican campaign would not - in all probability - have succeeded in winning this year. A better campaign with a credible economic plan might well have saved many Republican Senators and Congressmen from defeat.
In the end, we show that voters became more comfortable with Obama and less comfortable with McCain. Suburban, middle class voters and older voters who should have belonged to McCain are voting in substantial numbers for Obama. Obama persuaded voters that his policies would be favorable to the middle class, and that he understands them. Senator McCain did far worse at selling middle class policies or generating strong excitement about his leadership. Maverick did not become synonymous with bold or strong. By the end, more people considered Obama inspiring, and bold, as well as intelligent.
This election was a rejection of McCain as well as his party. When he could talk about security and danger he held his own despite sagging support for his party and no support for the president. When he could argue that offshore drilling would hold down gas prices, he also stayed close. Once he chose Governor Palin and started to talk about the economy his ratings on traits related both to empathy and competence declined. He failed to demonstrate that he would be different from President Bush or that he understood ordinary people or that his policies would be good for the middle class.
For the past year, we have been conducting weekly surveys for The Economist using the YouGov/Polimetrix Internet panel. For this article, we have collected a time series of items that show how and where the McCain campaign failed to achieve its strategic objectives. The analyses presented here are preliminary, but, from the perspective of election day, we can start to understand how the campaign unfolded.
Download the complete report, with detailed charts and a description of the methodology.
By Guest Pollster on November 4, 2008 11:15 PM
| Permalink
| Comments (3)
| TrackBacks (0)
October 29, 2008
By Guest Pollster
Nick Panagakis is president of Market Shares Corporation, a marketing and public opinion research firm headquartered in Arlington Heights, Ill.
A few weeks ago you may have seen a debate regarding voter indecision on these pages. David Moore argued that in a poll, decided voters who said there was a chance they could still change their minds before election day should be counted as undecided voters
This post is an update based on new poll data. I argued then that this wasn't indecision. I said response to the hypothetical, point-in-time "if the election were held today" question will yield some voters willing to decide on a candidate who won't rule out the possibility that some incident or disclosure, however remote, could lead them to vote otherwise which is not indecision. In other words, this is not candidate induced indecision but calendar induced because the election is still weeks away.
ABC is the only poll that follows up its ""could change mind" question with another that asks chances of doing so. See current ABC Poll numbers here. Click PDF report:
The columns show: 1. likely or registered voters would definitely vote for a candidate, 2. any chance "could change mind", which breaks down to, 3. "good chance" could change, and 4. chances "pretty unlikely" which is thisclose to no chance in h*ll.
The table shows that over seven weeks, chance of mind-changing drops from 20% to 9%. Moreover, good chance of doing so drops even faster, from 8% to only 3% overall (2%-3% of Obama voters, 3%-4% of McCain voters, last three reports.). In 2004, ABC polls showed the same "good chance" trend up to a few weeks before the election.
This effect appears predictable, regardless of election, regardless of candidate or campaigns. Imagine if polls up until last week were showing undecideds 10 to 20 points higher - or still showing 9 points greater this week. Again, this not indecision between the candidates as I understand it but the understandable effect of asking how voters would vote today, the only way to characterize election contests over time. ABC tracking poll results in the days to come will continue to confirm this.
By Guest Pollster on October 29, 2008 4:26 PM
| Permalink
| Comments (1)
| TrackBacks (0)
By Guest Pollster
Nick Panagakis is president of Market Shares Corporation, a marketing and public opinion research firm headquartered in Arlington Heights, Ill.
This week our poll for the Chicago Tribune reported that Illinois Governor Rod Blagojevich had a job approval rating of 13% versus 71% disapproving of his performance - astonishingly low job approval. Some 16% had no opinion, which includes ambivalence. Our poll of 500 likely voters was fielded October 16-18.
But another Illinois poll found even lower numbers - an October 13th Rasmussen poll of 500 voters as reported by the Huffington Post.
Their question used a four-point scale asking respondents to "rate the way that Rod Blagojevich is performing his role as Governor". Results were: Excellent 0%; good 4%; fair 29%; poor 65%. Huffington reported individual ratings. But approval is often reported by combining the top two and bottom two scores. In this case, approval would be 4% and disapproval 94%, quite a difference from our 13%/71%. (Don't know response could be lower because of the 4-option scale or because this was an automated poll.)
Some background. In my earlier years, I conducted phone and exit polls for WBBM-TV, the owned and operated CBS station in Chicago. About a year after Jane Byrne was elected Mayor, we obtained her job approval using the four-point scale, combining the excellent to poor ratings to approve/disapprove. Byrne's late husband Jay McMullen, a former Sun-Times reporter, wrote to the station's general manager objecting to the use of that scale. So we did another poll asking both the excellent to poor rating and approve/disapprove questions.
McMullen was right. Not only was her approval score higher than excellent-good combined, but some of the "fair" raters also said "approve" when asked. Moreover, when asked reasons for rating Byrne the way they did, we got answers like "doing a pretty fair job" from those rating her both "fair" and "approve".
Turning to more current examples, two national pollsters use the four-point scale, Zogby and Harris. Harris asks "only fair" not fair. All others use the dichotomous approve/disapprove. George Bush approval ratings dating back to January 2001 appear on the Pollkatz site.
In the chart, Zogby polls are light gray diamonds and Harris polls dark gray diamonds. Note how often these symbols appear at the bottom of clusters of other scores, Zogby more often than Harris. The chart confirms my findings nearly three decades ago.
The four-point job rating has its supporters in the polling community. But it can be not compared with dichotomous approve/disapprove questions.
By Guest Pollster on October 29, 2008 4:24 PM
| Permalink
| Comments (1)
| TrackBacks (0)
October 26, 2008
By Guest Pollster
Guest Pollster Clark A. Miller is an Associate Professor at Arizona State University. His post expands on a comment left on Pollster.com on Friday.
As Mark Blumenthal and Nate Silver have both noted in detail of late, the design of likely voter models can significantly impact how pollsters interpret and transform the raw data of voter samples into the topline results we see at pollster.com, fivethirtyeight.com, and other sites covering election polling. In turn, Mark and Nate observe, likely voter model design depends significantly on judgments that pollsters make about how to model the likelihood that any voter sampled will actually turn out and vote in the election. As we have all seen in the last few days, differences in how such judgments get made by different pollsters, combined with differences in the samples of voters collected by each poll, can mean the difference between a 1-point and a 14-point spread between the respective candidates for President.
A key challenge for consumers of polls - whether citizens, journalists, or politicians - is sorting out to what extent the likely voter model or the underlying raw data sample is responsible for variations in poll outcome. In fact, this sorting out of how judgments made by modelers impact model design and outputs is a general challenge in the use of science to inform policy choices, which I have studied for much of the past two decades. Judgments like this are inevitable in any scientific work, which is why policy officials turn to experts to make judgments on the basis of the best available knowledge, evidence, and theories.
One case that I have looked at in detail is the use of computer models of the Earth's climate to make predictions about whether the planet is experiencing global warming. As I'm sure most of you know, models of climate change have been viewed skeptically by many people. I believe the trials and tribulations of climate modelers - and also their approaches to addressing skepticism about their judgments - offer three useful insights for pollsters working with likely voter models.
- Transparency - climate models are far more complex than most polls, but climate modelers have made significant efforts to make their models transparent, in a way that many pollsters haven't. (In much the same way, computer scientists have called for the code used in voting machines to be open source.) By making their models transparent, i.e., by telling everyone the judgments they use to design their model, pollsters would enhance the capacity of other pollsters and knowledgeable consumers of polls to analyze how the models used shape the final reported polling outcome. They would also do well to publish the internal cross-tabs for their data.
- Sensitivity - climate modelers have also put a lot of effort into publishing the results of sensitivity analyses that test their models to see how they are impacted by embedded judgments (or assumptions). This is precisely what Gallup has done in the past week or so, in a limited fashion, with its "traditional" and "extended" LV models and its RV reporting. By conducting and publishing sensitivity analyses, Gallup has helped enhance all of our capacity to properly understand how their model responds to different assumptions regarding who can be expected to vote.
- Comparison - climate modelers have also taken a third step of deliberate comparisons of their models using identical input data. The purpose of such comparison is to identify where scientific judgments were responsible for variations among models, and where those variations resulted from divergent input data. Since the purpose of polling is to figure out what the data are saying, it is essential to know how different models are interpreting that data, which can only be done if we know how different models respond to the same raw samples.
The reason climate modelers have carried out this activity is to help make sure that the use of climate model outputs in policy choices was as informed as possible. This can't prevent politicians, the media, or anyone else from inappropriately interpreting the outputs of their models, but it can enable a more informed debate about what models are actually saying and, therefore, how to make sense of the underlying data. As the importance of polling grows, to elections and therefore to how we implement democracy, pollsters should want their polls to be as informative as possible to journalists, politicians, and the public. Adopting model transparency, sensitivity analyses, and systematic model comparisons could go a long way toward creating such informed conversations.
By Guest Pollster on October 26, 2008 12:46 PM
| Permalink
| Comments (9)
| TrackBacks (0)
October 24, 2008
By Guest Pollster
Guest Pollster Nick Panagakis is president of Market Shares Corporation, a marketing and public opinion research firm headquartered in Arlington Heights, Ill.
The National Council on Public Polls analyzed national presidential poll accuracy In 2004. The found that eleven of fifteen pollster margins were off by 0% to only 2% from the winning margin., well within the margin of error.
This year polls are now showing Obama from +5 points to +13 ahead, more variability than four years ago this week. This year a demographic variable appears to be having an acute effect on voting estimates. That variable is race and may explain the difference in poll margins.
I did a "what if" spreadsheet analysis with hypothetically variable percentages of black composition of total voters and corresponding variable percentages of the white/other races of turnout. I set a constant result for the vote for president by African-Americans.
Typically, polls lack sufficiently large samples of black voters to reliably estimate their voting intentions. I derived my estimate from a tracking poll's pooled results: 95% of African-American would vote for Barack Obama, 3% for John McCain and 2% for other candidates. A one or two point disparity from 95% for Obama doesn't make much difference in this analysis. If anyone has better results, please respond to this column.
The key variables are the racial distribution of voters and how white/race voters will vote.
Exit polls in the last few elections have shown African-American varying from 10% to 12% (in 2004) of total voters. My analysis ranged from 10% to a hypothetical 15% to check incremental margin gain.
General voter interest in this election may be so high, black composition could remain at 12%. But if it increases to 13%, that adds a one-point margin gain for Obama. For each one percentage point increase in black composition, the overall margin for Obama increases by one percentage point.
The other variable is how whites and others would vote. The following hypothetical scenarios cover most of the overall margins current polls are showing. They range from +4.8 to +9.4 Obama winning margins.
Assuming McCain is ahead by 7 points among whites/others and with 12% African-Americans of all voters, overall results are a Obama win of +4.8%. At 13% black voter of total turnout this would result in a +6.0 point Obama win.
Assuming McCain ahead by only 5 points among whites/others and with 12% African-American sample composition. This would yield an Obama win of +6.6 points. At 13% black voter composition results are a +7.6 point Obama win.
Assuming McCain down to +3 points among whites/others and with 12% black voter sample composition. This would yield an Obama win of +8.4 points. At 13% black voter composition results in a +9.4 point win for Obama.
In closing this analysis has nothing to do with the Bradley Effect theory. And nothing to do with reverse-Bradley. (Since when can a theory have it both ways?) Pre-election poll versus exit poll or post election analysis examining such variables could have confirmed or denied the effect.
By Guest Pollster on October 24, 2008 11:18 PM
| Permalink
| Comments (7)
| TrackBacks (0)
October 22, 2008
By Guest Pollster
Alex Lundry is the research director at the Republican polling firm, TargetPoint Consulting.
With only two weeks remaining, pollsters and journalists alike have rightly been reexamining the legitimacy of their polling numbers. In particular, there have been caution flags regarding three phenomena that may be unaccounted for in many of the public polling numbers: 1) the difficult to poll cell-phone only voter, 2) a possible surge in youth and minority turnout missed by likely voter models, and 3) the Bradley-Wilder Effect causing artificially deflated numbers for McCain.
Unfortunately, poll-watchers today have another reason to be wary: a likely over-report of voter registration, especially among African American voters, possibly causing surveys of both registered voters and likely voters to overstate support for Barack Obama.
A 2007 article in Public Opinion Quarterly (link, gated) by Andrew Fullerton, Jeffrey Dixon, and Casey Borch, looked specifically at the problem of registration over-reporting - in which unregistered respondents inaccurately state they are registered voters. Their analysis relied upon National Election Study (NES) validation studies between 1976 and 1980 (the most recent year for which both registration and voter validation data are available - an analytical shortcoming the authors freely admit to). Seeking the drivers of this behavior, they found that blacks are more likely to overreport their registration (along with those that are better educated, live in the "Deep South", and have strong partisan beliefs). The implications of this are particularly relevant to this year's election polls, as the authors detail in this critical point:
If the level of registration overreporting is comparable today, as we believe it is, this subpopulation inflates the number of potential voters in pre-election surveys because they are typically based on samples of self-reported registrants. More importantly, if our finding that blacks are more likely to overreport registration than voting holds true today, as we think it may, this could skew the results of pre-election surveys, likely in favor of a Democratic candidate given blacks' historical affiliation with this party.
If ever there was an election in which black respondents felt a social desirability bias to over-report their registration, this would be it. Support for Barack Obama among African Americans is nearly monolithic, and we are treated to frequent numeric and anecdotal accounts of increased enthusiasm and engagement among the black community. A reasonable person would conclude that an unregistered African American, called to participate in a survey, would feel some sort of pressure (either known or unknown) to say that he or she is indeed registered, and continue with the survey.
How significant could the bias be? Word of increased registration and enthusiasm among African Americans makes it difficult to assign a precise number, but the data itself can at least provide us with a guidepost: between 1976 and 1980, 11% of NES respondents overreported their registration. Among this group, two-thirds (7% of all respondents) later claimed in a post-election study to have actually voted.
These findings should lead us to be especially wary of recent polling in Georgia and North Carolina showing Barack Obama within striking distance of John McCain, as Fullerton et al.'s analysis indicates that residence in the "Deep South" - states with the heaviest concentration of blacks - also makes a meaningful difference in registration over-reporting (though, to be fair, North Carolina is considered a "Peripheral South" state in their treatment). Still, one way to mitigate this problem - voter registration based sampling - is used by Insider Advantage, a frequent pollster in Georgia, as well as PPP, which has been active in North Carolina.
Still, as much as these findings intuitively "make sense" there are a number of reasons to be skeptical: first, the authors themselves point to a number of issues with their analysis (old data, problems with the validation of African Americans' registration records, etc.), and second, the very reasonable assumption that even if this effect did exist, it could be cancelled out if African Americans turnout at higher rates than pollsters predict they will.
Despite these limitations, Fullerton et al.'s analysis should give pollsters and poll consumers sufficient pause as they read the inevitable flood of horserace results over these remaining weeks.
By Guest Pollster on October 22, 2008 11:25 AM
| Permalink
| Comments (20)
| TrackBacks (0)
By Guest Pollster
Alan I. Abramowitz is the Alben W. Barkley Professor of Political Science at Emory University in Atlanta, Georgia. He is also a frequent contributer to Larry Sabato's Crystal Ball.
It may take some time for historians to decide whether the contest between Barack Obama and John McCain should be considered one of the most important presidential elections in American history. But we already know that it will be the most thoroughly polled presidential election in American history. Lately it seems that not a day goes by without dozens of new national and state polls being released. There are now no fewer than eight national tracking polls underway. These are polls that interview a random sample of voters every day and then combine the results over three or more days, adding one new day and dropping one earlier day, in order to measure changes in candidate preferences within the electorate.
In the past few days there has been a lot of speculation in the media about whether the presidential race has been tightening. Some pundits have argued that such tightening is inevitable in the final days of a presidential race. So what do the national tracking polls tell us about the state of the race between Obama and McCain with less than two weeks left until Election Day?
An analysis of the seven national tracking polls that have been up and running since at least October 12th (Rasmussen, Gallup, DailyKos/R2K, Diageo/Hotline, Battleground, IBD/TIPP, and Zogby) leads to several conclusions. Perhaps the most important one is that despite differences in sampling, interviewing, and weighting procedures, Barack Obama led John McCain on every day in every poll. Beyond that basic finding, however, there are some clear differences in the results of these seven tracking polls.
The results in Table 1 show that while all seven tracking polls have had Obama ahead over the past ten days, the size of that lead has varied considerably. During this time period Obama's average lead has ranged from a low of 4.4 points in the IBD/TIPP Poll to a high of 9.8 points in the DailyKos/R2K Poll. In addition, some of the tracking polls have shown much more volatility than others: the standard deviation of Obama's lead has ranged from a low of 0.8 points in the Rasmussen Poll to a high of 3.9 points in the Battleground Poll which has had both the smallest (1 point) and the largest (13 points) Obama lead in the past ten days. In contrast, Obama's lead in the Rasmussen Poll has varied only from a low of 4 points to a high of 6 points. In general, polls like Rasmussen that weight their results by party identification tend to produce more stable results than polls that do not weight by party identification.
There is no evidence in these data of any tightening of the presidential race over this time period.
Figure 1 shows the trend in Obama's average lead in six tracking polls that provided results every day between October 12 and October 20 (the Battleground Poll does not report results on the weekend). While Obama's lead increased in some polls and decreased in others during this period, the results in Figure 1 show that the overall average changed very little. Obama led by an average of 6.5 points on October 12thand he led by an average of 7.0 points on October 20st, the final date included in this analysis.
If there is no overall trend, then what explains the day to day movement in the tracking polls? One possibility is that most if not all of the day to day movement was due to sampling variation-that it was nothing more than random noise. In order to test this hypothesis, I calculated the correlations among the day-to-day results of the seven tracking polls over these ten days. The correlations between individual pairs of polls varied considerably. Some were strongly positive, some were very weak, and some were strongly negative. Nothing much should be made of this, however, because of the very limited number of days on which these correlations were based. What is significant, however, is that the average correlation among the seven tracking polls over this ten day period was -.06. This means that there was basically no relationship in the day-to-day movement of these polls during this time period. Whether Obama's support was going up or down in one poll was unrelated to whether his support was going up or down in the other six polls.
The lesson that should be drawn from these findings is not that there is any fundamental flaw in the tracking polls. Random variation is unavoidable in public opinion polling. What these findings do indicate, however, is that poll-watchers should not pay too much attention to the day to day movements in these polls unless they see all or most of them moving consistently in one direction over a period of time. Similarly, polling organizations should avoid overemphasizing the significance of the day to day movements in their own polls and pay more attention to whether their results are consistent with those of other polls.
By Guest Pollster on October 22, 2008 11:19 AM
| Permalink
| Comments (9)
| TrackBacks (0)
October 20, 2008
By Guest Pollster
Today's Guest Pollster contribution comes from Jason Dempsey, who is an infantry officer assigned to the Army's 10th Mountain Division and the author of the forthcoming book, Our Army: Soldiers, Politics, and American Civil-Military Relations. He also has an article on the political attitudes of military personnel in the most recent issue of the The New Republic. The views presented here are his own and do not reflect the views of the U.S. Army or the Department of Defense.
While the veteran vote is not attracting as much attention as it did in 2004 it is still a salient election issue, and we could use more discussion of available data. For the most part, the most current available data is provided by the Military Times family of newspapers. [Note--does anyone know if NAES is attempting to do a focused military survey again this year?] However, the Military Times surveys have to be used for what they are: Surveys of subscribers to the Military Times papers (Army Times, Air Force Times, etc.) As such they are not representative of the entire military population. And we should note that they don't claim to be, although that is often lost in interpretations and use of the data.
Briefly, I'd like to address the methodology of the surveys and the ways in which these surveys can be useful, some trends revealed in these surveys since 2004, and some thoughts on the results of their 2008 election survey.
Methodology
First off, I think the Military Times do a good job of explaining that these surveys are not representative (see here), even if the headlines and commentary resulting from these surveys often imply otherwise. As the crowd at Pollster.com is the type that likes the fine print I think this is a great venue for discussing the potential as well as the limitations of these surveys.
The survey in the news last week was this year's election-specific survey. In September the Military Times sent e-mails to about 69,000 subscribers. (They sent original messages to about 80,000, but many came back as undeliverable. Some of this should be expected, and appropriately discounted, given the high mobility of the active-duty military community, but it is not clear how many with invalid addresses were active as opposed to being retirees or in the Guard/Reserve). From this they collected responses from 4,515 retirees, 1,515 members of the National Guard and Reserves, and 2,982 active-duty members of the military (although of these 316 were left out of analysis because they were not registered or did not intend to vote). Of the active-duty respondents, 1,543 were in the Army. I limit analysis in my research, and here, to the active Army population within the surveys as I can appropriately compare this subgroup with the overall Army population. However, I think the discussion of the representativeness of the Army subsample probably applies equally to the other active-duty groups.
As with previous Military Times surveys the respondents in 2008 were disproportionately white, male and officers. The actual Army population is about 85% male, 14% regular commissioned officers (not including Warrant Officers), and 60% white. The active-duty members of the Army who responded to the Military Times poll were 90% male, 45% regular commissioned officers, and 71% white. Furthermore, the Army's junior enlisted ranks are dramatically underrepresented in the Military Times surveys. About 47% of the Army serves in the ranks of E-1 through E-4. These ranks comprise only 6% of the active Army population included in the 2008 Military Times survey. (The samples of each of the previous Military Times surveys are nearly identical in the degree to which they represent the active military population). Bottom Line: these surveys should in no way be used to assess aggregate attitudes across the force.
However, this does not mean that the Military Times surveys aren't valuable (that is far from the case). Rather, it highlights that interpretations of the Military Times survey results have often been inappropriately extrapolated to the entire military population.
These surveys can be useful in two ways. First, they can be useful as a gauge of opinion trends. While the results of these surveys might not present an accurate estimate of overall military attitudes in a given year, over time they reflect how the opinions of a portion of the military are shifting. By extension we might assume that the rest of the military is shifting to a similar degree, even if the starting point is not the same. (See the discussion of Robert Shapiro and Ben Page on 'parallel publics' in The Rational Public).
Secondly, if we limit analysis of the survey data to senior officers then the 'subscriber bias' is likely to be minimal, in that the attitudes of senior officers in the Military Times subscriber database are likely to be similar to the attitudes of senior officers generally. Whereas a junior soldier or officer who subscribes to the Army Times is likely to have a more careerist outlook than his or her peers, in that subscribing can be interpreted as an act of dedicated interest in the profession as a whole, the difference between subscribers and non-subscribers is likely to be more muted in the senior officer ranks.
Trends: 2003 to 2007
If one assumes these surveys can be useful as a gauge of opinion trends, if not a comprehensive view of aggregate attitudes, then the Military Times surveys do tell us something about military attitudes over time. I believe we can also view them as fairly accurate portrayals of the opinions of the subset of senior Army officers. Below are the results of what I found when parsing out the opinions of active-duty Army officers in the ranks of major and above (typically 10 to 20+ years of service) from the 2003 through 2007 Military Times annual surveys.
The single datapoints reflect the results from the Citizenship & Service: 2004 Survey of Military Personnel (Completed with Bob Shapiro and support from the Institute for Social and Economic Research and Policy at Columbia University. See here). The C&S Survey included a higher number of women and minorities and the resulting data was weighted to reflect the general Army population on the dimensions of rank, race/ethnicity, and gender. Notably, the results from comparing senior officer attitudes from the C&S Survey with the attitudes of senior officers in the Military Times survey show a pretty close match.
These Military Times survey results show that support for the Republican Party among senior members of the Army, the group most likely to identify as Republican, declined significantly between 2004 and 2006 before leveling off at about 49% in 2007. Also interesting is that the data show no corresponding change in support for the Democratic Party.
Because the Military Times did not conduct these surveys before 2003 we can't assess what this means historically, but we do have data from Ole Holsti's and James Rosenau's Foreign Policy Leadership Project surveys that were conducted every four years between 1976 and 1996. Looking at this data, the military is experiencing a shift comparable to what occurred between 1976 and 1980. During that period military leaders shifted decidedly toward the Republican Party. By the end of Carter's presidency the proportion of senior military leaders who identified with the Republican Party had increased by 13%. This data show a shift of comparable magnitude--only during this administration the military has begun to shift away from the Republican Party. Over the last three years the Military Times surveys have shown a decline in Republican Party identification of 14% among active-duty Army respondents and an overall decline of 13% among senior Army officers.
Notes on 2008
Unfortunately the 2008 Military Times Election survey did not ask party affiliation. They did, however, ask respondents both who they planned to vote for during this election and who they voted for in 2004. Not exactly panel data, but this again offers an opportunity to assess shifts in attitude among survey respondents.
The primary headline to come out of the Military Times surveys was that 68% of respondents backed the Republican presidential candidate, John McCain. However, lost in the analysis was a significant shift in support for the Democratic nominee. Looking at just the subset of active-duty members of the Army in the Military Times poll, 64% of these respondents reported voting for George Bush in 2004 and 15% reported voting for John Kerry. As for the 2008 election, 66% planned to vote for McCain while 25% reported planning to vote for Barack Obama.
There are two significant points to draw from these results. The first is the 10-point uptick in support for the Democratic candidate. While not indicative of a reversal of military preferences among officers, this increase in support for the Democratic candidate signals a significant shift in military opinion and indicates that military aversion to the Democratic Party may be on the wane.
A key discussion point from initial reports on the survey was that black members of the military overwhelmingly indicated support for Barack Obama, but looking at the demographics of those who reported shifting their preference to Barack Obama in 2008 reveals that this dynamic was not driven solely by minority respondents. Of those who shifted from Bush to Obama, 95% were male and 55% were white (n=79, again, I am only looking at active-duty Army respondents). Among those who voted for neither the Republican nor Democratic candidate in 2004 but were planning to vote for Obama in 2008, 77% were male and 39% were white (n=121). This indicates that the increased support for the Democratic presidential candidate among members of the Army is due to both a shift of the Army's traditional voting block away from the Republican Party as well as an infusion of new, predominantly minority voters into the Democratic column.
The second significant point to draw from these results is that McCain has been able to hold onto military votes at a time when the Republican brand is hurting nationally. He is holding a slightly higher portion of the senior officer vote than Bush did in 2004. This is probably not indicative of a shift back to 60% Republican Party identification among senior officers, but is probably due more to his own veteran status. Among active-duty Army respondents, 73% felt that the veteran status of the candidates was important in making the decision about who to vote for (32% felt it was 'very important'. Another 41% felt it was somewhat important.) I suspect that this explains a good portion of McCain's military support and that post-election assessments of the party identification of senior officers will be closer to the 2007 figures.
Finally, however, it must also be noted that comparisons with 2004 are problematic. As outlined by Jeremy Tiegen in his analysis of the veteran vote in 2004, the Swiftboat attack ads were successful in getting otherwise Democratic voters to vote for George Bush, so 2008 may be the 'norm' although, again, McCain's strong identification as a veteran further muddles analysis. Hopefully someone will attack the data and give us an answer after the election.
For those interested enough in military attitudes and polling to have read this far there will be more analysis of the attitudes of the active-duty Army (to include the enlisted ranks) in my forthcoming book Our Army: Soldiers, Politics, and American Civil-Military Relations due from Princeton University Press in 2009.
By Guest Pollster on October 20, 2008 10:32 PM
| Permalink
| Comments (4)
| TrackBacks (0)
September 22, 2008
By Guest Pollster
Nick Panagakis is president of Market Shares Corporation, a marketing and public opinion research firm headquartered in Arlington Heights, Ill.
Some polls are reporting the cross-over Clinton voters; i.e., the percentage of Clinton primary/caucus voters who tell pollsters they would vote for McCain in November. In the first half of September this was reported by CBS, ABC, Pew, and Quinnipiac polls. Results ranged from 12% to 25%with an average of 19%.
While the numbers are interesting, how will they affect the November election? The findings are in need of some context if you are willing to accept some assumptions
First consider how many votes Clinton won in state primaries and caucuses. According to realclearpolitics.com she won 18 million votes after including Michigan and state caucus estimates. Assuming they all vote in the general election, multiply 18 million by the 19% average above and you get 3.4 million November voters who say they will vote for McCain.
What percentage of the general election vote will they represent? Assuming turnout will be near the record 122 million votes cast in 2004 (source: FEC), those 3.4 million Clinton cross-over voters represent 2.8% of all voters. If similar magnitudes of Clinton voters make good on their intentions in states that prove to be battlegrounds, this could make the difference in a close election.
This year the Democratic primary remained a tight contest much longer keeping many Clinton voter hopes alive so similar cross-over data from past elections may not be useful.
Another benchmark would be cross-over Democratic voters voting for any Republican candidate in past exit polls. The New York Times "Super Table" of past exit poll results provides answers. 1992 to 2004 exit polls show 10% or 11% of Democrats voting for the Republican candidate. (Earlier years are much higher due to Reagan Democrats, not likely based on pre-election polls this year.)
In three of those last four elections Democrats won the popular vote overcoming the 10%-11% Democratic cross-overs who voted for the Republican. So the question becomes: will that 2.8% of Clinton cross-over voters add to those numbers or be mostly included in those numbers?
By Guest Pollster on September 22, 2008 2:18 PM
| Permalink
| Comments (12)
| TrackBacks (0)
September 3, 2008
By Guest Pollster
John
Coleman is a the Chair of the Department Political Science at the University of Wisconsin-Madison
Joe Biden has supported President Bush 70% of the time. You
may not have heard this mentioned at the Democratic National Convention or in
Barack Obama's acceptance speech.
The Obama team--and Obama himself--has been working hard to
link John McCain to George W. Bush by noting that he "votes with Bush 90% of
the time." And if 90% isn't enough Bush for you, Democrats note, McCain
supported the president 95% of the time in 2007. One Obama ad
even lists this voting record as the first plank in McCain's economic program.
The figures being used by Democrats are presidential support
scores computed by CQ Weekly, a leading weekly magazine
monitoring events in Washington.
The score is based entirely on recorded roll-call votes in Congress. CQ identifies those votes where
the president has taken a clear stand and then records whether a senator or
representative voted in the president's preferred direction. The votes need not
be key on the president's agenda or be anything the president encouraged
Congress to do--they are simply cases where CQ has determined a clear
presidential position. In the Senate, the president's nominations, which are
usually noncontroversial, are a sizable portion of the votes used by CQ to
compile its support score. In 2007, nominations were 30% of the votes used by
CQ to calculate presidential support in the Senate.
As the chart below shows, John McCain has indeed voted
consistent with the preferences of President Bush about 90% of the time on
these presidential support roll-calls. This has been roughly the same level of
support as the average Republican senator.
McCain's presidential support level was 95% in 2007, but
this is somewhat misleading. Because he was running for president, McCain was
present for only 38 of the 97 roll calls CQ used to calculate the presidential
support score. There were 442 roll-call votes in total in the Senate in 2007. Looking
at only those votes for which both McCain and Obama were present that year--33
votes--McCain's support score was 94% while Obama's was 48%. CQ also noted
in a recent post that McCain, Obama, and Biden voted on less than half the
presidential support votes from January through August 2008.
Using the same figures the Obama campaign has used to tie
John McCain to President Bush, Biden was a 77% supporter of President Bush's
positions in 2002, 70% in 2004, and over a 50% supporter of Bush in 4 of the
president's 7 full years in office. Up through the August 2008 congressional
recess, Biden
had supported Bush's positions 52% of the time since January 2001. Obama
himself supported the president's positions just under 50% in 2006 and 40%
since he joined the Senate in January 2005.
It is doubtful that many Americans hearing the Obama team's
90% charge against McCain realize that Obama and Biden themselves have
supported the president anywhere from 33 to 77% of the time during his term.
In addition to linking McCain to Bush, another goal of the
Obama campaign in using the 90% support figure is to blunt McCain's claim to be
a maverick who shows independence from his party. Establishing McCain's
independent credentials was a major theme at the Republican National Convention
on Tuesday night.
Given that even Obama and Biden sometimes had relatively
high levels of support for Bush, a better measure of independence than the
presidential support score would be to look at the party support score, also
calculated by CQ Weekly. Looking at
"party votes"--those roll-call votes on which a majority of Republicans oppose a
majority of Democrats--CQ calculates whether a senator voted with his party's
majority or against it. The party support score is the percentage of times a
senator voted with his party majority on party votes. There were 266 party
votes in the Senate in 2007, or 60% of all Senate roll-call votes.
Looking at his party support scores during the Bush
presidency, the chart below shows that McCain regularly was less supportive of
his party than the average Republican senator. His voting in 2007, when McCain
was frequently out of Washington
and missing more roll-call votes than usual (he voted on 48% of the 266 party
votes), is an exception.
McCain's professed independent streak is supported by these
data. About 75 to 85% of the time, McCain voted with his party's majority. More
frequently than the average Republican, however, McCain voted with the
Democratic majority rather than the Republican majority on votes that put the
two parties on opposite sides.
Obama and Biden, on the other hand, have both been more
likely than the typical Democratic senator to vote with the Democratic party
position. In each of his three full years, Obama voted over 95% of the time
with the Democratic majority on party votes. McCain reached 90% only once, in
2007.Biden's party support level has hovered between about 90 to 95%. From
these data, McCain can more credibly make the claim that he is willing to buck
his party. He has voted against his party majority about 15 to 25% of the time
across the Bush years, compared to about 3% for Obama and 5 to 10% for Biden.
I've plotted these data in a different format in the chart
below. Here, zero on the left axis indicates the baseline party support level
of the average senator for each party. I then plot the difference between the
average Republican senator's party support and McCain's, and the average Democratic
senator's support and Obama's and Biden's. During the Bush years, McCain was
usually about 5 to 10 percentage points less likely to vote with his party than
the average Republican senator. Obama's party support level was about 10 points
higher than the average Democratic senator, while Biden was usually between
about 5 to 12 points more likely to vote with the party majority than the
average Democrat.
These numbers burnish McCain's independent credentials, at
least compared to his two senatorial rivals. But they also point to one of the
key dilemmas of the McCain candidacy. To weaken McCain's maverick image,
Democrats can tie McCain to Bush by emphasizing McCain's presidential support
percentage, while not mentioning the sometimes high Bush support level of his Democratic
opponents themselves. McCain can respond by noting that, compared to his
rivals, his party support percentage shows he is less likely to vote along
party lines and has more of an independent streak. Emphasizing that streak may
endear him to independents and some Democrats, but it is of course one of the
chief aspects of McCain's legislative life that has historically created
problems for him within his own party and among party activists. It is one of
the tasks of the Republican convention to convince Republicans of the virtue of
that independent streak as a matter of character, even if they disagree with
McCain on policy particulars.
By Guest Pollster on September 3, 2008 12:46 PM
| Permalink
| Comments (14)
| TrackBacks (0)
August 27, 2008
By Guest Pollster
Today's guest pollster contribution comes from Michael P. McDonald, an Associate Professor of Government and Politics in the Department of Public and International Affairs at George Mason University and a Non-Resident Senior Fellow at the Brookings Institution.
A media storyline surrounding the Democratic convention is how a sizable number of Hillary Clinton supporters are backing John McCain over Barack Obama. A recent CNN/ORC poll provides grist for the mill. Twenty-seven percent of self-identified Clinton supporters are reported backing John McCain, an increase from 16% in a similar June survey.
Yet, there are indications that something is amiss in this survey. CNN reports they interviewed 1,023 adults. The organization does not report the sub-sample size of Democrats who support Clinton, but they do provide a margin of error of this sub-sample from which we can infer the number of Clinton supporters. The reported margin of error for Democrats who support Clinton is 7.5 percentage points, which is equivalent to 171 persons assuming a simple random sample. That is 16.7% of all adults in the survey, which when applied to my 2006 voting-age population estimate of 227 million persons means that there are 38 million self-identified Clinton supporters among Democrats in the CNN/ORC poll (with a 95% confidence interval between 20.9 and 54.9 million persons).
As one might recall, Clinton received 18 million votes in the primaries. If she had received 38 million votes, she would be accepting the Democratic Party's nomination on Thursday.
The question arises, who are these 20 million or so self-identified Democrats who support Clinton who did not participate in the primaries? It is difficult to tell without analyzing the survey in depth. While there are many reasonable explanations for the discrepancy between the election and survey results, a plausible explanation consistent with the large percentage of self-identified Clinton supporters who report supporting McCain in a two-way contest against Obama is that the CNN/ORC questionnaire is worded in such a manner that elicits persons who self-report supporting McCain to report that they are a Democrat who supports Clinton for the party's nomination.
The implication is obvious: if these surveys that purport to measure Clinton supporters who will vote for McCain actually measure McCain supporters who would like to see Clinton as the Democratic nominee, the media storyline of Democratic dissention quickly unravels.
By Guest Pollster on August 27, 2008 5:09 PM
| Permalink
| Comments (30)
| TrackBacks (0)
August 13, 2008
By Guest Pollster
Today's guest pollster contribution comes from Humphrey Taylor, who has served as chairman of The Harris Poll, a service of Harris Interactive, since 1994.
When I started work in market research, I spent the first month learning to interview people face to face, in central Scotland. It was a great experience. At the first house, a woman answered the door and, as I nervously explained that I wanted to interview her, she shut the door in my face. My supervisor, a cheerful, middle-aged woman, rang the bell again and, all smiles and self-confidence, easily completed the interview. The same thing happened at the next house, and my self-confidence hit rock bottom.
At the third house, to my enormous relief, I actually managed to complete the interview with an elderly Scottish lady. But as I thanked her, she said, with a twinkle in her eye, "Och, surely you don't believe all the things the folks tell you, do you?" This may have been one of the most valuable comments anyone has ever made to me about survey research, and I have remembered it many times over my working life.
The sad truth is that all too often we researchers naively accept what respondents tell us without questioning if it is really "true." The following exercise explains part of the problem. Ask someone which candidate or party they prefer and they will usually give you an answer (and, by the way, it will probably be true). But then ask "Why?" and the conversation will probably go something like this:
"Because of his/her/their policies."
"Which policies specifically?"
"His. . . um.. economic (or Iraq, health care, etc.) policies."
"What are his economic policies?"
Push harder and you will probably find that this voter really doesn't know what the candidate's economic (or most other) policy proposals really are. But this does not stop people from having a strong preference for one candidate over another, or believing that they would handle the issue mentioned better than their opponents.
One model of how voters choose candidates is that they are like juries. They listen to the candidates and carefully consider their policy proposals before deciding which way to vote. Unfortunately this theory is almost never true.
There are many reasons why it is so difficult to understand people's motives. One is that most people don't understand themselves and often rationalize their attitudes and behavior. Sometimes they surely deceive themselves and sometimes they knowingly bend the truth or tell outright lies. There is a growing body of literature that documents the unreliability of replies given to interviewers where there is a "socially desirable" answer. Large numbers of people lie in telephone and in-person surveys about whether they believe in God, go to church, give money to charity, clean their teeth regularly, drive over the speed limit or drink alcohol. Many people who do not vote claim that they do. And the number of people who say they voted for a sitting president tends to go up when he is very popular and down when his ratings fall.
Another problem is that people give inaccurate answers not because they are lying but because their memory is imperfect. And many people's honest predictions of their own future behavior are notoriously inaccurate.
When it comes to voting, there are many factors which influence voters' preferences, most of which they are often unaware of. While the candidates' positions on the issues (or voters' perceptions, which may be inaccurate, of their positions) are an important factor, there are several more powerful ones.
Voters often explain their votes based on the candidates' track record. Obviously these are important but voters' perceptions of politicians' track records vary greatly depending on their political and ideological views. Some voters think President Bush should have been impeached. Others think history will show him to have been a great president. While perceptions of politicians' record often influence voter preferences, the reverse is also true -- that voter preferences have a huge impact on perceptions of their track records.
So what other factors have a big impact on voting behavior? One is family, friends and people they work with. Most people vote the same way as most of the people they like and socialize with.
A candidate's voice, looks, style and rhetoric are all enormously important. Franklin Roosevelt, the only president to be elected four times, was the perfect candidate -- good looking, with a beautiful voice, a commanding presence and a wonderful way with words. But if you had asked people why they voted for him they would probably have referred to his policies or his track record and what they thought he would do.
One of the reasons Ronald Reagan was so successful as a politician was that he, and his pollster Dick Wirthlin, understood that "values" were more important than "issues." Reagan mastered the art of persuading people he shared their values, so that many people who did not support his positions on some key issues voted for him anyway. In addition he, and his aide Michael Deaver were masters of the photo-op, casting Reagan in great settings that made him look very presidential. But voters would not tell you this influenced their votes
There are many other favors that influence candidate preferences and voting behavior that are rarely mentioned by voters. Political advertising has a huge impact but few people believe, or tell surveyors, that they are influenced by advertising. The media voters are exposed to matter a lot. Those who read the editorials in the Wall Street Journal or The New York Times have their opinions shaped, or reinforced, by what they read. And those who watch Fox News get a very different world view than those who watch other television stations. As dictators and media moguls know well, the media's ability to shape public attitudes is very powerful.
So next time a voter tells you his vote is determined by the candidates' positions on the issues, treat this with a large dose of salt.
By Guest Pollster on August 13, 2008 4:13 PM
| Permalink
| Comments (0)
| TrackBacks (0)
August 6, 2008
By Guest Pollster
Nick Panagakis is president of Market Shares Corporation, a marketing and public opinion research firm headquartered in Arlington Heights, Ill.
This post is the forth installment of a dialogue between pollsters David Moore and Nick Panagakis about the best way to measure and report how many voters are "undecided." See their earlier installments here, here and here.
I agree with much of what David Moore says in his response, including percentage undecided that seems too low as is currently being reported. Where we differ is on terminology. The potential for mind-changing is a lot less than you think.
Yes we are "interested in portraying what the electorate is thinking today". Now that general election national polling is underway, we will be interested in finding (needless to say) whether voters did change their minds about the candidate(s) since the last poll asking how they would vote "if the election were held today".
My issue is about reporting results with low conventional undecideds followed by a large number in the 20%+ range who could still change their minds. It's enough to give readers and viewers whiplash.
In my last post on this subject I hypothesized that such high numbers are not "indecision" as implied by "could change their minds". I said some voters willing to decide on a candidate in a poll won't rule out the possibility that some incident or candidate disclosure, however remote, could lead them to vote otherwise.
The ABC Polling Unit provides some validation of this. Their polls have been asking this question of decided voters since 2004: "Would you definitely vote for ___ or is there a chance you could change your mind and vote for someone else?" This has been asked three times since May this year and eight times in 2004, from June 20 to September 26. This year, "could change your mind" has ranged from 25% to 29%, similar to response levels seen in current polls, dissimilar wording not withstanding. In 2004, "could change your mind" was 28% in the June 20 poll then steadily declined to 16% in late September.
But unlike other polls, ABC then probed potential mind-changers by asking "Is there a good chance you'll change your mind, or would you say it's pretty unlikely?" So far this year, about half say "pretty unlikely" as did respondents in June, 2004 polls. July to September 2004 showed another pattern. 'Pretty unlikely" voters began to consistently outnumber "good chance" of mind-changing voters by a ratio of 2 to 1. This could mean that two-thirds of possible mind-changing voters in current polls, if asked their chances of doing so, would rate their chances as pretty remote. Should mind-changing as currently being presented be part of any story when the chances of doing so are so slim? I don't think so. I prefer the ABC qualifier.
Another thought. Shouldn't there be some analysis to validate such high could change their mind numbers? The analysis could compare poll stated undecideds with "could change their mind" levels with actual candidate vote preference changes from poll to poll and to election outcomes.
Another subject. David mentioned the recent CBS poll. According to their release they had 12% undecided which seems reasonable to me. If you go to pollster.com's national summary you will find many polls with much lower undecideds. However, half of Gallup's higher undecideds shown there are actually vote for "neither" which should not be combined with undecideds in that table. Click the Gallup links. Moreover, "neither" response is not very meaningful. It would be more precise to replace it with "vote for other" and "won't vote" with non-voters excluded from the base for calculation of voter percentages. All for now.
By Guest Pollster on August 6, 2008 4:55 PM
| Permalink
| Comments (0)
| TrackBacks (0)
July 30, 2008
By Guest Pollster
Nick Panagakis is president of Market Shares Corporation, a marketing and public opinion research firm headquartered in Mt. Prospect, Ill.
This is in response to David Moore's July 25th column about use of a broader measure of voter indecision. For the first time, I also asked a similar question before the February 5th Illinois primary but am now having doubts about it's usefulness.
In the final CNN/University of New Hampshire primary poll, over 90% of voters stated their preference for a candidate in the commonly used "if the election were held today" forced choice question. That poll had Obama up by 9%. But Clinton won by 2.6 points. The candidate estimate error was 5.8 points, that means 5.8% high on Obama and 5.8% low on Clinton, near the average of all NH polls. When voters in that poll were asked if they were definite, leaning, or "still trying to decide", some 21% said still trying to decide which was the subject of Moore's blog.
Among the 21% who were "still trying to decide", that could mean 6% of all voters switched from Obama to Clinton or, a net 6% more voters switched from Obama to Clinton than from Clinton to Obama. The 21% more than covers such movement.
Other New Hampshire polls showed comparable numbers: Gallup's "could change mind" and in late December the LA Times' "might end up voting for someone else" both yielded 27%. I checked polls in other states that asked similar questions of decided voters and show comparable high percentages with no evidence that such mind-changing ever took palace.
My first issue is that the forced choice "if the election were held today" question historically comes close to the actual outcome, even though some voters may not have reached final closure when asked. I wouldn't call this "indecision" after so many could decide in response to the standard question. I believe it means some voters who are wiling to decide on a candidate in a poll won't rule out the possibility that some incident or disclosure, between now and election day, could lead them to vote otherwise. Isn't that what campaigns including negative elements are all about? This response is more conditional, perhaps remote, depending on unknown future events, not indecision. If it were indecision, a lot more polls than New Hampshire would have been be off the mark this Spring. In the post New Hampshire period, I cringed when I saw such numbers being reported. I think they de-values polls. There must be some better way of reporting these findings rather than "candidate A is up by 9 points - but 30% could change their minds".
During the week preceding the February 5th Illinois primary, our Chicago Tribune poll showed Obama ahead by 31 points in that primary, very close to the actual outcome. Our poll also got a similar number just days before election day - 24% of decided and leaners said they could "still change their minds". Could it be that a few days before any election, somewhere around 20%-25% of voters in all polls always say they could still change but most never do? Based on the Illinois outcome, not many minds were changed as is the case in most polls. To me, it seems that how voters would decide today has served us pretty well with some exceptions such as New Hampshire. (The question read: "Between now and next Tuesday, is there some chance that you could still change your mind about voting for this candidate...or have you definitely made up your mind?")
Re-calculating our Illinois Democratic poll numbers to combine possible mind-changers with undecideds as Moore did with the New Hampshire poll resulted in: Obama 44%, Clinton 16%, Others 1%, and 39% undecided. (The apparent reason for 39% here was an increase in conventional undecideds due to Edwards dropping out the day before interviewing began. Edwards did have 15% support in Illinois in a poll conducted a few days earlier by St. Louis Post-Dispatch,/KMOV-TV poll.) According to MSNBC, the NEP Illinois exit poll found 19% of voters who said they decided in the last 3 days, the period after we completed interviewing, close to our conventional undecideds. But the recalculated 39% undecided above that included voters who could "still change their mind" is twice as high as the 19% of voters NEP found deciding on a candidate during the 3 days before that election.
In the Illinois Republican primary, 36% of voters and leaners said they could change their minds. McCain was ahead in the poll by 23 points and went on to win by 19 points, a 2-point error on candidate estimates. Moore did not include a comparable number for the New Hampshire Republican primary but all polls matched the outcome.
In conclusion, perhaps in the New Hampshire Democratic primary this year such mind-changing took place. The state has always been a minefield for pollsters. The challenge for pollsters was mostly situational. This was a fluid situation, akin to trying to catch a falling knife. The campaign period was compressed, shortest-ever in New Hampshire, only 5 days after Obama's Iowa upset. Obama was described as over-confidant. Clinton perceived as a victim by some.
There were methodological challenges. Turnout that this year turned out to be historically high (a forewarning for us pollsters in later states). Only 52% of voters in the New Hampshire Democratic primary were registered Democrats according to the exit poll and 19% were first-time primary voters, a challenge for likely voter screening. According to one pollster, their best estimate of the New Hampshire outcome was based on all registered voters; i.e., no sample reduction at all for likely voters. The final chapter on this election has not yet been written. Neither has the value of routinely reporting that 25% or more of voters are undecided.
By Guest Pollster on July 30, 2008 5:51 PM
| Permalink
| Comments (1)
| TrackBacks (0)
June 27, 2008
By Guest Pollster
Today's Guest Pollster article comes from Thomas Riehle, a Partner of RT Strategies.
If it were a different month on the calendar - say, October - the Obama campaign might be concerned to see that the groups most likely to be truly undecided, not leaning even a little toward Obama or McCain, comprise some voters he must be counting on:
- Women (21% undecided), women ages 40-64 (24%), women ages 65 and older (25%), women with less than a 4-year college degree (24%), and
- Registered voters in the Northeast (20%) and Great Lakes (20%).
Moreover, no one at the Obama campaign can be happy to see that the vote is currently tied among women with a college degree or more (43% Obama - 42% McCain) - highly educated women having become one of the most reliably Democratic groups in the electorate. Obama may start to win back support from among the relatively large group of McCain supporters currently to be found among women who voted for Clinton in Democratic primaries or caucuses (25% now support McCain), college-educated Clinton primary voters (28% McCain), moderate or conservative Clinton primary voters (23%) - but right now, those are some significant defections.
The calendar says June, not October, and undecided voters eventually will make up their minds. All in good time. For now, the McCain camp and Obama's camp are looking for indications of subtle trends moving in the early stages of the general election. For that kind of tracking, many campaigns use a tool called the "Hierarchical Vote." It divides support into 7 categories, from most pro-McCain to most pro-Obama, and tracks movement from one category to another across the 7 categories of support. Tracking changes month-to-month in the Hierarchical Vote overall among all voters (and within 50 or more subgroups) gives a campaign the insight needed to focus resources on the groups who are ready to move right now.
On the Cook Political Report website, you will find posted a Hierarchical Vote analysis from the past four Cook Political Report / RT Strategies polls (March through June). I hope you find it a unique and useful way to delve under the familiar topline vote totals and see what's really going on as we approach Independence Day. We'll keep updating this Hierarchical Vote table as Election Day approaches - just as the campaigns will do. Please let me know what you think and what you learn in reviewing these results.
By Guest Pollster on June 27, 2008 1:31 PM
| Permalink
| Comments (14)
| TrackBacks (0)
May 28, 2008
By Guest Pollster
Today's Guest Pollster's column comes from Robert M. Eisinger, a political science professor at Lewis & Clark College and the author of The Evolution of Presidential Polling (Cambridge U. Press).
The Obama "phenomenon" is a product of many things, most notably a superbly smart candidate and a sharp, disciplined campaign team, both of whom clearly articulated a resonating message that mobilized voters. What we don't know if how many of those supporters galvanized around Senator Obama simply because he is not Senator Clinton. This is not to say that Senator Clinton is without fans. To the contrary - she has many. They are devoted and dedicated, imparting the kind of loyalty that any political candidate would desire.
But as evidenced by the reaction to her recent mentioning of Robert Kennedy's assassination, Senator Clinton appears to be a lightning rod - people are either repelled or attracted to her.
Arguments for an Obama-Clinton dream ticket suggest that Senator Clinton's keen intelligence, legislative acumen and support among Democrats outweigh her negatives. However reasoned this claim is, it is potentially flawed in a critical way currently understudied by the public opinion and political cognoscenti. The problem is that we do not know enough about her positives and negatives, especially among voting Democrats and swing voters.
By itself, the "favorability question" is crude and insufficient indicator of likeability. It does not claim to measure the intensity underlying that favorability or lack of favorability. For example, a May 23, 2008 Newsweek poll asks, "Who would you MOST like to be nominated as the Democratic Party's presidential candidate this year...Hillary Clinton (or) Barack Obama (choices rotated)?" "Do you support (INSERT CHOICE) strongly or only moderately?" The poll then asks, "We'd like your overall opinion of the presidential candidates. As I read each name, please tell me if you have a favorable or unfavorable opinion of this person - or if you have never heard of them before this interview. What about (INSERT - READ AND RANDOMIZE). Do you have a favorable or unfavorable opinion of him (or her)?"
Favorability questions specifically asked about Hillary Clinton in the past have been worded in numerous, thoughtful ways, including the following: 1
CBS News/New York Times: Is your opinion of Hillary Clinton favorable, not favorable, undecided, or haven't you heard enough about Hillary Clinton yet to have an opinion?
Gallup/USA Today/CNN: I'd like some overall opinion of some people in the news. In general, do you have a favorable or unfavorable opinion of Hillary Clinton?
Yankelovich/Time/CNN: Please tell me whether you have generally favorable or generally unfavorable impressions of [Hillary Clinton], or whether or not you are familiar enough with [Hillary Clinton] to say one way or another.
Each of these questions is carefully written, but they do not capture the intensity that may lie beneath the answer. In fact the Yankelovich/Time/CNN question employs the phrases "generally favorable" and "generally unfavorable", allowing the respondent to articulate her overall impression, but in doing so, diffuses the potential passion or force embedded within that answer.
If respondents were asked to place their favorability/un-favorability on a seven point scale, then one might get a better sense of the potential polarizing nature of the answer, or put another way, the strength of that favorability and its opposite. Such an option is costly in that it requires an additional question to be asked, and more nuanced data analysis.
Anecdotal conversations in the blogosphere, in the taxi cab and the around the water cooler - reveal that many citizens - men and women, Democrats, Republicans and Independents - have a palpable and deep disdain for Senator Clinton. Different blog, cab and water cooler discourses tell us that Senator Clinton is revered. Scholars of public opinion and savvy journalists are appropriately suspicious of these unrepresentative remarks. Sure, the plural of anecdote is data, but we wonder if selection bias (i.e., we surround myself with like-minded folk; we listen more carefully only to extreme answers) taints our perspective and analysis.
The Obama campaign should be privately measuring the favorability intensity for all prospective Vice Presidential nominees. If not, then they are avoiding a datum that may be critical to their electoral success. Media polls should explore this question as well; given the dearth of the intensity question, the answers will undoubtedly surprise us. //END
1 From Barry C. Burden and Anthony Mughan, "Public Opinion and Hillary Rodham Clinton," Public Opinion Quarterly, Vol. 63, No. 2 (Summer 1999), 237-250.
By Guest Pollster on May 28, 2008 3:20 PM
| Permalink
| Comments (11)
| TrackBacks (0)
May 23, 2008
By Guest Pollster
Today's Guest Pollster contribution comes from Robert M. Eisinger, a political science professor at Lewis & Clark College and the author of The Evolution of Presidential Polling (Cambridge University Press).
There are few things more dangerous to sensationalized journalism than when anyone over-analyzes poll data. A recent Quinnipiac Poll shows Senator Clinton defeating Senator McCain in Ohio and Florida, but Senator Obama losing such head-to-head match-ups against Senator McCain. A SurveyUSA poll shows similar results in Missouri and North Carolina. Clinton defeats McCain, but Obama does not. These polls, it is argued, are worrisome for the Obama campaign, and serve as evidence among some Clinton supporters that she is a stronger candidate in swing states.
Beware. Poll answers, regardless of the question, must be placed in some context. The absence of at least one follow-up question may have yielded an interesting context from which to interpret the head-to-head answers provided. Imagine, for example, that the pollsters asked the following question:
"Imagine that Senator Obama eventually becomes the Democratic presidential nominee, and Senator Clinton enthusiastically campaigns for him. If the 2008 election for President were being held today, and the candidates were Barack Obama the Democrat and John McCain the Republican, for whom would you vote?"
One could even imagine tweaking the question by revising "Senator Clinton," "the Clinton campaign," "Senator Obama," and "the Obama campaign." Such questions are reasonable one to ask, especially when one is reminded that both Senators Obama and Clinton have stated that they would endorse the Democratic nominee. There is good reason to believe that Senator Clinton would be magnanimous and enthusiastically support the Democratic ticket. Similarly, there is no reason to conclude that the inclusion of this question would necessarily result in poll numbers that would greatly assist Senator Obama; it is quite conceivable that some of Senator Clinton's supporters do not want to a President Obama under any circumstance, and that Independent voters may be turned off by Senator Clinton's endorsement of anyone.
No doubt the Obama campaign is not rejoicing after reading the poll data. They would prefer numbers that show Senator Obama defeating Senator McCain in all states at all times. Senator Obama wants to win the swing states, and right now, Florida, Ohio, North Carolina and Missouri are not securely in the Democratic camp.
But there is something noticeably absent about asking about a viable scenario in which the Democratic presidential candidates, especially Senator Clinton, unite behind the winner, even if the nominee is Senator Obama.
[Typos corrected]
By Guest Pollster on May 23, 2008 3:29 PM
| Permalink
| Comments (3)
| TrackBacks (0)
May 6, 2008
By Guest Pollster
Humphrey Taylor has served as chairman of The Harris Poll, a service of Harris Interactive, since 1994.
Thanks to Doug Usher for his contribution to the debate on the panel-based online methodology for political polling. I am glad to see that he acknowledges the value and viability of this method for national polls. But I am puzzled as to why he thinks this method will not work in Congressional races (at least I think that is what he is saying). He writes of the "sour spot" based on the fact that political polls need to reach "a narrow population for which pollsters do not have well defined web contact information".
I assume he means by this that we cannot sample a geographic area because we do not know where people live. Of course we can, and we do this easily. We and others who have large panels, know the states in which people live, so that takes care of senate races.
What about congressional districts?
Some panels also have zip code information, and those that do not can screen for it. In so far as some zip codes straddle the boundary with another district we can screen for streets or even addresses if necessary. And of course this problem is the same ,or possibly worse, for RDD telephone samples ,as telephone exchanges may also straddle the boundaries between districts. Furthermore many people now take their telephone numbers with them when they move from one district to another.
My comment that was quoted by Doug Usher was taken out of context. I certainly believe that online political polling methods are "the wave of the future". My mention of cell phones was specifically in reference to telephone surveys of people aged 18 to 29. I am sorry if that was not clear.
By Guest Pollster on May 6, 2008 12:09 PM
| Permalink
| Comments (0)
| TrackBacks (0)
May 1, 2008
By Guest Pollster
Berwood Yost is Director of The Floyd Institute's Center for Opinion Research at Franklin & Marshall College. Kirk Miller is B.F. Fackenthal Professor of Biology and Senior Research Fellow at The Floyd Institute's Center for Opinion Research.
The 2008 Democratic presidential primary on April 22 put Pennsylvania in the national spotlight for a long six weeks. Members of the media followed the candidates into the Keystone State intending to learn more about its people and its politics. Not far behind the media came the pollsters--some media even brought their own pollsters. Pennsylvania voters were besieged by pollsters in unprecedented numbers. There were 39 publicly released surveys, which included more than 30,000 interviews with the state's voters, during only the last three weeks of the campaign. This is a tremendous increase in polling activity compared to the 26 polls released in the final three weeks of the 2004 presidential campaign in Pennsylvania or the 15 released during the final three weeks of the 2006 Senate campaign.
Taken together, the pollsters who pestered Pennsylvanians did an adequate job of predicting the final outcome: 36 of the 39 polls in April predicted a Clinton victory and the three outliers were all conducted by the same polling organization. We agree with Charles Franklin's assessment that the aggregate performance of the Pennsylvania pollsters was good. Figure 1 is a frequency distribution of the predictive accuracy of the 39 public polls released in Pennsylvania. It shows that there was a slight bias in the polling estimates toward Barack Obama (meaning the polls in Pennsylvania underestimated Hillary Clinton's margin of victory), but that this bias was small and, according to the exit polls, not surprising because late deciding voters moved in larger proportions toward Clinton.
Figure 1 Frequency Distribution of Predictive Accuracy
Some individual pollsters faired much better than others in the accuracy of their estimates. Figure 2 shows the predictive accuracy and corresponding confidence interval for each of the 39 polls conducted between April 1 and 22 in Pennsylvania, arranged by the number of days prior to the primary the survey was completed. Those pollsters who produced a biased estimate, meaning the confidence interval for their estimate did not overlap zero, are labeled in Figure 2. Three of the four polls conducted by Public Policy Polling (PPP) were biased and all were biased toward Obama. Two of the three polls conducted by American Research Group (ARG) were biased and one of SurveyUSA's three polls showed bias. One ARG poll showed that Clinton and Obama were tied; the other, seven days later, showed Senator Clinton ahead by 20 points. The SurveyUSA poll that missed also showed Senator Clinton ahead by 20 points. The measure of predictive accuracy we used shows that the pollsters' final estimates were mostly in line with the final election results.
Figure 2 Predictive Accuracy of Individual Polls by Date of Poll
The misses identified in Figure 2 are not related to sample size. Four of the surveys that missed had four of the eight largest samples; the other two that missed had sample sizes that were only slightly below the median size. There is a relationship in these analyses, as one would expect, between sample size and the widths of the confidence intervals, but there is no relationship between sample size, width of the confidence interval, and the likelihood that a survey was biased. We don't know what methodological choices matter most in producing unbiased polls without further examination of the methodological choices the pollsters make. Some might conclude that pollsters who use inter-active voice response (IVR) technology to collect data are more prone to bias because two of the three pollsters who produced biased estimates use IVR, but not all IVR pollsters produced biased results.
Another interesting question we tried to answer is whether the polls converged on the end result as election day approached. Depending on the method used, the answer is a qualified, "slightly." Figure 3 shows the predictive accuracy of each poll as a function of days before the Pennsylvania primary. The trend line fitted to the figure is produced by a LOWESS iterative locally weighted least squares regression. The red dots identify the six biased polls noted earlier. The curve indicates that the polls began to converge until about two weeks prior to the election, that they remained relatively constant for about a one-week period, and then began to converge again over the final days of the campaign. If the six biased polls are removed from the analysis, the convergence is not dramatically improved.
Figure 3 Predictive Accuracy of Individual Polls by Date of Poll with Fitted Regression Line
Measuring Predictive Accuracy
We used the measure of predictive accuracy developed by Martin, Traugott and Kennedy (2005) A Review and Proposal for a new Measure of Poll Accuracy. Public Opinion Quarterly, Volume 69 (3): 342 - 369. Their method compares the ratio of the estimated percent of voters voting for each candidate to the ratio of the final vote tally for each. The natural log of this odds ratio (ln odds) is used because of its favorable statistical properties and the ease of calculating confidence intervals for each estimate. The confidence interval for a poll that reasonably predicts the final outcome of the primary election will overlap zero. Senator Clinton's votes or projected votes were the numerators in all the ratios we calculated so negative values for ln odds represent an overestimate in favor of Senator Obama and positive values represent an overestimate in favor of Senator Clinton. According to this measure, a poll is biased if its confidence interval does not overlap zero. The polling results used in this analysis were taken from Pollster.com.
By Guest Pollster on May 1, 2008 3:30 PM
| Permalink
| Comments (1)
| TrackBacks (0)
By Guest Pollster
Douglas Usher is the Senior Vice President of Widmeyer Communications and formerly Vice President at the Democratic polling firm, The Mellman Group.
"The survey research and marketing industries need to recognize that the Internet and cellphones, not landlines, are likely to be the wave of the future." So says Humphrey Taylor, chairman of the Harris Poll.
I met Humphrey Taylor once - in 1999. He pitched Harris Online services to the Democratic polling firm where I worked, and his team said that telephone surveys in politics would likely be replaced by web surveys after that election cycle.
Was he right about political polling? Hardly - in fact, he couldn't have been more wrong. Let me make this as clear as possible: no professional political pollster on either side of the aisle has ever used web-based surveys for quantitative research in their campaign practice.
And as any pollster.com reader understands - and all serious consumers of political polling know - you can count on one hand the number of public pollsters using online methodology for political polls. Even John Zogby, who claims that his firm has "since the mid-90's... utilized the Internet as a means of providing the public with instant access to the day's best public opinion research," has like most pollsters used telephone polling this cycle.
Internet polling is a growing industry. I use it all the time for my clients - indeed, it rules many aspects of consumer research. So, why the disconnect for politics?
Because quantitative political research for nearly all levels of American politics hits the "sour spot" of internet research.
Let me explain.
Internet-based research is perfectly suited for certain types of public opinion research:
- Qualitative research: in-depth, group level research designed to evaluate reactions to specific ideas, issues, and stimuli (like campaign ads) - research which provides rich feedback, but is not projectable on the population at large. The internet provides a (virtually) limitless pool of volunteers that will provide quick feedback about a candidate, product, print or television ad. It provides a reasonable - and often less expensive - alternative to focus groups, without the travel.
- Quantitative research among broad populations: For the broadest audiences - "adults" nationally, "likely voters" nationally, and "likely voters" in some states - internet research can provide a reasonable (and again, less expensive) alternative to telephone polling. The opt-in panels that internet research vendors build - if properly cleansed and refreshed on a regular basis - have been demonstrated to be reliable proxies to telephone research for point-in-time quantitative measurement.
- Quantitative research among narrow populations where e-mail contact is previously established: This type of research includes organization membership research, or a survey of loyal customers, or an internal corporate survey. One of the fallacies spread by those who sell public opinion research services on the internet is that because people are on the web, they are reachable on-line. But unless someone provides you (or an organization) with their e-mail address, it is nearly impossible to find them. However, for internal organization research, internet research conducted of a complete (or near-complete) population by e-mail has become an excellent alternative to phone surveys.
These three types of research describe most of the public opinion research for which clients pay money - hence, the internet has become a valuable research tool.
And qualitative public opinion research is well-suited for the internet (finally, the end of notoriously unreliable mall-intercepts!)
However, quantitative political public opinion research -- polling -- hits the internet's "sour spot" because it requires reaching a narrow population for which pollsters do not have well-defined web contact information.
How well do you think Harris Interactive's national panel maps on to likely voters in New York's 26th Congressional District? If you were polling Indiana's primary, would you feel comfortable that the list of e-mails that you bought from a vendor actually contained properly registered voters in the state with past primary vote history?
Some internet survey vendors claim that they have representative general election statewide panels. This may be true - but how many times can you go back to that panel before you exhaust it? Pollsters in competitive races will track data for 30 days or more - well beyond the capacity of internet vendors in even the largest state.
It's not because political pollsters are "old-fashioned" that they don't conduct web-based quantitative research - it's because there is no reliable way to reach their candidates' electorates online in a way that meets even a modest level of methodological rigor.
None of this is to discount concerns about telephone polling - ever-lower response rates, and caller-ID and cell-phone only households that makes reaching people on the phone more difficult than ever.
But, for political polling, internet-based research has not proven to be the panacea once (and continually) promised.
UPDATE: Humphrey Taylor responds.
By Guest Pollster on May 1, 2008 3:12 PM
| Permalink
| Comments (1)
| TrackBacks (0)
April 1, 2008
By Guest Pollster
Yesterday, in a burst of blogger exuberance, I posted some charts emailed by my long ago employer Harrison Hickman, the Democratic pollster now associated with the firm Global Strategy Group who also conducted surveys earlier this year for John Edwards. I gave Hickman and his associate credit for the charts but then provided my own interpretation, comments I subsequently qualified. This morning, I did what I should have done in the first place, which is offer Hickman the opportunity to describe the charts in his own words. Harrison's summary follows below.
--Mark Blumenthal
The initial (and only) purpose of the line charts Ben Margolis and I sent Mark yesterday seems to have been obscured by our failure to provide explanation with the charts and some of the verbal vines their publication stimulated ("Day-of-Week Effect in Gallup Daily?"). Hopefully I can provide something of a corrective for the former.
1. We submitted the charts without explanation but with obvious doubts about their significance, statistical and otherwise. The subject line of my original e-mail to Mark was "spurious or what?"
2. The point of the exercise was to note that all the hoopla about Obama or Clinton being ahead or behind by more or less at a specific point was ignoring a persistent pattern in the data. (The time period covered was since the departure of the Sainted Senator Edwards.) The point was that the hoopla was misguided, not that the pattern itself is all-telling. We certainly never intended to suggest that particular changes could be associated with specific days or that there was any iron law of anything at work. Our message: if you don't like the results today, wait a couple of days. If you do, it might be wise to exercise some restraint. In that vein, Mark is correct in urging caution about reading too much into day-to-day changes. I would urge similar caution in the interpretation of two techniques under discussion here.
3. The rolling average technique was developed to introduce a cost-effective way to report opinion data more or less continually in critical points of a campaign, and there are a variety of different ways to calculate those averages. But it is important to note that the "smoothing" artifact is the reason the technique is useful, not the reason it is misleading. An on-going series of one-day polls would be more misleading for campaign professionals and poll consumers than rolling averages.
4. Perhaps the most important statistical point to understand about these types of polls is that a sample is not a sample until it is completed. Before its completion, a sample is not "random" even in the colloquial sense of the term. It is for this reason that no one should mistake the partial results of stand-alone samples as precise, no matter how extensively those partial data are weighted. This is particularly important to remember when confronted with early wave results of election day polls (exit polls).
5. One should be mindful of but not obsessed with any particular statistical test. Estimation error is the most reported but hardly the only type of error in opinion research. It is treated as more important than it is and than the other types of errors because (a) it has the veneer of precision because it is a number and (b) it easier to understand and better researched than other categories of errors. Here is a simple measure of the its importance: "sampling error" so-called is taught in the introductory course but other types of errors are saved for later in a student's learning. Here's another: If you read any questionnaire carefully and think seriously about the methods used to gather the data, you almost always will find sources for potentially greater "error" than estimation error in what is reported.
6. In fact, a legitimate argument can be made that estimation errors are not really an applicable statistic for most opinion polls we see. The underlying assumption of sampling error is that the sample in question is random, and random has a very precise statistical definition. For a host of reasons, the samples in most polls do not qualify as random in a strict sense and, in too many cases, even under the loosest standards. Harris or Gallup (forgive me for mot remembering which) used to report a table of mathematical estimation error ranges but also something called ranges generated "from observation." I do not recall that they ever explained the source of the observations but found the presentation refreshing as an implicit statement about the limitations of statistical error calculations.
7. Two final observations from reading comments. As consumers and practitioners, recognize that political arguments are still political arguments even when they are dressed up with statistical language. And, finally, do not assume that any pollster is part of a larger conspiracy against your preferred candidate until you have ruled out (a) incompetence and (b) the possibility that things are not as rosy as you want them to be.
Harrison Hickman
Global Strategy Group, LLC
P.S. Not to suggest that there is a day-(or period-)of-the-week effect in the Gallup data, but as of a few minutes ago, it seems that the stop-the-presses 10-point "lead" Obama enjoyed this weekend is now four points. An up-to-date version of our original charts is below, including a line based on calculation beginning the week after Super Tuesday.
By Guest Pollster on April 1, 2008 2:53 PM
| Permalink
| Comments (1)
| TrackBacks (0)
March 28, 2008
By Guest Pollster
[Today's Guest Pollster contribution comes from Alex Lundry, research director at the Republican polling firm, TargetPoint Consulting.]
TargetPoint Consulting recently partnered with the Cook Political Report and RT Strategies, adding a new political research question called the Net Promoter Score (NPS) to their most recent national omnibus survey (March 6-9, N=802). This measure, adapted from the world of consumer research, attempts to measure voter enthusiasm and passion for a candidate. The results provide some new understanding to how the general election for President could shape up given either an Obama or Clinton candidacy.
First introduced by Frederick Reichheld in the Harvard Business Review and since popularized in his book, "The Ultimate Question," the NPS is used in the business world as a customer satisfaction metric, measuring a customer's likelihood to recommend a product, brand or company to someone else. This captures a number of difficult-to-quantify emotions, attitudes and preferences, by posing it as a recommendation. A recommendation is the ultimate endorsement, showing just how passionately you feel about a particular company or product. A recommendation means putting your own reputation on the line; an indication of loyalty, passion, and even the latent potential for word of mouth buzz.
The question is simple:
On a 0 to 10 scale, with 0 being "not at all likely" and 10 being "extremely likely," how likely is it that you would recommend voting for [INSERT CANDIDATE NAME] in the next election to a friend or colleague?
The NPS is calculated by subtracting the number of detractors (ratings of 0-6) from the number of promoters (ratings of 9 and 10). In the business world, +16 is the median score of more than 400 companies across 28 industries; CostCo has one of the highest known scores at +81. (See the NPS website for more details and similar statistics).
Studies have shown a direct and significant correlation between a business' score and company growth - specifically, a 7 point increase in overall NPS or a 2 point reduction in the percentage of detractors can each account for one percent of positive growth, thus indicating the potential electoral consequences of this measure once adapted to political polling.
To be fair, the NPS is not without it's own set of detractors and it's validity in the political world remains to be seen. For now we can only speculate about any correlation with electoral outcomes. Nonetheless, there is some promising historical data: TargetPoint began tracking the NPS on a generic congressional ballot in September of 2005 through August of 2006 and the results did seem to forebode the Republican fall from favor and the impending Democratic advances of that November. During that time the GOP NPS had a distinctively downward slope, falling from a high of +56 to a low of +11; meanwhile, the generic Democratic NPS trended upwards from a low of +32 to a high of +56.
But what about this year's election? In an Obama/McCain match-up McCain leads 45-43, but the NPS indicate some form of an "enthusiasm advantage" for Obama: among Obama voters, the NPS is +28 (53% promoters minus 25% detractors); while 48% of McCain voters are promoters and 31% detractors for a NPS of +17. Hence an Obama advantage of 11 points.
The Clinton/McCain ballot (McCain leads 47-45) again shows a Democrat enthusiasm advantage, though a slightly smaller one of 8 points (McCain: 44% promoter, 33% detractor, +11 NPS; Clinton: 48% promoter, 29% detractor, +19 NPS).
Though there is little surface difference between the candidates, deeper analysis indicates two critical demographic differences: enthusiasm among youth and Independent voters. The NPS among Independents voting for Obama (+30) is a stunning forty-nine points higher than the score among Independent McCain voters (-19). Interestingly, McCain actually wins Independents against both Clinton and Obama, but his Indy voters are much less enthusiastic than either Obama's or Clinton's. Clinton's NPS among her independent voters is also negative (-5), and a full 45 points short of Obama's. It appears that a Clinton candidacy would remove any passion or enthusiasm among Democrat-voting Independents.
Finally, we see nearly identical performance among 18-40 year olds. McCain's NPS among this age group is +7 and -5 against Obama and Clinton respectively; Clinton actually performs worse than McCain here, with a negative score of -13, while Obama dominates at +32. Again, we are left wondering what would happen to this youth enthusiasm should Clinton become the nominee.
Keep in mind that these are scores among people already voting for that particular candidate. While a vote is still all that matters on Election Day, a recommendation driven campaign can produce new votes faster, cheaper and in a more trustworthy and impactful way than traditional campaign appeals of advertising, direct mail and robo-calls.
By Guest Pollster on March 28, 2008 1:39 PM
| Permalink
| Comments (5)
| TrackBacks (0)
March 14, 2008
By Guest Pollster
(Today's Guest Pollster's contribution comes from Professors Robert S. Erikson and Karl Sigman of Columbia University.)
In late February, SurveyUSA interviewed 600 registered voters in every state for a total of 30,000 interviews, ascertaining preferences in a McCain-Obama and a McCain-Clinton race. The focus was a new set of electoral maps of red and blue states based on who led each state in the survey. Based on who won each state in the SurveyUSA survey, Obama defeats McCain 280 to 258 while Clinton defeats McCain 276 to 262 in the Electoral College.
Of course SurveyUSA's mammoth undertaking at best presents a snapshot of the states at one point in time. And even if all the niceties of polling were perfectly met, the allocation of states as "red" or "blue" is problematic due to sampling error. Here, we take the analysis of the SurveyUSA 50 state polls one step further. Rather than assign states based on who leads in the state surveys, we assign states probabilistically to the Democratic or Republican candidate based on the SurveyUSA state polls. Then, based on these probabilistic estimates, we ask the question, given the SurveyUSA results, what are odds of an Obama or Clinton victory in the Electoral College?
To do this, we conducted one million simulations (in MATLAB) of the Obama-McCain contest and then one million more simulations of the Clinton-McCain matchup. In each case we assume that the state estimates were correct except for sampling error. Using sampling theory and the assumption of simple random sampling, we draw one million estimates of the vote for each state. In each case we draw from a normal distribution with the observed mean (percent Democratic vs. percent Republican) and the standard deviation determined by the number of respondents in the state reporting a preference (always slightly under 600).
What do our results show? First, we pooled the state polls to ascertain the national vote, weighing each state's percent in proportion to the size of its House delegation. We also assign the District of Columbia as a 436th district and assign each Democratic candidate 85 percent of the vote to McCain's 15 percent. With these assumptions, the national popular "vote" is tight as of late February. Obama wins 51.5 percent versus McCain's 48.5 percent. Clinton also wins by an even razor thin margin, 50.7 to 49.3. With 30,000 cases, both estimates are statistically significant. McCain would be in the actual popular vote lead less than one time in 20.
That being said, our simulations yield a 88% chance of Obama beating McCain (with 306 Electoral College votes on average versus 233 for McCain), and a 74% chance of Hillary beating McCain (with 285 Electoral College votes on average versus 253 for McCain). About one percent of our simulated outcomes were Electoral College ties. (We ignored within-state variation in Maine and Nebraska, which divide their electoral votes by district.)
On the one hand, we find the expected numbers of electoral votes (the average from the simulations) for Obama or Clinton to be slightly higher than SurveyUSA reports. On the other hand, there is sufficient variance in the outcomes, so that McCain wins a nontrivial portion of the simulations, even with Obama as the opponent. Our two million simulations remind us that the popular vote winner is not always the Electoral College winner, although probably due mainly to chance -- the lottery aspect of the Electoral College -- and not any identifiable partisan bias in the 2008 Electoral College.
________________________________________
We thank Linda Liu for her technical assistance.
By Guest Pollster on March 14, 2008 2:18 PM
| Permalink
| Comments (40)
| TrackBacks (0)
January 15, 2008
By Guest Pollster
(Today's Guest Pollster contribution comes Professor Helmut Norpoth of Stony Brook University).
New Hampshire voters may mystify pollsters and pundits, but they have acquired an uncanny sense of picking candidates that go on to the White House. Whatever accounts for Hillary Clinton's surprising showing in her party's primary in New Hampshire, that victory makes her the best bet for Democrats to win the general election in November; likewise, John McCain's victory in the Republican primary in New Hampshire makes him the best hope for the GOP to retain the White House in November. These predictions are derived from a forecast model I developed that uses primary performance as the sole short-term predictor of the vote in the general election (the "Primary Model"). I have applied the model, with slight modifications, in the last three presidential elections, in which it correctly predicted the winners of the popular vote several months before Election Day. (See my 2004 paper in PS: Political Science & Politics). A race between the two New Hampshire winners, so the forecast, would be a nail-biter, with Clinton edging McCain by a margin of just a single percentage point of the two-party vote.
The use of primary elections to predict the outcome of the vote in the general election has some compelling advantages. One, it puts the estimation of a forecast model on a firm footing by letting us use elections all the way back to 1912, when presidential primaries were inaugurated. Two, it makes it possible to include both incumbent and opposition candidates in the model; granted, the incumbent candidate's performance may prove more powerful, but the effect of the out-party's primary showing is not negligible. And finally, the use of primaries as a predictor permits an unconditional forecast of the November vote at a very early moment. No ifs and buts. If one is willing to go with the outcome of the New Hampshire Primary, one can do it right now. The only uncertainty that remains is which of the match-ups will result from the nomination process. Chances are we may not have wait until the national conventions.
To measure primary performance in a standard format that allows for comparison across elections with varying numbers of candidates, I use an equivalent of the two-party vote in general elections. A candidate's primary showing is expressed as his or her vote relative to that of the winner (or in case of the winner in relation to the second strongest candidate). For incumbent-party candidates, the measure is adjusted, depending on whether they are sitting presidents or not. Moreover, the New Hampshire Primary is used only since 1952, when the state switched to a presidential-preference type of primary; prior to 1952, the model relies on the vote in all primaries.
Even though primary performance is the key, giving the model its name, the Primary Model also enlists a cyclical pattern of the presidential vote: the tenure of a party in the White House typically lasts between two to three terms. A compelling explanation for that dynamic is the term limit in presidential elections. Except for FDR, American presidents have eschewed running for more than two terms; and have been barred from doing so since then. The rule guarantees that incumbent presidents are missing from those contests in some periodic fashion, as is the case in 2008. In many such instances the absence of a sitting president with a high degree of popularity may improve the chances of the opposition party of capturing the White House. Given his high approval rating, Bill Clinton's ineligibility in 2000 probably hurt the Democratic prospects that year, although the absence of a much less popular George W. Bush in 2008 may be a blessing for the GOP. In any event, elections without a sitting president in the race tend to favor the opposition party more than elections with an incumbent running for another term. The Primary Model handles this dynamic by way of an autoregressive process (the presidential vote in the two previous general elections). In addition, given the use elections as far back as 1912, the model applies an adjustment for pre-1932 long-term partisanship.
From 1912 to 2004, the out-of-sample forecasts of the Primary Model pick the winner of the popular vote in 23 of the 23 elections, with 1960 being the only exception (and yes, that record includes Gore's popular vote win in 2004). The prediction equation for the presidential vote in 2008 (expressed as the Democratic share of the major-party vote) is:
.361 (RPRIM - 55.6) (-1) + .124 (DPRIM - 47.1) +.368 (VOTE04) -.383 (VOTE00) + 50.7 = .361 (RPRIM - 55.6) (-1) + .124 (DPRIM - 47.1) + 49.4
where RPRIM and DPRIM represent the primary support of the Republican (incumbent party) and Democratic (opposition party) nominees for President, capped within a 30-70 percent range, and Vote04 and Vote00 the Democratic vote shares in 2004 (48.8%) and 2000 (50.3%). The measure for the Republican candidate is inverted (-1) because the Democratic vote is used as the dependent variable. The formula produces the following forecasts of match-ups between the leading contenders in either parties (the vote for each match-up being the Democratic percentage of the two-party vote):
The PRIMARY MODEL predicts that in a race of New Hampshire Primary winners, Democrat Hillary Clinton would narrowly defeat Republican John McCain in the November general election (50.5 to 49.5 percent of the two-party vote). The predicted margin of victory, however, is so small that the confidence attached to this forecast is less than 60 percent, given the size of the forecast standard error (2.5). In match-ups between the Republican primary winner and Democratic primary losers, McCain would end up in a virtual tie with Barack Obama (49.9 to 50.1 percent) while defeating John Edwards (52.1 to 47.9 percent) by a margin close to one unit of the forecast standard error (2.6). At the same time, in match-ups between the Democratic primary winner and Republican primary losers, Clinton would dispatch Mitt Romney, Mike Huckabee, and Rudolph Giuliani by margins way beyond that error range. Finally, in match-ups between primary losers, both Obama and Edwards would beat any of the Republicans, and quite handily so in most cases.
That is no sign of partisan bias. Rather, it has to do with the Model assigning more weight to the primary performance of incumbent-party candidates than to the performance of out-party candidates. Nominating a primary loser, or even a candidate with a lackluster primary showing, costs the incumbent party more dearly than it does the out-party. Candidates not listed in the forecast table would do no better than the weakest one in their respective parties.
By Guest Pollster on January 15, 2008 1:35 PM
| Permalink
| Comments (7)
| TrackBacks (0)
