I'd like to contribute a few thoughts on the performance of the public polls during the recently concluded New Jersey Gubernatorial race. On this topic, I bring a unique perspective, as the pollster for the Christie campaign, and I'd like to offer my thoughts not as any type of authority, but rather to contribute to an important professional discussion.

I should mention that, for what it's worth, some observers may have been surprised by the results on November 3rd, but neither Governor Elect Christie nor his advisers were surprised.

Before the cement hardens and ink dries on the post election wrap up, let me offer the following five thoughts:

1. The automated polls were more accurate than the live interview public polls, due in part to the methodology of the live interview polls.
   From polls that were in the field for an entire week (Quinnipiac) or even longer (FDU), to polls that oversampled Democrats (Democracy Corps, among several others) to polls that asked every single name in the ballot (Suffolk), an essential reason for the poor performance of the live interview polls had less to do with the fact that a live person was administering the poll and more to do with methodological issues.
2. The partisan spread in the polls ought to be reported up front.
   Some public pollsters make it difficult to determine how many Republicans, Democrats and unaffiliated voters they interviewed. Why not just put it into the toplines? Reporters and bloggers should demand this before they report on the results. Not to pick on Quinnipiac, but they had Corzine and Christie winning about the same amount of their own partisans, and they had Christie winning Independents by 15 percentage points, and yet they STILL had Christie trailing overall by 5 points. Quinnipiac did not publish their partisan spread, but then an astute blogger was able to ascertain the fact that there were, in fact, too many Democrats in the sample. Other polls, notably Democracy Corps, regularly produced samples with too many Democrats (though, in their parlance, some of these were "Independent - Lean Democrat"). That their sample was loaded up with Democrats had the obvious effect on their results. Whether this was intentional or not, I would leave to others to speculate.
3. In general, RDD methodology is a bad choice in New Jersey, if the goal is predictive accuracy.
   In New Jersey, there are many undeclared voters (commonly but mistakenly referred to as Independents). These undeclared voters identify themselves as Republicans or Democrats - even though they are not registered that way. In our polls, we frequently showed a Democrat registration advantage that matched their actual registration advantage - but when it came to partisan ID, the spread was more like a six point Democrat advantage. By using a voter list, we knew how a respondent was registered - and by seeing how they ID'ed themselves, we gained insight into the relative behavioral trends of undeclared voters and even registered Democrats who were self identifying as Independents. Public pollsters who dialed RDD missed this. Partisan identification in New Jersey is not enough, if the goal is to "get it right."
4. The public polls oversampled NON voters.
   Again, this is a function of RDD versus voter list dialing. It is easy for someone to tell a pollster they are "very likely" to vote. With no vote history and no other nuanced questions, the poll taker has little choice but to trust the respondent. Pollsters who use voter lists have the benefit on knowing exactly how many general elections a respondent may have voted in over the past five years, or when they registered. By asking several types of motivation questions, the pollster can construct turnout models that will have a better predictive capacity. The public polls did not seem to do this.
   
   To this end, we had heard all about the "surge strategy" that the Corzine campaign was going to employ. This refers to targeting "one time Obama voters" and driving them out in force on election day. With voter lists, we were easily able to incorporate some "surge targets" into our sample. After running our turnout models, we saw no evidence that the surge voters would be game changers.
5. The Daggett effect was overstated in the public polls.
   Conventional wisdom holds that Independent candidates underperform on election day. But the reality is, many analysts could have easily predicted Daggett's collapse, based not on history, but on simple a simple derivative crosstab: for example, voters who were certain to vote for Daggett AND had a very favorable opinion of him. They could have asked a "blind ballot" where none of the candidate choices were read. We did these things - and we estimated Daggett's true level of support to be around 6%.

What does it mean for next year? At the very least, journalists, bloggers and reporters need to ask more questions about the methodology and construction of the poll sample. They need to understand the partisan spread, and the extent to which it conforms to reality. They need to know how long the survey was in the field. They also need to beware of polls being released that are designed to manipulate opinion rather than manage it. They need to ask if certain polls are being constructed to reflect what is happening, or if they are being constructed to reflect what the poll sponsor would LIKE to happen. The public polls add to the dialogue, and given their ever increasing contributing role, we all ought to be more demanding when reporting their results.

These comments are prompted by the paper Comparing the Accuracy of RDD Telephone Surveys and Internet Surveys Conducted with Non-Probability Samples by Yeager, Krosnick, et al, and by Mark Blumenthal's two excellent articles in the National Journal reviewing their paper.

The paper's conclusions were based on a comparison between six "benchmarks" and the findings of the various polls they examined. They assumed that the benchmarks were perfectly accurate, and that any differences between the polls and the benchmarks were "errors." I believe that this is not the case and that some of the benchmarks were inaccurate because of the social desirability bias that is often found in surveys where respondents are interviewed, by telephone or in-person, by live interviewers.

Social desirability bias occurs where respondents are not comfortable telling interviewers the truth because they are embarrassed to do so, or where their behavior or attitudes may be seen as unethical, immoral, anti-social or illegal.

Our online surveys have always found substantially more people than our telephone surveys who tell us they are gay, lesbian or bisexual (by a 3-to-1 margin). Our online surveys also find fewer people who claim to give money to charity, clean their teeth, believe in God, go to religious services, exercise regularly, abstain from alcohol, or drive under the speed limit.

Furthermore, in-person surveys by the Census Bureau report substantially more people claiming to have voted in elections than actually voted. If there is a better explanation than social desirability bias, I haven't heard it.

This conclusion - that surveys with live interviewers underreport "socially undesirable" behavior is supported by the data used by Yeager et al.

Our online survey, used by Yeager, found more smokers and more people having had 12 drinks in a life time than either the benchmark surveys conducted by government agencies or the RDD sample (and our own telephone surveys). Our online survey found that (to the nearest whole number) 28 percent were smokers compared to 26 percent in the RDD sample and 22 percent in the benchmark survey. Our online survey found only eight percent who had not had 12 drinks in their lifetime compared to 15 percent in the RDD sample and 23 percent in the benchmark survey.

Another government study, the NHANES study reported that 24.9 percent of adults said they were smokers but that blood tests showed that an additional 4.5 percent had smoked in the previous 24 hours but had not reported it when asked by an interviewer. The resulting NHANES estimate of 29 percent is closer to our estimate of 28 percent than to Knowledge Network's 26 percent or the RDD sample's 24 percent.

Two of the six benchmarks used by Yeager et al come from government sources where one would not expect to find any social desirability bias. In both cases, the Harris Interactive data were slightly closer to the benchmark data than were the findings of the RDD telephone survey. Our surveys found 28 percent of adults with passports compared to 30% for the RDD sample and the 23 percent in benchmark. Our survey found 92 percent having a driver's license compared to 93 percent in the RDD sample and the 89 percent benchmark.

In addition to the presence or absence of live interviewers there is one other reason why our online polls may have less social desirability bias than most telephone and in-person surveys. Our panel members have agreed in advance to be surveyed, which suggests that they trust us with confidential information, and are therefore more likely to tell the truth.

All this evidence suggests that the Harris Interactive data used by Yeager et al is generally more accurate than the RDD sample and that some of the so-called benchmarks probably overstate socially desirable behaviors because they were obtained in surveys with interviewers.

Saturday Night Live's sketch mocking Obama prompted CNN to run a story stating that the 'SNL' Obama sketch marks end of [Obama's] honeymoon. Actually, SNL is not leading public opinion here. Polling suggests that Obama's honeymoon ended in early August. Since then, Obama's job approval rating has remained essentially flat.

If you are an Obama supporter, you might ask how this is possible, since an Oct. 1-5 AP-GfK survey shows a resurging six percentage point increase in support for Obama since their Sept. 3-8 survey. Or, if you oppose Obama, you might point to the slight downward trend in Obama's job approval among all polling firms from early September clearly evident on Pollster.com.

What is going on here is that Pollster.com's trend line behaves fine when there are lots of polls to average together, but it does not work as well when two daily tracking polls are averaged together with more sporatic national polling. The two daily tracking polls - Gallup and Rasmussen - consistently find lower Obama job approval ratings than other polling firms. In addition to these two daily tracking polls, there are approximately bi-monthly internet polls from YouGov/Polimetrix and Zogby that also consistently show lower Obama job approval numbers compared to other polls.

These so-called "house effects" whereby different pollsters consistently report different numbers is well-known. I do not want to get sidetracked into speculation about why these polls have lower numbers, since we really cannot know what the true population value is for Obama's job approval rating.

What is interesting is what happens when these polls are disaggregated into two types (1) the tracking and internet polls and (2) all other polls.

To examine the first type of polls, let's use Pollster.com's filter tool to include all internet polls and the two daily tracking polls.

According to this trend estimate, Obama's job approval rating leveled out in early August at about 50 percent, and may be slightly increasing since.

To examine the second type of polls, let's use Pollster.com's filter tool to exclude all internet polls and the two daily tracking polls.

According to this trend estimate, Obama's job approval rating leveled out in early August at about 53 percent.

Seen in this light, Obama's job approval rating has remained steady since early August, and it is here that Obama's honeymoon likely came to an end. Most pollsters took a vacation during August, except those conducting the first type of polls, which show lower Obama job approval than the second type. The bump up in Obama's job approval at the beginning of September is an artifact of the increased number of the second type of polls conducted when Obama delivered his health care speech to Congress. Subsequently, the absence of the second type of polls allows the first type of polls to again dominate the trend line, thereby giving the appearence that Obama's approval is now decreasing from the (non-existent) short-term early-September rally. The different mixes of the first and second types of polls are confounding the trend line and incorrectly coloring perceptions of the direction of Obama's job approval rating. Indeed, if you squint closely at Pollster.com's trend line for all pollsters, you'll see a long-term periodicty that apparently fluctuates along with the mix of the first and second types of polls.

[Editor's Note: So that Professor McDonald's commentary will always match the graphics, we  replaced the embedded, interactive version of charts with screenshots, although you can click the link above each chart to see the most recently updated version with the filtered polls he selected].

The polling and pundit world is now looking to see if President Obama's speech will rally public support for his health care reform plan. In addition to looking at the stream of polls that will now follow, I direct your attention, hot off the presses, to the latest issue of the journal Political Communication. A timely article by Brandon Rottinghaus provides a broader political science view on presidential efforts to influence public opinion. What we know from George Edwards' book, On Deaf Ears: The Limits of the Bully Pulpit (Yale, 2003), is that it is difficult for presidents to succeed at influencing public opinion. However, Rottinghaus's article provides evidence for why Obama correctly chose to take his best shot in a nationally televised speech.

The article uses "a comprehensive data set spanning 1953 to 2001," to examine "several strategic communications tactics through which the presidents might influence temporary opinion movements." Specifically, it finds that "presidential use of nationally televised addresses is the most consistently effective strategy to enhance presidential leadership, but the effect is lessened for later serving presidents." In contrast, other strategies such as those involving domestic travel do not have positive effects and "televised interactions"--press conferences and the like - tend to have negative effects. While some may not be surprised with these findings, it is good to have empirical evidence to wrestle with.

But getting to the point, how will this now play out for Obama? My sense is that Obama's speech will come out on or above average in impact, though there is a question of what its half-life will be. What I see as most important, however, is not the new polls that we will soon see (if they are not out already). Putting Rottinghaus' article aside, what will count most is not what the public thinks at this moment, but rather the extent to which Democratic leaders unite around Obama's plan (which may well be close to Baucus'?); it is this elite consensus that will enable any positive effect of the speech to last or even widen. This assumes that the consensus will be more salient and striking than any continued Republican opposition.

Echoing the famous political scientist, V.O. Key, what matters more than the immediate polls is political leadership more broadly. The speech itself is the start of what could be a stronger consensual message than we have seen to date from Democratic and potentially other political leaders. The relevant public opinion research comes from Richard Brody's book on presidential leadership, (Assessing the President: The Media, Elite Opinion, and Public Support. Stanford, 1991), John Zaller's seminal book on public opinion (The Nature and Origins of Mass Opinion, Cambridge, 1992), and what Ben Page and I examined (The Rational Public. Chicago, 1992).

Larry Jacobs and I (Politicians Don't Pander, Chicago, 2000) looked at the President Clinton's 1993-94 health care reform effort from this perspective. What happened there was the Democratic leaders never supported any Clinton plan, and this, along with the strong Republican leadership opposition caused the public to become apprehensive and turn against health care reform. This happened much earlier in the legislative process than what occurring now, as the Clinton plan got to Congress later in Clinton's first term. In contrast, we are at that same juncture now --- however, earlier in Obama's first term but later in the legislative process, as there are now actual bills that have made it through congressional committees. Clinton never made it that far. The Democrats now have a better chance than Clinton did, since at this moment they are poised to unite around a president's plan. But if they don't do that quickly, then it's 1994 all over again. If by all appearances they come together, they can prevent public support from tapering off and very likely increase it.

In the end, Obama may have timed his entry into the fight just right--it's earlier than when Clinton entered the actual legislative fray in 1994--and this may have been the only way he could have gotten a major health care reform bill through. Given the financial crisis, the stimulus bill, and the two wars, he may well have been stopped in his tracks earlier on--without the health care reform bills making it through multiple committees as they have. He needed to enter the fight when he could rally congressional support in both houses, with drafted legislation in hand and already substantially debated. Of course we will never know since as we can't replay history. For now, the main point is don't just watch the polls-watch the leaders. The public will not just be responding to Obama but to the extent to which he has liberal, blue dog, and any (albeit unlikely) Republican leadership support.

Technological capabilities can become temptations to conduct research studies that add nothing to our knowledge of public opinion, just because we can. Get thee behind me, Satan!

For example, it would be no problem, technologically, to display squiggly lines with the moment-by-moment reactions of a panel of viewers to the blathering of the talking heads on news show panels. The Onion demonstrates what a mess that would be, in a parody entitled "New Live Poll Allows Pundits to Pander to Viewers in Real Time."

What would happen if we let the talking heads see whether viewers at home agreed or disagreed with what they were saying, "using the Insta-Poll Tracker on our web site"? The talking heads would become self-conscious about the direction of their own squiggly line and start tailoring their statements...word by word...to make the squiggly line go up.

Insta-polls like September 9th's CNN/Opinion Research Corporation poll of adults who watched President Barack Obama's address to Congress may have a similar effect on poll respondents. Mark Blumenthal correctly points out the age-old problem of such polls--the partisan make-up. Last night, the audience for this address was heavily weighted with Obama supporters rallying to watch their leader, supplemented with a few civic-minded Americans who would watch any Presidential address, regardless of their own partisanship. Of the 427 adults in this study, all of them interviewed September 5-8 in advance of the speech, and all of whom indicated both an intention to watch the speech and a willingness to be re-interviewed after the speech, 18% were Republicans, 45% Democrats. These kinds of post-speech poll samples always skew heavily in favor of the speaker. Pollster.com's report on this poll last night squeezes out what knowledge can be gleaned by comparing the "bump" among this group of speech watchers to the bump registered among similarly situated groups of speech watchers in the past.

The problem with this kind of insta-poll may be exacerbated when the study is designed, as this one was, to compare the pre-speech responses of speech watchers to opinions after the speech. In the pre-speech survey, I would guess that respondents would strive to express their opinions as forthrightly as possible, as most survey respondents do. In the follow-up poll after the speech, however, I am afraid respondents would be like the Onion's self-conscious pundits. They'd be aware that they are about to become as much a part of the story as South Carolina Republican Rep. Joe Wilson who heckled the President. They'd tailor their answers to make their leader look good. Drawing much of a conclusion from their answers would not be any fairer than judging the entire Republican caucus by the boorishness of a few Members.

I woke up on Tuesday morning to find several emails pointing me to Gary Langer's blog posting, which quoted extensively from a supposedly new paper by Jon Krosnick. These data and results appeared previously in a paper, "Web Survey Methodologies: A Comparison of Survey Accuracy," Krosnick coauthored with me and presented at AAPOR in 2005. The "new" paper has added some standard error calculations, some late arriving data, and a new set of weights, but the biggest changes in this version are a different list of authors and conclusions.

The 2005 study compared estimates from identical questionnaires fielded to a random digit dial (RDD) sample by telephone, an Internet-based probability sample, and a set of opt-in panels. Of these, Internet probability sample had the smallest average absolute error, followed closely by the RDD telephone survey, and the opt-in Internet panels were around 2% worse. In his presentation of our paper at AAPOR in 2005, Krosnick described the results of all the surveys, both probability and non-probability, as being "broadly similar." My own interpretation of the 2004 data, similar to James Murphy's comment on AAPORnet, was that although the opt-in samples were worse than the two probability samples, the differences were small enough--and the cost advantage large enough--to merit further investigation. Even if it were impossible to eliminate the extra 2% of error from opt-in samples, they could still be a better choice for many purposes than an RDD sample that cost several times as much.

Krosnick now concludes that "Non-probability sample surveys done via the Internet were always less accurate, on average, than probability sample surveys" and, tendentiously, criticizes "some firms that sell such data" who "sometimes say they have developed effective, proprietary methods" to correct selection bias in opt-in panels.

In fact, the data provide little support for Krosnick's argument. The samples from the opt-in panels were, as we noted in 2005, unrepresentative on basic demographics such as race and education because the vendors failed to balance their samples on these variables, while the two probability samples were balanced on race, education, and other demographics. This is not a result of probability sampling, but of non-probabilistic response adjustments. It is too late to re-collect the data, but the solution (invite more minorities and lower educated respondents) doesn't involve rocket science.

Instead, Krosnick tries to fix the problem by weighting, and concludes that weighting doesn't work. A more careful analysis indicates, however, that despite the large sample imbalances in the opt-in samples, weighting appears to remove most or all selection bias in these samples. Because the samples were poorly selected, heavy weighting is needed and this results in estimates with large variances, but no apparent bias. In fact, if we combine the opt-in samples, we can obtain an estimate with equal accuracy to the two probability samples.

First, consider the RDD telephone sample. The data were collected by SRBI, which used advance letters, up to 12 call attempts, $10 incentives for non-respondents, and a field period of almost five months. Nonetheless, the unweighted sample was significantly different from the population on ten of the 19 benchmarks. RDD samples, like this one, consistently underrepresent male, minority, young, and low-education respondents. These biases are reasonably well understood and, for the most part, can be removed by weighting the sample to match Census demographics.

Next, consider the Probability Sample Internet Survey, conducted by Knowledge Networks (KN). The unweighted sample does not exhibit the skews typical of RDD. How is this possible, since the KN panel is also recruited using RDD? Buried in a footnote is an explanation of how KN managed to hit the primary demographic targets more closely than SRBI (which had a much better response rate). The answer is that "The probability of selection was also adjusted to eliminate discrepancies between the full panel and the population in terms of sex, race, age, education, and Census region (as gauged by comparison with the Current Population Survey). Therefore, no additional weighting was needed to correct for unequal probabilities of selection during the recruitment phase of building the panel." That is, the selection probabilities that are supposedly so important to probability sampling were not used because they would have generated an unrepresentative sample!

The opt-in panels, for the most part, were not balanced on race and education. Only one of the opt-in samples, Non-Probability Sample Internet Survey #6 actually used a race quota. Another, the odd Non-Probability Internet Sample #7, claims to have sent invitations proportionally by race and ended up with 46% of the sample white, despite a 51% response rate. (This survey will be excluded from subsequent comparisons.) Non-probability Sample Internet Survey #1 involved large over-samples of African Americans and Hispanics. I could find no explanation of how Krosnick dealt with the oversamples in the 2009 paper, but it should either match exactly (if the conventional stratified estimator is used) or be far off (if the data are not weighted). In fact, the proportion of whites and Hispanics is off by 1% to 2%.

The selection of a subsample of panelists for a study is critical to the accuracy of opt-in samples. Regardless of how the panel was recruited, the combination of nonresponse or self-selection at the initial stage along with subsequent panel attrition, will tend to make the panel unrepresentative. In 2004, we instructed the panel vendors to use their normal procedures to produce a sample representative of U.S. adults. The practice then (and perhaps now for some vendors) was to use a limited set of quotas. If you didn't ask most opt-in panels to use race or education quotas, they wouldn't use them.

Even without correcting these obvious imbalances, the opt-in samples provided what most people would consider usable estimates for most of the measures. For example, the percentage married (unweighted) was between 53.7% and 61.5% vs. a benchmark of 56.5%). The percentage who worked last week (unweighted) was between 53.6% and 63.1% (vs. a benchmark of 60.8%). The percentage with 3 bedrooms (unweighted) was between 41.2% and 46.1% (vs. a benchmark of 43.4%). The percentage with two vehicles (unweighted) was between 40.1% and 46.9% (vs. a benchmark of 41.5%). Home ownership (unweighted) was between 64.8% and 72.8% (vs. a benchmark of 72.5%). Has one drink on average (unweighted) was between 33.8% and 40.2% (vs. a benchmark of 37.7%). The KN sample and phone samples were better, but the difference was much less than I expected. (Before doing this study, I thought the opt-in samples would all look like Non- probability Sample Internet Survey #7.)

The 2009 paper attempts to correct these imbalances by weighting, but the weighted results do not show what Krosnick claims. He uses raking (also called "rim weighting") to compute a set of weights that range from .03 to 70, which he then trims at 5. The fact that the raking model wants to weight a cell at 70 is a sign that something has gone wrong and can't be cured by arbitrarily trimming the weight. If there really are cells underrepresented by a factor of 70, then trimming causes severe bias for variables correlated with the weight and not trimming causes the estimates to have large variances. In either case, the effect is to increase the mean absolute error of estimates.

The fact that the trimmed and untrimmed weights have about the same average absolute error does not mean that weighting is unable to remove self-selection bias from the sample. The mean absolute error is a measure of accuracy. It is driven by two factors: bias (the difference between the expected value of the estimate and what it is trying to estimate) and variance (the variation in an estimate around its expected value from sample to sample). The usual complaint about self-selected samples is that you can never know whether they will be biased or the size of the bias. Inaccuracy due to sampling variation can be reduced by just taking a larger sample. Bias, on the other hand, doesn't decrease when the sample size is increased.

Obviously, uneweighted estimates from these opt-in samples will be biased because the vendors ignored race and education when selecting respondents. This wouldn't have been difficult to fix, but it wasn't done. Apparently very large weights are needed to correct demographic imbalances in these samples, but the large weights give estimates with large variances and, hence, a high level of inaccuracy. If one tries to control the variance, as Krosnick does, by trimming the weights, then the variance is reduced at the expense of increased bias. The result, again, is inaccuracy. We are asking the weighting to do too much.

A simple calculation shows that all of Krosnick's results are consistent with the weighting removing all of the bias from the opt-in samples. One way to combat increased variability is to combine the six opt-in samples. Without returning to the original data, a simple expedient is to just average the estimates. Since the samples are independent and of the same size, the average of 6 means or proportions should have a variance about 1/6 as large as the single sample variances. The variance is approximately equal to the square of the mean absolute error which, after weighting, was about 5 for the opt-in samples, implying a variance of about 25. If there is no bias after weighting, then the variance of the average of the estimates should be 25/6 or approximately 4, implying a mean absolute error of about 2%.

How does this prediction pan out? If we average each of the weighted estimates and compute the error for each item using the difference between the average estimate and the benchmark, the mean absolute error for the opt-in samples is 1.4% -- almost identical to the mean absolute error for each of the weighted probability samples. That is, the amount of error reduction that comes from averaging the estimates is about what would be predicted if the all bias could have been removed by weighting. Thus, the combination of these six opt-in samples gives an estimate with about the same accuracy as a fairly expensive probability sample (which also required weighting, though not as much).

There is no reason, however, why you should need six opt-in samples to achieve the same accuracy as a single probability sample of the same size. If the samples were selected appropriately, then we could avoid the need for massive weighting. It is still an open question what variables should be used to select samples from opt-in panels or what the method of selection should be. In the past few years, we have accumulated quite a bit of data on the effectiveness of these methods, so there is no need to focus on a set of poorly selected samples from 2004.

Probability sampling is a great invention, but rhetoric has overtaken reality here. Both of the probability samples in this study had large amounts of nonresponse, so that the real selection probability--i.e., the probability of being selected by the surveyor and the respondent choosing to participate--is not known. Usually a fairly simple nonresponse model is adequate, but the accuracy of the estimates depends on the validity of the model, as it does for non-probability samples. Nonresponse is a form of self-selection. All of us who work with non-probability samples should spend our efforts trying to improve the modeling and methods for dealing with the problem, instead of pretending it doesn't exist.

There's been a lot of discussion of how seniors, who already are on Medicare, appear to be the least supportive age group of President Obama and the Democrats' plans for enacting health care reform. Seemingly at the center of seniors' concerns is the idea of cutting federal support for a program called Medicare Advantage. According to a Los Angeles Times article:

Although scaling back payments would have no effect on a sizable majority of Medicare users, it would create an opening for opponents to make the blanket allegation that the president wants to cut back on Medicare benefits -- as some Republicans are already starting to say.

Also, of course, seniors were more likely to vote for John McCain in last year's presidential election than were younger voters, who went overwhelmingly for Obama.

The diagram below (which you may click on to enlarge) compares different age groups' attitudes toward health care reform in four recent polls. Compiling these percentages was not as easy as I thought it might be, for a variety of reasons. First, only some pollsters make a public release of cross-tabulations between demographic characteristics and health care-related attitudes (other pollsters reserve such cross-tabs for paid subscribers). Second, age cross-tabs on a common attitude item were not always available. My plan was to use general favor/oppose items toward Obama and the Democrats' reform plan, but such an item was not always available so I had to substitute other types of items, as described below. Third, different pollsters use different cut-points to create their age groups. There's always a youngest age group, for example, but some pollsters bracket it from 18-29 whereas others use 18-34; similar discrepancies exist for other age groups, as well.


Having said all this, the pattern of seniors showing the least support for Obama/Democratic reform plans is clear and well replicated. For any given color of bar (purple, light blue, green, or orange; each representing a different pollster and question), the shortest height is with the seniors.

One other thing to notice is that two polls, ABC/Washington Post and The Economist/YouGuv, only reported on a 30-64 broad middle-age group rather than having two groups like other pollsters; whether groups in the lower and upper halves of the 30-64 age range were combined because they did not differ much in their responses, or the pollsters never broke 30-64 year-olds into smaller subsets, I don't know. For these two polls, I have taken the percentage on the respective attitude measures attributed to 30-64 year-olds and plotted them twice (linked by a light-blue or green horizontal line), where a 30s-40s group and a 50s-60s group would ordinarily go. Now that these "housekeeping" matters are out of the way, here are the question wordings used:

Survey USA (Aug. 19): “Now I am going to tell you more about the health care plan that President Obama supports and please tell me whether you would favor or oppose it. The plan requires that health insurance companies cover people with pre-existing medical conditions. It also requires all but the smallest employers to provide health coverage for their employees, or pay a percentage of their payroll to help fund coverage for the uninsured. Families and individuals with lower- and middle-incomes would receive tax credits to help them afford insurance coverage. Some of the funding for this plan would come from raising taxes on wealthier Americans. Do you favor or oppose this plan?”

ABC/Washington Post (Aug. 13-17): “Reform’s supported by 58 percent of adults under age 30, but 44 percent of 30- to 64-year-olds and just 34 percent of seniors, apparently concerned about its potential impact on Medicare” (this quote comes from an article and does not depict the actual survey item).

Economist-You Gov (Aug. 16-18): “If President Obama and Congress pass a health care reform plan, do you think you personally would receive better or worse care than you receive now?" (% Saying Better).

Kaiser Family Foundation (Aug. 4-11): “Do you think you and your family would be better off or worse off if the president and Congress passed health care reform, or don’t you think it would make much difference?” (% Saying Better).

The four polls above were not the only ones that made some type of age-related comparison. Others did, as well, but their age groupings and/or survey items appeared non-comparable in some way to the four polls whose results I plotted. Two additional polls are as follows:

A Harris Interactive poll used what I think are the most interesting age-group descriptors (shown in Table 2 of the linked document): "Echo Boomers (18-32), Gen. X (33-44), Baby Boomers (45-63), Matures (64+)." Harris plotted the percentage of respondents in each age group who rated Obama's job performance in various issue domains as "fair" or "poor." On health care, higher percentages of Matures (71%) and Gen. X (69%) gave Obama these unflattering ratings than did Echo and Baby Boomers (each 62%). Along with some of the figures from other polls plotted above, this finding from Harris shows a non-linear trend (i.e., support does not decline in perfect progression from the youngest to the oldest voters).

Finally, a Penn, Schoen, & Berland poll released in conjunction with AARP reported only comparisons between respondents younger than 50 and 50-plus. A section of this poll's report entitled "Specific Policy Proposals" (on pages 6-7) is perhaps the most worthy of attention. On most of the items, the younger respondents are more favorably inclined, but on others, there is little or no difference.

(Cross-posted to Health Care Polls)

Perhaps the most contentious issue among congressional negotiators and interest groups in Washington, DC (and elsewhere) is the so-called public option. The idea is that the government would create a new health-insurance program (modeled to one degree or another on Medicare, the government insurance program for seniors) that people could join. Proponents argue that, by having it compete with private insurers, the public option would help control costs. Opponents, on the other hand, see the public option as yet another government intrusion into an area they feel should be left to the private market.

Where does the public seem to stand? Not surprisingly, the public option has been widely polled, and we shall focus exclusively on it today. As seen in the diagram below (which you can click on to enlarge), levels of support for the public option vary widely according to different polls, despite the relative consistency of question wording (all the survey items refer in some fashion to the public option being a government health-insurance program that would compete with private insurance companies). The predominant trend, I would say, is that a majority of respondents supports a public option, with five of the eight polls showing between 52-66 percent in favor.


Still, though, two other polls show support in the mid-40s and one poll (Rasmussen) has support way down at 35%. What to make of this? Let's start with Rasmussen. Whereas Rasmussen's presidential-election polling has tended to be highly accurate (relative to the actual results), other types of polls from this outfit appear to have had a Republican slant. Here are some examples:

*Whereas most polls tended to have George W. Bush's job-approval ratings during the waning months of his administration in the low-30s or even the 20s, Rasmussen consistently had it around 35%.

*Whereas virtually every pollster other than Rasmussen has shown a majority of voters to prefer the Democrats (at this early point) in next year's U.S. House elections, Rasmussen has been showing the Republicans in the lead (albeit with large percentages undecided).

Polling analysts refer to systematic differences in the results (on the same basic issue) between different survey firms (or survey "houses") as house effects. These may stem from different firms' practices regarding question-wording, sample weighting, etc. On health care reform and other issues, it looks to me as though Rasmussen has a substantial house effect.

There's one other aspect of the public-option polling I'd like to point out. As can be seen in the diagram above, I have highlighted in red the words "option" and "offering" in the wording of some of the survey items. It appears that wordings stressing the voluntariness of the public option (i.e., that it is an "option," or something "offered" to the consumer) tend to elicit higher support than wordings that don't highlight voluntariness as much. This is just a hunch. If anyone has other explanations for the large variation in support between the polls, please share them in the comments section.

(Cross-posted to Health Care Polls)

Matthew Yglesias calls the public "ill-informed and hypocritical" based on a New York Times poll that found "Most Americans continue to want the federal government to focus on reducing the budget deficit rather than spending money to stimulate the national economy... [y]et at the same time, most oppose some proposed solution for decreasing it."

The problem, however, is that the available evidence doesn't support Yglesias's conclusion (which is encouraged by the way the poll is framed in the Times). When you look at the raw poll results (PDF), you'll see that the public prefers reducing the deficit to stimulating the economy 58%-35%, but 53% oppose cuts in public services and 56% oppose higher taxes. Those numbers may seem "ill-informed and hypocritical," but the problem is that we're dealing with aggregate data (this is what is known as an ecological inference problem). We can't draw any strong conclusions about the proportion of individual members of the public who have incoherent preferences about deficit reduction without access to the raw data. Ideally, we would break out the members of the public who advocate deficit reduction over stimulus and see how many of them oppose both higher taxes and reduced services. That's the quantity of interest, but it's unfortunately not available to us at this point.

Update 7/30 12:12 PM: Yglesias has generously updated his post to note that you "can't infer very much about individual preferences from this aggregate data."

Just to briefly elaborate on the point I made last week, here are comparable plots of President Obama's overall job approval and approval of his handling of health care:

As you can see, what's happening on health care is a leading indicator of the end of Obama's honeymoon period. As we return to our normal, highly polarized political climate, most Republicans and Republican-leaning independents will disapprove of a Democratic president's performance in office and his handling of high-salience issues, especially in a bad economy. As a result, Obama's numbers will inevitably decline across the board -- this reality shouldn't be surprising to anyone who works in or reports on politics.

Going forward, we should focus on more important questions. First, how much will Obama's approval numbers decline? Given the state of the economy, it wouldn't be surprising to see him in the low- to mid-40s by the end of the year. Second, what is the distribution of opinion on Obama's handling of health care? Aggregate public opinion on the issue is less relevant than how it's playing in the states of key senators whose votes will determine the fate of the legislation in Congress.]]> http://www.pollster.com/blogs/nyhan_the_end_of_the_obama_hon.php http://www.pollster.com/blogs/nyhan_the_end_of_the_obama_hon.php Barack Obama Mon, 27 Jul 2009 11:23:10 -0500