Breaking news: a survey indicates that…
The survey says…
Are political polls flawed or is the public expecting too much?
From: Christopher Borick
Wednesday, November 2, 2022 9:53 a.m.
Illustration by iStock.com/A-Digit
Perspective is a feature of Muhlenberg Magazine. This article originally appeared in the Fall 2022 issue.
In the 2016 presidential election, polls underrepresented Donald Trump’s vote. However, the polling errors that year were not that far off from those of previous elections. For example, in 2012, polling averages saw Barack Obama gain nationally by about one point. He ended up winning by about four points. Nobody seemed to care about the difference because Obama was ahead in the polls and eventually won re-election. In the Muhlenberg College Institute of Public Opinion (MCIPO) final poll in 2016, Trump trailed Pennsylvania by four points, but he won by less than a point. In both cases, the polls were off by similar margins, but in 2016 the MCIPO estimate was on the wrong side of the line of winners and losers. A similar disparity between polls and results was seen in other states that Trump narrowly won, such as Wisconsin and Michigan. This raised the possibility that the poll had encountered a systematic problem, and so pollsters sought to identify the problem and offer solutions.
One of the changes made by the MCIPO and other pollsters after 2016 concerns the weighting of the data. Pollsters almost always weight the data because random sampling usually does not lead to a sample that matches the parameters of the voter population. For example, the voting population is typically about 51 or 52 percent female, so if a sample ends up with only 47 percent female respondents, a pollster will weight the sample so that the results match the population. The MCIPO has traditionally weighted items such as gender identification, racial identification, and party registration.
Historically, education level has not had the same impact on voter choices as factors such as gender and race. This situation has changed dramatically since 2016. Today, people with college degrees are increasingly voting for Democrats, and people without college degrees, especially white people, are increasingly voting for Republicans. If the 2016 MCIPO polls had been weighted by education, they would have been about two points more accurate..
So the MCIPO started weighting for education midway through 2018, and those polls were more accurate. However, in 2020 we have seen similar things again polling errors that understated Trump’s support, MCIPO and many other pollsters. A range of survey methodologists dug deep into the 2020 results to see what the problem was, and there is no conclusive conclusion.
In “2020 Pre-Election Polling: An Evaluation of the 2020 General Election Polls,” the American Association for Public Opinion Research shared possible explanations for polling errors. “If the voters most supportive of Trump were the least likely to take the polls, the polling error can be explained as follows: self-identified Republicans who choose to take the polls are more likely to support the Democrats and those who who choose not to respond to surveys. are more likely to support Republicans,” the article said. “Unfortunately, this hypothesis cannot be directly assessed without knowing how nonrespondents voted.”
MCIPO did not change its weighting for the 2022 midterm elections, which had yet to take place at the time of this writing, to try to account for possible non-response. Attempting to weight such a “nonresponse bias” is guesswork, and the institute’s models were quite accurate in the last midterm election. Even if MCIPO produces a methodologically sound poll, its estimates could be a few percentage points lower than the final results, because that’s how sampling works. If you toss a coin 100 times, the most likely outcome is that you will get 50 heads and 50 tails, which reflects the reality of a two-sided coin. But sometimes you’ll get, say, 55 heads and 45 tails, not because of a methodological failure (eg, a faulty draw) but rather because of simple sampling error. To expect a poll, or even aggregates of polls, to reflect election results is to place far too high expectations on the methods employed. If the polls show a candidate ahead by three or four points in a race, don’t be shocked if that candidate narrowly loses. Methodological limitations and even modest last-minute shifts in voter sentiment could lead to such results.
With increasing challenges facing public opinion researchers, MCIPO is entering its third decade of operations with some important choices before it. In the spring semester of 2023, MCIPO will move into the new campus building, the Fahy Commons. There, the students and I will think about what’s next: online platforms that we can design and administer. Polls have changed significantly since I started conducting polls in the 1990s. Back then, telephone polls were entirely fixed. Phone numbers were associated with the state the phone was in, and response rates were relatively high. Today, there is a highly fragmented communication scene that makes sampling difficult, and getting individuals to agree to participate in a survey has never been more difficult.
Web-based survey options will certainly be an important part of MCIPO’s future, but the transition isn’t easy. The problem with some online surveys – those where participants are recruited via email – is that they may not capture a representative sample. There is no globality directory of e-mail addresses which is equivalent to a universe of telephone numbers or postal addresses serving as sampling frames. A potential way around this problem is called an online probability survey. People complete online surveys, but they are recruited by probabilistic means (by mail or telephone).
That’s my vision of where the institute is headed: once we get to our new facilities, we’re going to be fully committed to building our online probability panel for Pennsylvania, and maybe a more specialized one for Lehigh Valley. These investments will position MCIPO to further its mission of providing students with opportunities to engage in high-quality public opinion research and produce polls that accurately reflect the attitudes, beliefs, and behaviors of the populations we research. to better understand.
Christopher Borick is professor of political science and director of the Muhlenberg College Institute of Public Opinion.