With a wildly successful voting cycle behind them, some pollsters believe that tactics widely adopted in 2022 could help carry them through the next presidential race. But even adherents of the tactic say it may not be a panacea, especially if former President Donald J. Trump is once again on the ballot.
Pollsters are increasingly weighing surveys based on who respondents remember voting in past elections, in addition to adjusting for standard demographics such as race and age. This tactic has long been used in other countries to improve polling accuracy, but has only recently become widely used in the United States.
“We are all freaked out,” said Cameron McPhee, chief methodologist at SSRS, a CNN polling partner and polling agency that is weighing some of its polls on the 2022 vote. He added, “We are all excited about the changes we made to the 2022. , but I think there is still a big question mark” ahead of 2024.
Given the recalled votes, pollsters can more easily correct partisan imbalances in who responds to polls, and in recent years Democrats have tended to respond to polls at higher rates than Republicans. Perhaps more importantly, weighting on recalled votes could specifically increase the influence of Trump supporters, a group that polls have struggled to accurately measure in 2016 and 2020.
Adoption of the tactic by pollsters in the United States is far from universal. Several leading pollsters achieve accurate results without it — including The New York Times/Siena College, which FiveThirtyEight named the most accurate political pollster in America after its 2022 cycle.
Overall, 2022 is one of the most accurate years for voting in recent history, according to a analysis by FiveThirtyEight. Many polls “will probably get 2022 right even without that extra weighting step, because we did,” said Patrick Murray of Monmouth University Polls.
After 2016, post-election analysis found that opinion polls consistently underrepresented less educated voters, who tend to support Trump disproportionately. To remedy this, pollsters widely adopted education as an additional survey weight, and the poll-accurate cycle in 2018 appears to reflect a return to normalcy.
But in 2020, the polls were more biased than any modern election, overrepresenting Democratic support by nearly five percentage points, compared to three percentage points—a more normal amount of error—in 2016.
“I think one of the reasons for the success of 2022—and even to some extent 2018—is because Trump himself is not on the ballot.” said Mr. Murray. “If history is a guide, we will probably see that nonresponse go into 2024 based on how the Republican nomination went.”
The 2020 election presents another different challenge — it takes place in the middle of a pandemic. The poll found that some Americans, stuck at home and lonely, are more likely to respond to the survey. While that may initially be seen as a boon, it may introduce more bias if it means unequal adherence to stay-at-home orders adding another source of bias in who picks up the phone.
The weighting on the drawn sound is not without its concerns.
Voters shown to have poor memories of who they voted for or even whether they voted at all, were typically more likely to remember voting for the winner. One study Canadian voters find up to a quarter of voters inconsistent when considering who they have voted for.
This misrepresentation of past votes can push opinion polls in different directions depending on who won the most recent election. In 2022, that means respondents are more likely to say they have endorsed Joe Biden, and pollsters using the recalled vote will eventually reduce their weight, meaning Republicans are endorsed.
But with previous victors from different sides, the effect would be reversed. An assessment by The Times found that weighing the 2020 polls in favor of a 2016 recall vote would make them even more biased against Mr Biden. And report of the American Association for Public Opinion Research examining how the 2020 poll could be improved found that those that considered recalled votes did no better than those that did not.
Similarly, in 2022, the weighting with the withdrawn vote will make the Times/Siena poll less accurate. As published, without considering the recalled votes, the latest polls for the Senate, governor, and House elections had an average error of less than two percentage points and a zero bias toward Democrats or Republicans. When weighed using the recalled votes for the 2020 election results, the average error will increase by a percentage point, and overall the polls will lean slightly Republican.
But that may be a consequence of other decisions The Times made, which included weighting demographic information available on voter files that wasn't always available to other pollsters.
Other pollsters have found the ballot method to yield a significant increase over the usual weighting scheme. SSRS uses various weighting methods in 2022, including recalling votes for some of its polls, and also experimenting with weighting on political identification. Its post-election analysis found that using recalled votes as the weight would be the most accurate overall approach, increasing average accuracy by more than three percentage points compared to just the weight on the standard demographic.
“It's a method of violence,” said Clifford Young, president of US public affairs at Ipsos. “That is, we don't really know what it is corrected for. Is it true only for that can't be ignored nonresponse? Or is it true to coverage bias? Or maybe a selector issue? Maybe all three.”
Even so, pollsters are generally optimistic. “What the evidence shows is that it puts us in a much better place on our polls than not using them,” said Mr Young, noting that he believes most pollsters will consider an earlier vote in 2024. “ I think the evidence so far suggests it does more good than harm.
He added, “If we use the same weighting and correcting method that we used in 2020 in 2024, we will miss the mark.”