Following the election Find Out Now has conducted an internal investigation into the likely cause of election polling error (in addition to the investigation conducted by Electoral Calculus based on polling from our panel). In summary, Find Out Now voting intention polling performed well relative to other pollsters. Although we didn’t over-estimate Labour, we had too few Conservative voters and too many Reform voters in our surveys. Given we didn’t experience this in 2019, we have sought to find the cause of this problem, with far more theories being disproved rather than solutions found. However, one potential cause could be the age structure of the 2024 Conservative vote.

Find Out Now is a single-source panel, with 100,000+ respondents daily answering survey questions. Most respondents have been on our panel for years (two thirds of active panelists since prior to the 2019 General Election) and we hold data for all surveys a respondent has previously done, giving us a strong basis to investigate respondent behaviour. This gives us the ability to both see potential trends specific to our panel, and identify issues which are potentially affecting all online polling in the UK.

We didn’t conduct a proprietary public voting intention series prior to the General Election. However, on 3rd July (day before the election) we ran a N=2000 nationally representative voting intention survey, to benchmark our performance for the future. Final results are below:

Understandably we can claim no credit for positive results when we didn’t publish them at the time. However, following analysis conducted by Will Jennings it is encouraging that when compared to other final voting intentions, this would have been equivalent in absolute error (11.3%) to the third most accurate result out of 18.

It is notable that this voting intention poll had almost no Labour bias – the final figure of 35.2% being the lowest of any pollster and only 0.5 points higher than Labour’s final GB vote share of 34.6%. However, the Conservatives were significantly underestimated (3.9 points – at 20.4% instead of 24.4%) and Reform UK overestimated (3.3 points – 18.0% instead of 14.7%).

These results require explanation, especially given VI results from our panel in 2019 had no such issue. For comparison, our 2019 VI would have had an absolute error of only 6.1 points (better than the best VI from this election cycle).

Our voting intention methodology

Polling error can be caused either by survey methodology (such as decisions about question wording or weighting), the raw data, or a combination of both of these. Given Find Out Now is primarily a panel, we’ve seen how different methodologies can achieve very different results despite using the same sample – the methodology we chose for our own voting intention survey influenced by these findings.

We used a simple structure which asked if someone would vote, only included those who said they would, and prompted ‘Don’t know’ respondents (with no option for ‘Prefer not to say’) again, including them in the headline voting intention figure if they then gave a valid vote choice. This was then filtered – note different to ordinary rake weighting – to nationally representative proportions for age, gender, region and 2019 General Election vote. 

We don’t see any obvious blindspots in the methodology of this survey:

• We didn’t include a ‘Prefer not to say’ option to avoid respondents – such as “Shy Tories” or those who don’t want to admit to not voting – selecting this instead of their true vote choice, given this would depress forecast Conservative support and inflate turnout. Perhaps because of this, we had no differential completion rate for the survey by 2019 GE vote
• For similar reasons, we didn’t include in the final calculation respondents who said ‘Don’t know’ on their vote choice even after the squeeze question, as often we find these respondents are unlikely to actually vote 
• We were able to achieve a 2000+ nationally representative sample collection entirely on 3rd July, making distortions caused by late swing unlikely
• Our filtering methodology should provide a higher quality sample than surveys of equal size that deploy rake weighting (i.e. as every quota group was perfectly filled and no upweighting was required)
• There should be no issues with “false recall” as we don’t reask people’s 2019 GE vote but instead rely on their answer given previously, in most cases from years prior
• There’s a balance to be struck on which and how many weighting variables to use – adding more can address potential weak points of the sampling, but also increases the chance of distortion caused by researcher bias. In this case, we found adding additional weighting variables (such as socio-economic group, ethnicity or education) to the survey made the final results less rather than more accurate, with our original light-weight model working best

This suggests that the issues we faced – of too few Conservative respondents, and too many Reform – do not derive from the survey methodology used.

Reallocating ‘Don’t know’ respondents

One potential change which did make a significant difference was to reallocate all ‘Don’t know’ respondents who said they would vote, back to their 2019 party. This eliminated almost all error and would have given the final vote share an absolute error of only 5.9 points – i.e. the most accurate of all final election VIs, and a significant margin better than the current best (Verian on 7.3 points). 

However, we would be hesitant to use this methodology going forward. When recontacted, those respondents who said ‘don’t know’ the day before the election gave a wide variety of reported vote choices, e.g. less than half of the 2019 Conservatives in that group reporting that they then voted Conservative on 4th July. So this is likely a case of two errors canceling out, and would be unlikely to work or even compound issues in the future – especially given the low base of 2024 Conservatives.

Conservatives disengaging from polling

Given the error seemingly derives from the sample rather than methodology, our leading hypothesis was that a significant number of more committed Conservative voters (those who stuck with the party in 2024) have disengaged from taking part in political polling. This means that despite the existence of quota groups for 2019 Conservative voters, these would be filled with less committed Conservatives, who in turn are more likely to switch to another party. I.e. a new version of the “Shy Tory” effect.

This hypothesis worked well on theoretical grounds. It is a unified reason to both explain the understated Conservative vote share and the overstated scores seen for Labour and Reform. It also makes sense as an explanation for why the polling error was relatively large, and why the industry experienced this error in 2024 but not in 2019. This is also something which is only likely identifiable on a panel level – given it involves working with ‘missing’ respondents rather than a specific final dataset.

However, although this was our leading theory and we ran lots of tests on it, over that process we comprehensively disproved it:

We found no evidence that Conservative voters were quitting the panel at greater rates. The above graph (taking all pre-Partygate active users on our platform) found no difference in activity levels between Labour and Conservative respondents. If anything, Labour voters actually had a slightly higher quit rate. Similarly when taking groups that were demographically identical but differed only in their Con/Lab support, there was no difference in quitting behaviour between them.

An alternative pathway was that while existing respondents may not be quitting at differential rates, perhaps new Conservative respondents were not joining the panel at the same rate as Labour respondents. Given one third of our active panelists had started doing surveys with us since after the 2019 GE election, an imbalance in their political orientation would be large enough to have downstream effects. However, although indications were that our panel had become more Labour-leaning over time, once controlling for age this relationship completely disappeared. 

As such, there was no obvious panel-based explanation for the problem of too few Conservative voters (and too many Reform and Green).

Conservative age distribution

One speculative answer relates to the age distribution of the 2024 Conservative vote. Our voting intention surveys have a very low number of Conservatives in younger age groups. 

	2019 Con (British Election Study)	2024 Con (from our 3rd July voting intention)
18-24	28.4%	6.9%
25-34	25.0%	4.9%
35-44	32.3%	5.1%
45-54	45,4%	12.4%
55-64	52.4%	16.7%
65-74	60.0%	33.7%
75+	59.0%	50.0%

It is a traditional weakness of polling that younger people (and among them particularly men and the less politically engaged) are less likely to do surveys. With only the most engaged young people doing surveys, this naturally distorts the results, and often depresses the voteshare for the Conservatives. For example, a YouGov survey in September 2023 made headlines when they stated only 1% of 18-24s planned to vote Conservative – such a drastic fall likely driven more by weaknesses of the polling rather than a genuine shift among that group. Similarly, evidence from polling young people in the United States (such as here or here) note the specific difficulties in gaining accurate samples of them.

From this a speculative argument would be that the extreme age distribution of 2024 Conservative support, paired with traditional polling weaknesses on younger less engaged right-wing respondents, has combined to underestimate overall Conservative support. This may not have been a problem in the 2019 General Election, as even if there was a clear age skew to the results, according to the 2019 British Election Study (seen as the gold standard, using random probability sampling) the Conservatives still gained 28.7% of the vote in 2019 among under-45s. So because the Conservative age distribution was far less extreme, polling weaknesses on younger right-wing respondents were not exposed.

If you assumed the drop in Conservative voteshare among under-45s was consistent with their drop in voteshare across the electorate as a whole, they should have been on 14.8% with that age group. However, in our 3rd July voting intention polling this was 5.4%, and of other pollsters Verian had them on 9.9%, Ipsos on 7.5%, or Savanta on 9.9%. Perhaps their vote share did drop significantly more (from already low levels) among younger voters compared to the average, which the polling then accurately reflected. Or alternatively, it could be that their vote share among younger voters passed a threshold where pollster weighting or modeling was less able to hide their weaknesses among that age group – exaggerating the extent of the decline. 

After the 2024 General Election, the British Polling Council invited its member organizations to share research into the performance of polls at the election and the cause of polling error – particularly relating to the over-statement of the Labour vote intention share . Articles from other BPC members can be found here.