At Survation we’ve had our fair share of mistakes but also consistently accurate results among the range of challenging-to-get-right services we offer – from constituency polls, to regional and national polling and modelling – our Multilevel Regression and Post-stratification (MRP ) model for clients in GE 2017 also indicated “no overall majority” – not contradicting our standard polling work. We tweeted some of this MRP work overnight publicly on twitter and we will offer our first MRP work to peer review. Details at the bottom of this post.

Given 5 years of measurably accurate election work in Scotland, the UK and more recently new markets like Italy it’s important I feel to show our record and make clear the “official” 2017 scorecard as per the British Polling Council.

Survation were the most accurate polling company in 2017 with polling error of less than 1%

When Theresa May called the election, polling figures from some firms showed huge Conservative lead (up to 25 points) over Labour that her party could have 200 seats, thumping majority. As we now know, even the seemingly best laid plans..

Survation have three ‘final’ polls for this election: online for the Mail on Sunday June 4th, showing the Conservatives with a one-point lead over Labour; a telephone poll for Good Morning Britain, June 4th (same result) and an official “final poll” June 6th, corroborating the trend seen in the others. For the June 6th poll we interviewed 1000+ in Scotland, 500 in Wales and 1000+ in England.

Focusing additionals calls in “marginal” seats in the South West, North & Scotland, combining our three ‘final’ polls, we interviewed over 5000 people in the final week. All of research pointed at a steep narrowing of the Conservative lead.

As is usual in this industry, when pollsters produces “outlier” polls (ones that shows a very different implied forecasts than the average) our figures were mocked by political scientists, veteran election experts, Labour and Conservative MPs alike. One former Conservative party candidate turned national radio presenter tweeted that Survation were clearly “Muppets” – he had his own detailed forecast of a Tory landside). I was also greeted with abject mockery by the guests and presenter BBC Daily Politics who, to their credit, invited me back after the election.

Survation have been polling the UK and Scotland for over six years. Using a unique telephone method and constantly improving our online sampling, we…

• Were the most accurate polling company in the Scottish Independence Referendum
• Produced a 52% Leave figure for the EU Referendum a week prior to polling day, as well as 52% leave figure for clients on the day itself
• Conducted a telephone poll the night before General Election 2015 which correctly caught the Conservative lead over Labour
• Produced highly accurate final polling for the Holyrood 2016 elections
• Were the most accurate pollster by far in the 2017 General Election (see table below based on the official British Polling Council definitions of final polls)

How do we do it?

Simply put, in online and telephone polling, we “get it right” by ensuring our samples are representative of the population. Failures in this area were what caused the industry to miscalculate so badly in 2015, though most pollsters have improved significantly on this since.

However, the industry in 2017 significantly overestimated the Conservative lead over Labour, based on a series of untested assumptions about which types of voters would turn out to vote. Assumptions like this had the combined effect of suppressing the true Labour voting intention.

An in a “snap” election, pollsters were caught out, lacking real evidence to make a diffferent judgement about their turnout modelling.

Before the election Patrick Sturgis, the respected Professor of Research Methodology and Director of the ESRC National Centre for Research Methods, implied that these assumptions were the “Achilles Heel” of the industry. A pollster could survey a perfectly representative sample, but still double the Conservative lead over Labour using this method.

At Survation, we treat each election as a unique event; we do not allow the results of previous elections to skew our results by over-correcting for past errors. Instead, we are very careful to ensure we target a representative sample of voters, and try to avoid making assumptions beyond what our respondents tell us. We have never made this a secret.

Patrick Sturgis chaired an enquiry into the polling debacle of 2015. He recommended that pollsters improve their results by focusing on finding a representative sample. This election has once again validated Survation’s forward looking approach to our sampling, and we will continue to focus on improving our methodology in line with his conclusions; we will not try to fix historical errors by changing our methods in untested ways.

Finally, a note from Dr. Chris Hanretty, Reader in Politics at UEA

A few weeks before the general election, Survation asked me to work with them on estimating how well each party would do in each constituency and to provide updates over the course of election night, some of which were alluded to on Survation’s twitter:

Keep the champagne on ice Labour watchers. Team still working away. We would expect the 2am deluge to point clearly us to the final outcome.

— Damian Lyons Lowe (@DamianSurvation) June 9, 2017

Producing seat level estimates is hard, and it can’t be done purely using traditional polling methods. Survation was one of the few companies to do constituency-specific polling in this election cycle, and published results from Edinburgh South, Battersea and Kensington and Chelsea. However, no one has the money to pay for polls in all 632 mainland constituencies. The Conservative Party alone have the funds to regularly poll their marginal and target seats.

To produce these seat level estimates, I used a technique called multilevel regression and poststratification (MRP). MRP is a way of producing estimates of opinion for small areas. It works by combining information from large national samples (in this case, upwards of 12,000 respondents) with information from the census. The responses given by respondents are modelled on the basis of their demographic characteristics and what we know about their constituency (its past voting history, how it voted in the EU referendum, and so on). This is the “multilevel regression” part.

In the subsequent post-stratification stage, we look up census data to see how many people of each demographic type live in each constituency, and make a prediction as to how many of those will vote for each party. In this way, the estimates, although they are derived from a national sample, end up being representative of the demographic make-up of each constituency.

You might be skeptical about the added value of MRP. You might think that if you get the headline national figure right in the way that Survation did, then working out seats is easy: just plug the numbers into a seat calculator.

This is wrong (the MRP estimates were more accurate than the numbers you would have got from applying a uniform national swing), and it misunderstands the potential of MRP. MRP allows opinion to be estimated on any national issue, not just vote share. If this government – or any other future government – decides to have another referendum, particularly one on a new issue, then MRP will be waiting.

Our work has many applications beyond elections and politics, opinion locally, nationally or internationally and the methods are transferable to a wide range of applications. So if we can help you or your organisation, large or small, get in touch by emailing me directly

2017 thanks…
In no particular order, we’d like to thank David Cowling, John Curtice, The staff at the Electoral Commission, David Clegg at Daily Record, Simon Walters and Glen Owen at Mail on Sunday, Carol Prest and Andrew Picken at Sunday Post, Terry O’Sullivan and Mark Kiff at Good Morning Britain and also within the media Politicalbetting’s Mike Smithson and Sky’s Faisal Islam who trusted / reassured us on our methodology despite the “outlier” results it generated.

Outside of Survation’s core team and Chris Hanretty, we would also like to thank Zsolt Kiss for his rapid assistance modelling on polling night.

Best Wishes,

Damian Lyons Lowe.
Founder & Chief Executive. Survation.