The Ontarian election, the death of the Likely voters models?

Coup d'oeil sur cet article

A little bit over a year ago, the polling industry in this country was coming off two huge failures in Alberta and BC. While some of them went for the cop-out excuse of saying that people changed their mind, many actually decided to improve their methodology. The BC election in particular seemed to indicate that with low turnout, you couldn't simply take the poll numbers at face value. We needed what we call "likely voters" models, similar to what is done in the States. Since some people are more likely and more committed to vote, it made sense to account for this. Think of it of weighting more the people more likely to vote (older, more educated, more motivated, etc)

And then we got the Ontario election. While most pollsters agreed that PC voters were more committed (and therefore the likely voters adjustments would help the Tories), we got the exact opposite. The Liberals performed mostly in line with expectations while the Conservatives barely received 31% of the vote. Much lower than the 35-36% from the polls among likely voters.

The table below shows all the polls released during the last week. For firms with a LV model (Ekos, Abacus, Ipsos and Angus-Reid), I included both sets of numbers. For the calculations of the margins of error, I kept the same sample sizes even though technically, the likely voters poll is less (but not all pollsters are transparent as to exactly how many respondents they lose while moving from decided to likely voters). I use two measures to evaluate the polls. The first one is the average absolute deviation. So if the PC was in 36% and got 31%, it's a difference of 5 points. This measure is very common but potentially depends on luck. We shouldn't necessarily expect pollsters to nail all four numbers, but they should have them within their margins of error. This is why our second measure counts how many parties were within the MoE for each poll.


Firms Method Likely Voters? Sample size LIB PC NDP Green Average absolute deviation # Party within MoE
Oracle Phone No 1000 35 36 24 5 2.2 2
Angus-Reid Online Yes 1866 34 36 24 5 2.4 2
No 1866 36 32 26 5 1.4 2
Ipsos Online Yes 1991 30 36 30 4 5.1 1
No 1991 33 31 30 5 3.1 2
Abacus Online Yes 1882 36 36 23 5 2.1 2
No 1882 35 32 26 6 1.9 1
Leger Online No 1052 37 37 20 5 2.8 2
Ekos IVR Yes 1311 42 36 17 3 4.3 0
No 1311 37 31 19 8 2.3 2
Forum IVR No 1054 41 35 20 3 2.9 1
Too Close To Call (adjusted average) 37.3 36.0 22.1 3.7 2.2
Actual results 38.64 31.25 23.76 4.84
As you can see, all the firms with LV did (much) worse while using the adjusted numbers. In other words, pollsters would have done better just using their normal numbers. In my final projections yesterday, I said that I wasn't sure why we needed all these models. Sure Alberta and BC were massive faillure, but the polls had performed relatively weel in other elections (among them, the recent Quebec one or the 2011 Ontario). At the end, I used the likely voters numbers (mixed with the normal numbers from the other firms) and I obviously shouldn't have. It was a little bit like playing the lottery and I lost.  Ultimately though, I'd still have overestimated the Conservatives. So I don't feel too bad. And the pollsters who had predicted a Liberal majority (Forum, Ekos) were right but with numbers that were very wrong.

Overall, no pollster did really well. In particular, nobody has more than 2 parties within its margins of error. It was like the ones who got the Liberals right then got another one completely wrong.

The best pollster was Angus-Reid, but not their LV numbers. Then we have Abacus (non-LV as well, althought the LV numbers weren't too bad). I said that to me, Abacus has the best LV model and I was right (although here, it's more a question of being less wrong). Ekos has the merit of being the only one to have the main two parties within its MOE (for its non-LV numbers) but its LV numbers missed all four (at least the LV model had that the Liberals wouldn't go down). Worse one is Ipsos (LV) which means this firm has finished last in two consecutive elections (they were last in Quebec). As for my own average, beside the fact that I shouldn't have used the LV numbers, I adjusted the Green down too much. Probably because since the Green were already lower because of the use of LV models, I didn't need to bring them down even more. I'll remember that.

I'll take a look at my projections in the coming weeks. What I know is that if you had entered the correct percentages in the simulator, you'd had got the results almost perfectly (56-30-21 to be exact). But pollsters will have to provide answer to some questions because this election wasn't particularly good for them and their likely voters models.

1 commentaire(s)

Thierry Soucie dit :
13 juin 2014 à 7 h 09 min

Mon simulateur, avec les résultats de l'élection, m'aurait donné:

57 OLP 31 PC 19 NDP

Ce qui n'est pas si mal. J'ai découvert que certaines des erreurs sont dues à des circonscription ayant eu des élections partielles depuis la dernière élection générale et que je n'avais pas inclus ces chiffres. Je devrai donc corriger cette situation et commencer à inclure ces résultats.