Thursday, 13 December 2012

Election 2012: the crowd beat or tied the experts

Nate Silver joke with Dune reference.  Much was made after the 2012 election of the capacity of a few high end nerd Data Scientists like Nate Silver who were able to use the mass of information the web makes available in real time to get a far more accurate projection of the election than most traditional firms and almost all pundits.

The story is that the crowd is great as a source of data, but that there is so much data that an elite class of new scientists are need to analyse the data.  The approach is that Big Data coming from the mass of users is great, but to understand the data you need massive experts.

Key: But a closer look at the analysis game shows that the crowd is not only a good source of data, but also a good source of analysis.

Take for example the Iowa Election Markets.  The IEM is a project looking to harvest the wisdom of the crowd by allowing them to place small cash wagers on the outcome of major events.  For example before an election you can buy or sell shares in one of the candidates.  Initially for $1 dollar you get two shares: one that will pay a dollar is the democrat wins and one that will pay a dollar is the republican wins.  People can then trade their shares with each other establishing a price.  The idea is that the prices will determine the crowd's attributed likelihood of an event.

On the other hand a more subtle way is to pay on vote share. For example election sees one candidate getting 60% of the vote and another getting 40% the market will pay out 60 cents for the stock of one and 40% for the stock of another.  Users can then trade the stocks to indicate a crowd sourced idea of how the vote share will turn out.  People will try to guess the value of each share based on the likely outcome of the election.  If they can buy the shares cheap they could make money.

IEM prices for votes share for Obama and Romney
Remember the players of this game, which is played online and is rather nerdy, have access to the mass of information published in polls.  They also have access to the news, social media and other people to form an opinion.  The market is open to any adult player who can pay for shares online. All the users need to do is mail a request form in.

Opposed to this is Gallup polls, one of the nations most prestigious polling services. Gallup polling conducts telephone surveys.  Based on the outcome of this surveys its experts use models it has developed to analysis the data and make a project.  Their projects always differ from the outcomes of the polls themselves as they use past experience to project who is likely to vote, and how people who did not respond may vote.

So on one side you have a well established team of experts engaged in direct research, with well established tools of analysis and a long track record in Gallup.  In the IEM you got a lot of people gambling who need no more than $5 to start.

How did both sides do in projecting the outcome of 2012 election?  Truth is it is not even close.  Smart money would be on the punters of IEM over the experts at Gallup.  

What Happened

Election 2012 outcomes from Huffington Post

Looking at vote share the latest numbers I have seen give 50.9% to Obama and 47.3% to Romney.  

Gallup gets it wrong

On Dec 4th 2012 Gallup poll issued its last project of 'likely voters.'  According to the experts at Gallup poll Romney would edge out Obama by 1% point.  

Obama  Romney
 %  %
11/1-4/12 49 50

So difference between projection and reality was 1.9% for Obama and and 2.7% for Romney meaning that  Gallup was off by 4.6%.  

This is being kind, if you take a longer project of you see that this was the best project Gallup could give.  Very significantly Gallup failed to call the winner of the election.

Iowa Election Market Got it Better. 

Date     Contract     AvgPrice
11/01/2012     UDEM12_VS 0.505
11/01/2012     UREP12_VS 0.492
11/02/2012     UDEM12_VS 0.516
11/02/2012     UREP12_VS 0.478
11/03/2012     UDEM12_VS 0.528
11/03/2012     UREP12_VS 0.48
11/04/2012     UDEM12_VS 0.507
11/04/2012     UREP12_VS 0.489
11/05/2012     UDEM12_VS 0.509
11/05/2012     UREP12_VS 0.487

Its easier to understand this table

Obama Romney
0.505 0.492
0.516 0.478
0.528 0.48
0.507 0.489
0.509 0.487

Average Average
0.513 0.485

So the difference between projection and reality for the 5 days before the election of the punters of Iowa election market was about 1%.  And these punters who all had no more qualifications then being able to make a $5 deposit called the winner.

Nate Silver

It is anticipated that some people may question my concentration on Gallup.  No question that Gallup poll is an established polling agency, but it may be that its methodology is outdated an new projections like Nate Silver's using Big Data of many polls and statistics are far better. 

Nate Silver's project was excellent as well, but the difference between projection and reality of Nate Silver was 1.1%, actually a little bit higher than the error of the IEM which differed by .9982%.  Okay that is close enough to be the same, but Mr Silver did not do better than a group of people gaming on a web site all of whom had no more qualifications then being able to pay a $5 registration!


As for projecting the outcome of the 2012 presidential election it turned out that the Iowa Election Market, which relies on the concept of 'wisdom of the crowd', formed a far better projection in the days running up the election than the experts at Gallup polling.  One may argue that they had an extra 2 days  to refine their guess, but this did not improve the accuracy of their betting.  Rounding of the IEM projection was 51% for Obama 48.5% for Romney, a far closer guess than Gallup projection of 50% for Romney and 49% for Obama.

True that Nate Silver's projections were much better than Gallup's on popular vote share, but they were no better than IEM.  Also there is a difficulty of selecting which expert is correct at any given time.  Before the election Nate Silver seemed to be something of an outlier.  Media coverage of the election faced the problem of many experts, some like Gallup had a longer established record than the 34 year old Nate Silver.  Honestly it would be very difficult to decide which expert has a valid method and which one does not.  As for the crowd sourced guess they sum of guess of people who did not need to be experts to play came out if anything a little better than Nate's opinion.

It is understood that a growing confidence in Nate Silver may have impacted IEM punters.  It is also understood that without the data from polling services, along with other sources of data, the IEM punters would have little basis to make their calculations.  I am not arguing that polling and expert analysis is invalid or outdated by crowd sourcing, rather than for people who make it their business to guess outcomes the 'wisdom of the crowd' as applied to analysis should not be ignore. 

No comments:

Post a Comment