Thursday, 24 January 2013

Big Data, now there is no excuse for just opinions

One of the most amazing impacts of the rise of Big Data is that today there is no reason for us to just have opinions. There is enough data out there to be able to argue from a position of informed understanding, always and about just about everything that can be measured.

Take a typical example that happened to me on a recent holiday I had a bit of a row with someone over the relative state of the German vs. the UK economy. Though right now Germany is doing better I said that over the past 15 years the UK had tended to be ahead, and the person I was speaking to was utterly certain that Germany had been booming and leaving the UK in the dust since the start of the 21st Century. Our German supporter then went on to say it was the lower public debt in Germany that had underpinned this economic boom.

So who way right?

Well you all know that since this is my blog I was right.  I actually doubt the person I was debating with would recognise his argument.  But for the sake of argument grant me that two people are debating in a bar, one person says that Germany, because of low government debt, has out performed the UK economy since 2001.  The other person says that German debt is not that low and that Germany has not outperformed the UK until more recently.

Just 10 years ago you would need to consult an expert in economics to get the answer to this.  But who could find an expert in economics?  Ordinary questions of economics were highly dependent upon the chance reception of data on TV, and TV does not contain a lot of real data.

But today things are different.  Big Open Data, much more than just Big Data, has the potential to transform our ability to know, to judge and in time to decide and vote.

Looking a the Data available from Google it is clearly that over the past 15 years the UK economy grew faster and had lower unemployment for almost every year up to 2008. The current dominance of German economy is recent event with very shallow roots.


In fact anyone who rests confident that the German present economic advantage is indicative of a long term established trend would be wise to look at the Irish data from 2001 to 2008.  Until 2008 the Irish growth data was mostly stronger than the UK data.  At present there is really no more reason to assume that the German current domination will last generations then there was to assume that the Irish rise was a major shift.

And as for the idea that low German debt is causing a long term boom, again the data is not there.  The assertion that low German debt is driving a economic boom over the UK is wrong on so many levels.  Firstly German debt levels have not been going down between 2000 and 2010, they were tending to go up.  Second until 2008 German debt and UK debt as % of GDP were roughly the same.  But even more disturbing is looking at Ireland.  Certainly during the Irish boom her debt was lower than both the UK and Germany for almost all years between 2000 and 2008, but then once her economy goes down her debt situation becomes very bad.

In fact the evidence of data is that low debt is a effect of economic growth but clearly not a cause.  Nations that prospered best the year before are less likely to have higher debt, but if their economies become weak then the debt rises.

The problem is that having this debate at a table the idea of bringing up some tables on a iPhone would not have easily solved the problem.

Looking at Big Data is is clear that I was right and this other person was wrong. The problem is right now is that though the data is out their their is not yet a very easy to use UI where you can easily ask Google for these kind of Big Data questions.  And there is not presently the tools that work well in community allowing people to convince others with the Data.  When was the last time you saw a Big Open Data presentation on Facebook?  (if you are like me it was about 15 minutes ago but I assume my social graph is very odd)

There is a real problem that all of this will be crowded out just as TV's educational opportunity will be crowded out.  I personally have had my views of the world transformed by Big Data over the past 5 years.  I have found myself becoming much more left of where I was say in 1999.  Big Data has changed my view of the world.  But I have long ago relieved I am different.  TV also changed my life as a child.  As a teen I watched Carl Sagan, Masterpiece theatre, and Shakespeare as a teen growing up in suburban Illinois where little of that was taught in schools.  TV help fill a gap in my intellectual development as I struggled with a learning disability that made it hard for me to learn to read.

But form most of my friends at school TV was watching MTV and smoking joints all day long.  For the most part TV stunted the intellectual development of most teens, while it enriched a very few.  This seems to be the trend of modern media: it extends inequality even when it offers options to promote equality.

Probably a few people are going to read Nate Silver's book at use Big Data to change their work, change their understanding of the world and change their political views.  But for most people the endless access to the Internet may be just another source of mind numbing entertainment.

As always the great question of media is how to engage a large quantity of people with Big Data.  Perhaps games that use Big Data, or including information  from Big Data more in news and even entertainment.  My big fear is that Big Data will come but Big Open Data will not be used by many, that data science will remain a black art making some richer and most no better off or even poorer.  From the data above you can see this need not be, but it might be.

No comments:

Post a Comment