Tuesday, 29 January 2013

Big Data and beauty: Hong Kong

Social Media map of Hong Kong Island, showing a lot of popular Flickr sites and Wikipedia attesting to the cultural significance and beauty of Hong Kong Island.
The Spirit of Cities by tinorose

I have been reading The Spirit of Cities: Why the Identity of a City Matters in a Global Age Daniel A. Bell & Avner de-Shalit; which is an amazing book full of ideas about what makes cities operate. Bell and de-Shalit avoid much dull analysis of statistics rather presenting stories, history and insight that bring the cities alive for the readers.

Image Flickr 
One of the things I keep thinking is if Social Media mapping can confirm some of the ideas in this book.  Using my Happan.in tool I think I have confirmed their observations on Tel Aviv and Jerusalem.
Image from Flickr
In their chapter on Hong Kong Daniel Bell makes what seems like a utterly subject assessment about Hong Kong, claiming that for "urbanites... the best view in the world is Hong Kong's skyline."

An interesting observation, and one that in the recent passed could only be taken as a matter of opinion.  But one of the implications of Open Big Data, especially data that can be mapped from social networks like Wikipedia, Flickr, Twitter and Foursquare, is that you can actually drill in to the opinions of people about places in a more scientific and objective fashion.  You can get counts, and more importantly relatives counts of the words people use, you can see the images, count the checkins, count the Wikipedia edits and see the tweets that are coming form a place or are about a place.

Image Flickr
So can I have a subject assertion like 'people who like cities love Hong Kong's skyline' and make it in to something I can test with social media Big Data?

Yes I think I can. I am not going to try and define 'urbanties', and just assume that urbanites are human beings many of who author web content or post photos to the Internet.  Also I am not going to try define 'the world's best view' and rather take the statement to mean:
The view of Hong Kong island from Kowloon is held to beautiful and significant by a large community of people.
Further operationalising this statement into something that can be measured:
  • The areas of Hong Kong Island seen from Kowloon will have a lot written about it on the web.
  • Images of this part of Hong Kong will be posted on the Internet.
Converting these to Social Media assertions, that can be tested:
  • There will be a large number of Wikipedia entires with a lot of edits about Hong Kong island around Central where there is a large skyline.  That is people are motivated to write about it.
  • There will be a lot of images posted on Flickr and many of these will be popular with users. That is people are motivate to post images about it.
Using my happan.in tool I can actually test both of these assertions.  Happan.in maps social media activity on a Google map, including geo-tagged information from Wikipedia and Flickr.  As you can see from this image there are in fact a lot of Flickr and Wikipedia images linked to this site.

Happan.in does not just map all Wikipedia article or Flickr image with a geo-location, it only post articles that have had a significant number of edits in Wikipedia or articles that have a significant number of views in Flickr.  Therefore it is not just a map of content, it is a map of popular and significant content.  

Image from Flickr
Using this mapping we can see that what Daniel Bell writes, minus some of the hyperbole, is accurate.  The view obviously is popular with a lot of people.  People are motivated to write and edit Wikipedia about the location and people post images which are often popular on Flickr about the area.

Okay granted that this exercise is a bit academic, I doubt many people will question that city lovers like the view of Hong Kong, and the implication that it can be confirmed with Big Data seems almost pointless.  But imagine other assessment that can potentially be tested with Big Data like this.  Some statements about places that might someday be measured by sociologists using social media are far from purely academic:
  • The best place to get a good education is in the city X.
  • People who live in region Y are at a major disadvantage in finding jobs.
  • You don't see any women on the streets at night in Z.
  • This city is getting run down despite the official economic reports.
  • The government is not that popular in this north of the city, more popular in the south.
Big Open Social Data provides the opportunity to test more common sense assertions about spaces that normally would have fallen outside of the strict measures of social geography.  They also provide potential sources of sociological data without the need for large budgets or official sanction.

Certainly as of today this work tells you more about the popularity of smart phones in an area than anything else, but as more and more people post more and more of their life narratives online to share with the global community the reach and agility of sociology is about to explode. 

Twitter analysis tracking #LondonBridge

UPDATE: My prediction that is was likely a false alarm seems to be confirmed. To sum up based on my Big Data analysis of social media in the area I concluded it looked like nothing major had happened other than the emergency response. If this is true I will have used analysis of Twitter to determine what was happening in real time.
Official and crowd reports are coming in of a incident at London Bridge train station which has required a fairly significant emergency response.

I am tracking the response on Twitter

The station is seeing something elevated tweeting, but nothing extreme yet. The incident started at 10 am via reports from both the crowd and confirmed official reports.

I am confident that what ever it is it is not involve an explosion or shots fired. Why? Because looking at the twitter stream coming from the area around the station there are only about 10% of tweets mentioning the incident. This indicates that no load noses or smoke has been seen. Also so far all tweets just concern the presence of emergency services, I have yet to see any solid data about what is going on inside.

For example tracking the recent fire in Brazil, very quickly the local twitter stream was dominated by chatter about the fire. What is striking about London Bridge is how normal tweets are that do not mention the presence of police, ambulance and fire service. Rumours on twitter presently talk about possible gas leak, gas leaks often turn out to be nothing.

So the lack of much twitter activity about what seems like a major emergency response while normal twitter traffic continues opens a interesting possibility:  this is a false alarm, and twitter activity (the lack of public alarm or reports of anything) indicated it was a false alarm.  So I will be following closely to see if twitter analysis determined the status of this emergency. This opens the potential not only for news tracking but for intelligence on emergency response, tracking the public mood and discussions in real time to see what it could be telling us about an emergency.

You can track tweets from near London Bridge with this tool

Monday, 28 January 2013

Further study of London retail traffic

Oxford Street twitter levels per hour, 250 KM around Oxford Street London Dec 19 2012 to Dec 25 2012
The same measure from Jan 21 2013 to Jan 27 2013

Theses two graphs above show the basis for my earlier prediction that UK retail sales would be low for Christmas.  Looking at the area around Oxford Street London, a major shopping centre, in the week before Christmas compared to a week in late January 2013 you notice no surge in the pre-Christmas traffic.  This work is still very new but I am becoming confident that larger crowds in a city like London mean more tweeting, and the run up to Christmas in London retail did not see a surge in tweeting.

Also telling is the Foursquare checkins for a popular shop Like Liberty near Oxford Street.  Though Liberty did see a surge on the 21st of December 2012, on every other day the foursquare checkins were on average lower in the week up to Christmas vs the week after Christmas.  

Foursquare Checkins for Liberty London from December 19 2012 until the end of the year.

A living vs a tourist location

Following up on my reading of the wonderful book  The Spirit of Cities by Daniel A. Bell and Avner de-Shalit, I have been thinking about how we can describe cities or places using social media data.  I am not just looking for the kind of dry descriptions of average income or crime rates or population density, what I am wondering is can we know what a place is like by looking at social media, can be class areas. I have been using my Happan.in tool to map major cities to see what social media can tell us about them.

In a previous post I looked at Bell and de-Shalif's description of the different 'spirits' of Tel Aviv and Jerusalem.  I found that there was something objective reflected in social media about what they said.  Tel Aviv is a place with far less cultural and historic significance, as judged by Wikipedia, than Jerusalem but far more commercial and social activity, as judged by Foursquare.

Social Media map of Jakarta, showing a flood of tweets in a vibrant active social media community
As opposed to Saint Mark's in Venice, where Wikipedia and Flickr images dominate, and foursquare sites are mostly for tourists.  A lovely place where people from all over the world come to visit, but not a vibrant living community in the square which has become essentially a museum.
Bell and de-Shalif go on to examine a well known danger for cities that become more historic than they are modern: decline of sustainable viable community.  This reminds me of a trip to Italy I took with my wife.  We visited, among other Naples and Venice.  Both cities are rich in artistic and historic value, but my wife complained that Venice seemed more like a museum while Naples was a vibrant city.  

Looking at where I live, London, I can see that different uses of land are reflected clearly in Social Media.  For example take Westminster, a place people go to see, vs Soho, a place people go to have fun.  Westminster social media map looks a lot like the Vatican's actually.  People do important things in both places, but very few people and not the kind of people who are going to use Foursquare to checkin to the office, one hopes.  

Social Media Map Westminster London, showing popular (not necessarily trending) Yelp, Foursquare venues, significant Wikipedia entries (those with several edits) and popular Flickr posts.
Social Media map the Vatican

Now below look at the map of Soho in London.  No doubt that Soho in London is of massive cultural significance, and we see a lot of Wikipedia entires associated with locations in Soho.  But look at that hive of popular Foursquare venues.  This swarm of venues shows a vibrant social life.  And that is precisely what you see in Soho, a historic area that hosts a vibrant living community.  I have long seen Soho London as one of my ideal city areas.

An extreme example of what I am talking about is the Pyramids in Gaza.

The Pyramids Egypt
The Wikipedia articles that cluster around a few building attest to the massive historical significance of each building.  But the 2 popular Foursquare venues tell us how this area is used socially.  People enter, look at the Pyramids, checking for the big Pyramid and checkin at the Sphinx and let their friends know they have been there.  Certainly, as anyone who has been there can tell you, the area of Giza is full of shop and places to eat but none of them deserve a Foursquare venue.  The Pyramids are all history and now modernity.  

Sunday, 27 January 2013

Social Media and high culture in Italy

I have create two social media maps using my Happan.in tool

The Vatican, a major cluster for geo-tagged Wikipedia entries.  Clearly a number of the worshippers and tourists are checking in with Foursquare.  

Saint Marco's in Venice, very much like Vatican but also a huge cluster of Flickr images.  So this is not only a popular tourist attraction, but one clearly judged highly for its visual beauty and not just its meaning.

Conclusions: First review shows that the Vatican and Saint Marks in Venice are similar.  There is a high density of Wikipedia entires to Foursquare, indicating a popular tourist attraction of great historic, cultural and religious value.  Saint Marks is also distinguished by a high number of popular Flickr images associated with the location.  This indicates Saint Marks added value as an artistic treasure.

To conclude, we can guess that a lot of people go to the Vatican for religious purposes as well as tourism, but that the square around Saint Marks is not seen as a religious site as much as a work of art which many higher end photographers share with the premium Flickr service.

City character: Jerusalem and Tel Aviv

I have been reading the excellent book The Spirit of Cities by Daniel A. Bell and Avner de-Shalit.  In it they write a portrait of Jerusalem.  They compare Jerusalem to Tel Aviv over and over again.  Essentially their observation is that Jerusalem is a city of religion and spirituality, of history and learning while Tel Aviv is a city of commerce, secularism, modernity and entertainment.

Having been to both cities this seems correct to me, but I wondered if there was some way to test this assertion using Social Media Big Data.  In an age of Big Data it may be possible to confirm of refute such observations which in the past rested on a subjective assessment.  Could I produce a definition of religious or commercial that I could use to test  what Bell and de-Shalits say?

Operationalising what they say I make these conclusions:
  1. Religion is an activity linked to culture and history,
  2. Fun and business are community actives linked to the commercial life of a city.
So if it is true that Jerusalem is a religious city concerned more with spirituality and Tel Aviv is a city of money, fun and modernity you would expect to see higher levels of commerce and more sites of trade inside Tel Aviv, and you would expect to see more culture and history associated with Jerusalem.

But how to test this.

I make an fairly obvious hypothesis in analysing social media data that Wikipedia content associated with a location is generally about culture and history.  Foursquare venue on the other hand measures commercial activity and business venues. Certainly major culture sites can be Foursquare venues, and major businesses will be in Wikipedia, but the authors of Wikipedia tend to concentrate on history and culture while Foursquare user like to checkin to bars, pubs, theatres, restaurants, etc.

So I can convert by two assumptions in to testable hypotheses:
  1. There should be more Wikipedia sites associated with the same area in central Jerusalem, in the Old City, then anywhere in Tel Aviv. That is Jerusalem is more culturally significant than Tel Aviv to world history and religion.
  2. There should be more Foursqaure activity in Tel Aviv city centre than anywhere in Jerusalem.
Using my Happan.in app I have been able to confirm these.  See the pictures below:

The Old City of Jerusalem, we see a heavy cluster of Wikipedia articles associated with sites in the Old City and the area around it in Jerusalem, reflecting the cities history and cultural significance.  
The same sized area in Tel Aviv has a much smaller scattering of Wikipedia articles associated with it.

Testing commercial density of the same areas, we see Tel Aviv is full of popular Foursquare revenues that line the main streets.
In Jerusalem on the other hand there are far fewer active Foursquare venues.
Conclusions: It is understood that Foursquare is relatively new and Wikipedia is in constant growth.  It may be that much of the difference observed could do to the relative poverty of Jerusalem (meaning less people with smartphones and data plans) and the international meaning of Jerusalem over Tel Aviv.  The distribution of Wikipedia and Foursquare in the two cities does not necessary imply that Jerusalem is a sacred city and Tel Aviv is a commercial centre.  If you read the Wikipedia entries and look at the foursquare venues this becomes more obvious, but is a complex task for machine learning to teach a computer to understand this.

But, the distributions of Wikipedia articles and Foursquare entries is what you would expect if Bell and de-Shalit's observations about the cities was true.  Reading the entries, I argue, would convince anyone that this 'spirit' of the two cities is clearly expressed in the social media content associated with both places on the web.

Given Popper's brilliant analysis of science, that scientific theories can not prove they can only fail to disprove, we can say that social media gives a level of scientific validity to the ideas of Bell and de-Shalit.

By the way excellent book, I am enjoying it so much I look forward to creating more tests of their conclusions.

Saturday, 26 January 2013

Twitter case study: London and New York at the end of 2012

New York levels of tweeting and re-tweeting 1 KM from center of New York, from December 20 to end of year.

Comparing tweeting levels in London vs New York for the end of 2012.  The blue boxes indicated Christmas.  Looking at the levels we observe the following:
  • More tweets come from New York, likely because of a higher density of population.
  • Center of New York tweeting did not go lower during Christmas as it did in London.
  • The last shopping days in London saw spikes in tweeter activity in London whereas New York saw high level of tweeting throughout the end of 2012.
London levels of tweeting and re-tweeting 1 KM from center of London, from December 20 to end of year.

Conclusion: Central New York is more use involved in the Christmas Holiday, with large groups of people going in to New York for Christmas while London saw active shopping before Christmas, but saw less activity on the Holiday itself and boxing day.  The low levels of tweeting on Boxing Day, a traditional sales day in London, could indicate weak consumer demand.

Tweeting levels and variation case study: Amsterdam and Athens

As part of my work I have been tracking levels of tweeting at major cities around the world.  In this case study I look at the first two on my list, Amsterdam and Athens, to see how very different cities are using tweeting very differently.  

Almost three days tweeting variation in the centre of Amsterdam, with 0 being midnight Jan 25 2012.  Scale is Log 3 to make variations easy to read.  The thin black line is a 10 day running average.
 In Amsterdam city centre tweeting is serious business.  For 1 KM radius around the center of Amsterdam you will almost never get under 200 get tagged tweets an hour.  There seems to be some variation with tweeting lowest around 11 ask and highest around later afternoon to evening.  Looking at the numbers it seems that tweeps in Amsterdam stays high from after lunch though to the early hours of the day.

Several days of tweets from Amsterdam, each vertical line reflects a 24 hour period.  Lines are drawn at 3 PM local time.

As the above graph clearly shows there is a 24 cycle in tweets coming from Amsterdam, with low points in the late afternoon repeating almost every 24 hours.  Take a look at this graph and try to find a low point that does not come right before the 24 line, there are very few. So though tweeting is high in Amsterdam is does show a 24 cycle, which is not surprising at all. What is a bit surprising is that the low points are late afternoon and not evening.

Same time period in Athens.

Athens city center has much lower tweeting.  Athens has far less usage of mobile tweeting than Amsterdam with the average tweets in a 1KM radius around the city center moving from low of about 20 to a high of 200.  We see a more pronounced pattern in the tweets.  Tweeting in Athens is an evening activity peaking around midnight and lowering during the day.  First speculation is that Greeks are using mobile tweets as a social platform, to organize night out.

Conclusions: Clearly there are some significant cultural differences in how tweeting is produced.  The differences between Amsterdam and Athens can not be accounted for in size, Athens is much bigger, or wealth really.  It is true that Athens is a poorer city than Amsterdam, but we see very high geo-tagged tweeting in Jakarta which is significantly poorer than Athens.

Given the low cost of entry level Nokia smart phones it is hard to see an argument that a people who smoke as much as the Greeks can't also afford mobile phones.  Rather cultural differences clearly have a strong impact on defining the rates of twitter adoption in communities.

Having been to Athens many times, and Amsterdam, I would suspect key facts in the differences are:

  • Both cities are major tourist cities, but Amsterdam attracts a large number of young tourists who are more likely to tweet.
  • Amsterdam is a major hub of IT development and excellence.
  • Greeks have preserved traditional social networks that stress face to face connections.  I assume a young Greek with a smart phone is more likely to see friends face to face on a regular basis than a young Dutch person, giving less reason to tweet other than meet up instructions. 

Friday, 25 January 2013

Tweeting from #Tahrir 2 years since #Jan25

Tracking very high tweeting 2 years after #Tahrir revolution began.  Lots of mobile tweeting, lots of re-tweeting, and lots of it about the Tahrir protest.

China Economy? The Biggest Question

I am deeply concerned about the China economy. Mostly because of the lack of Open and Big Data about China.  China remains a rather obscure nation, one where data that does not fit in to the Government's model is hard to come by.  Over the past year I have development methods for tracking food traffic and consumer discussions which I have applied to markets in the US, EU and Africa.  With these tools I have found I can get a unique view of the concerns of local people who have access to mobile technology, which is a growing segment of the population.

But China is closed to me, I don't have the Big Data made public that allows me to be pretty certain that say the US economy is in well established growth while the UK, Spain and Greece are stuck in recessions.  Its not just the economic being published, I can engage ordinary people via Twitter and Foursquare to get a view of their lives.  I can see the boom in mobile devices and creativity in Lagos and Jakarta,  I can read how many shops in Mall of America had excellent Christmases, and I can see that Oxford Street in London was rather dull this Christmas.

From China I just see a small trickle of tourists from major sites.  Its a black hole.  The people who are telling us all that China is going to continue to boom have missed ever bust so far.

Thursday, 24 January 2013

Big Data, now there is no excuse for just opinions

One of the most amazing impacts of the rise of Big Data is that today there is no reason for us to just have opinions. There is enough data out there to be able to argue from a position of informed understanding, always and about just about everything that can be measured.

Take a typical example that happened to me on a recent holiday I had a bit of a row with someone over the relative state of the German vs. the UK economy. Though right now Germany is doing better I said that over the past 15 years the UK had tended to be ahead, and the person I was speaking to was utterly certain that Germany had been booming and leaving the UK in the dust since the start of the 21st Century. Our German supporter then went on to say it was the lower public debt in Germany that had underpinned this economic boom.

So who way right?

Well you all know that since this is my blog I was right.  I actually doubt the person I was debating with would recognise his argument.  But for the sake of argument grant me that two people are debating in a bar, one person says that Germany, because of low government debt, has out performed the UK economy since 2001.  The other person says that German debt is not that low and that Germany has not outperformed the UK until more recently.

Just 10 years ago you would need to consult an expert in economics to get the answer to this.  But who could find an expert in economics?  Ordinary questions of economics were highly dependent upon the chance reception of data on TV, and TV does not contain a lot of real data.

But today things are different.  Big Open Data, much more than just Big Data, has the potential to transform our ability to know, to judge and in time to decide and vote.

Looking a the Data available from Google it is clearly that over the past 15 years the UK economy grew faster and had lower unemployment for almost every year up to 2008. The current dominance of German economy is recent event with very shallow roots.


In fact anyone who rests confident that the German present economic advantage is indicative of a long term established trend would be wise to look at the Irish data from 2001 to 2008.  Until 2008 the Irish growth data was mostly stronger than the UK data.  At present there is really no more reason to assume that the German current domination will last generations then there was to assume that the Irish rise was a major shift.

And as for the idea that low German debt is causing a long term boom, again the data is not there.  The assertion that low German debt is driving a economic boom over the UK is wrong on so many levels.  Firstly German debt levels have not been going down between 2000 and 2010, they were tending to go up.  Second until 2008 German debt and UK debt as % of GDP were roughly the same.  But even more disturbing is looking at Ireland.  Certainly during the Irish boom her debt was lower than both the UK and Germany for almost all years between 2000 and 2008, but then once her economy goes down her debt situation becomes very bad.

In fact the evidence of data is that low debt is a effect of economic growth but clearly not a cause.  Nations that prospered best the year before are less likely to have higher debt, but if their economies become weak then the debt rises.

The problem is that having this debate at a table the idea of bringing up some tables on a iPhone would not have easily solved the problem.

Looking at Big Data is is clear that I was right and this other person was wrong. The problem is right now is that though the data is out their their is not yet a very easy to use UI where you can easily ask Google for these kind of Big Data questions.  And there is not presently the tools that work well in community allowing people to convince others with the Data.  When was the last time you saw a Big Open Data presentation on Facebook?  (if you are like me it was about 15 minutes ago but I assume my social graph is very odd)

There is a real problem that all of this will be crowded out just as TV's educational opportunity will be crowded out.  I personally have had my views of the world transformed by Big Data over the past 5 years.  I have found myself becoming much more left of where I was say in 1999.  Big Data has changed my view of the world.  But I have long ago relieved I am different.  TV also changed my life as a child.  As a teen I watched Carl Sagan, Masterpiece theatre, and Shakespeare as a teen growing up in suburban Illinois where little of that was taught in schools.  TV help fill a gap in my intellectual development as I struggled with a learning disability that made it hard for me to learn to read.

But form most of my friends at school TV was watching MTV and smoking joints all day long.  For the most part TV stunted the intellectual development of most teens, while it enriched a very few.  This seems to be the trend of modern media: it extends inequality even when it offers options to promote equality.

Probably a few people are going to read Nate Silver's book at use Big Data to change their work, change their understanding of the world and change their political views.  But for most people the endless access to the Internet may be just another source of mind numbing entertainment.

As always the great question of media is how to engage a large quantity of people with Big Data.  Perhaps games that use Big Data, or including information  from Big Data more in news and even entertainment.  My big fear is that Big Data will come but Big Open Data will not be used by many, that data science will remain a black art making some richer and most no better off or even poorer.  From the data above you can see this need not be, but it might be.

Wednesday, 23 January 2013

#EUreferendum raises tweets in Whitehall London

There has been a clear rise in tweeting in Whitehall, in significant part due to discussion of a potential referendum on EU membership, but only about 20% of tweets coming from the area are actually about the referendum. Seems London tweeps have a lot to talk about.
  Real time tweeting levels from Westminster show tweets and re-tweets with locations within 250 meters of Parliament in London.

Heavy tweeting in the Parliament greeted came after the news that the PM would try for an referendum on EU membership in 2017.  But only about 20% of the tweeting was because of the referendum.  Seems Westminster is busy today for a wide range of reasons. 

Real time tweeting levels from Trafalgar Square show tweets and re-tweets with locations within 250 meters of the center of London.  Tweeting in this area was 10% of that in Parliament in the hours after the new broke.

Below is a list of tweets and retweets within 1 KM of Parliament.

Sunday, 20 January 2013

Tracking tweets from #inauguration2013

Tracking twitter at the first inauguration of the Web 3.0 age gave a unique real time insight in to the views of thousands of participants, many of them ordinary people.  From the color of tickets, to the wait times, to the lack of bathrooms and the crowd response to the Speaker of the House and Senator Kerry this event gave a unique uncensored view of national life from thousands of view points, never possible before.

When Biden took the oath we saw over 500 geo-tagged tweets and re-tweets an hour coming from 250 meters around the inauguration site.  Remember that the crowd was vastly dispersed over the entire Mall, so we were seeing a lot of tweeting.

Ordinary voices included things neglected by the major media, like comments on the weather from tired middle aged mothers or complaints about the lines at McDonalds and Starbucks. Suddenly Big Data is there, you can decide what features to concentrate on.  Perhaps you don't care about the historic meaning of the day as much as the impact of crowd events on traffic flows, or service at Starbucks, or which of the singers was most popular with the crowd, or just how many people wear sun glasses in the winter.  This kind of data is now there, if you take the time to mine it. 

The world has been changed by this hyper connectivity in a very deep way.

This tool tracks the intensity of tweeting from the Inauguration. This will be the first such inauguration in the mature age of Twitter! I am following tweets in real time and enjoying reading first hand accounts of the people who are there, which you can do with my tool at the bottom of the page.

Social Media map at 2013 Inauguration in Washington DC.  Over 300 Foursquare checkins.
Social Media map at 2013 Inauguration in Washington DC.  Over 300 Foursquare checkins. 

 Below is a list of current tweets coming from near the inauguration site.

Saturday, 19 January 2013

Twitter and Foursquare predicted UK economic slowdown

My prediction from December 24 based on my tracking to Twitter and Foursquare in London and Mall of America that the UK would experience a poor holiday shopping season, and that it could be so bad as to slip back in to recession has been confirmed, but traditional methods took a month longer than my real-time analysis based on social media associated with location.


 I was able to Foursquare and Twitter predictive ability for an economy.  I conducted the following study.

A twitter trending line for central London, including shopping area, my assumption in this analysis is that a correlation between foot traffic, tweeting levels and retail sales (still untested premise).  If this assumptions are correct I see no sign to indicate a rise in consumer traffic in central London.

I created a set of trackers listing to all the tweets and Foursquare trending at Mall in America in the United States and High Street London around Oxford Street.  I watched both spots closely,  for the week running up to Christmas, tracking both levels of geo-tagged social media usage, to get a measure a measure of traffic compared to normal levels, but also reading thousands of tweets as a measure of sentiment.

These conclusion soon became obvious:
  • London's High Street shopping area, around Oxford Street, was not seeing elevated foot traffic.  Shopping were not out on the streets in extremely elevated levels.
  • Consumer sentiment in London was far more dampened than at Mall of America, were people were openly discussing busy shopping and tweeting about purchases constantly.
So in real time, before Christmas Day, I was able to confidently predict that December would be a weak retail month in the UK.  I was not able to assign a precise number at how weak, but the crowds were simply not there and the sentiment was not there.

Based on this work I am now working to develop my extensive and rigorous models I hope can provide policy makers and analysts real times 'heads up' on the state of the economy in specific places.

Thursday, 10 January 2013

Track tweeting from #CES

This tool tracks the intensity of tweeting at the CES conference in real time.
This tool shows you tweets coming from around the CES conference in Los Vagas.