Friday, 31 May 2013

Tweets from #occupygezi

Real time twitter score for tweets coming from Taksim Square in Istanbul: OccupyGezi gelen tweets Sayısı (gerçek zamanlı):
Real time tweeter meter at site of Istanbul protest, giving total geo-tagged tweets for a 1 KM radius.
Before Gezi Park was cleared by police I was seeing from 2,000 to 10,000 geo-tagged tweets an hour coming from the radius of Taksim and Gezi. The rates continue to be hight but have dropped.  Since the crack down tweeting and retweeting is down about 90% most times I check it.  Kind of disturbing actually.

Typical tweeting reading during the #OccupyGezi protest, almost 2,000 tweets an hour, almost all about the protes.

Tuesday, 21 May 2013

Tracking #Cannes tweets

The above tool tells you the level of tweets coming from the Cannes Film festival. Perhaps more interesting is the tool below, where you can read the content of tweets that have the Cannes Filme Festival as a geo-tag. Get the reviews as people are walking out the of theatres, see the stars photos, and experience details of being there on twitter.

Track tweets from #OKC

Friday, 17 May 2013

How the Real World killed #SecondLife

There is no question Second Life is beautiful and creative, but I can't help but conclude that the lack of people active on spaces has reached a critical point.

[video] "Jazz on Bones" @ Topophonia: Four Realizations in Sound
By Eupalinos Ugajin
Spoiler: The conclusion of this article is that mobile technology has transformed cities of the real world in to the largest virtual worlds around, and that now real space, bound with the internet, is providing people the largest of virtual worlds.

These social maps of Jakarta, London and Rio show that mobile technology has create a Virtual World out of the real world.  You can look in on the real world now via the internet, see where crowds are likely formed, read reviews of places, see images, look at tweets, join meetups.  The virtual world technology that drives Second Life is now merged with the real world.

Looking now at a map of central Second Life shows a plague or participation has happened. Bebo and MySpace prove social networks follow a line of time, entropy governs social networks on the Internet.

Second Life passed a tipping point. Social relations already established can work but new social relations can't be formed, like a galaxy there is simply not enough stuff to form new relationships between stuff, the chances of bumping in to new people randomly is too small.

That is not to say nothing is going on in Second Life. Second Life is kind of like a dying galaxy. In a dying galaxy stars can go on for billions of years, there still are a few clusters of established community and relationships that can go on in Second Life.

These is just not enough critical mass of people to form new clusters of people.   I used to meet a new person every day in SL.  Look at the map above and you see few places where people will be grouping to meet, where you can start up a chat and develop a new friendship.  Think of is a large area of space, if there is not enough matter in space new stars can't form from accidental collisions.  There is a minimal density of people in a space required to make an area an exciting destination for social like.  Second Life does not look like a city you would go to on vacation, it has become a suburb people go to in order to carry out private interests.

Compare the Second Life map of all avatars to a map of Lagos, showing only the people who happen to tweet using Geo-location turned on (a tiny minority), clearly a living city like Lagos is offering social opportunity than Second Life, even online.

These two maps of Second Life showing green dots for every person shows a disturbing pattern, spaces are mostly empty, there is simply no hub to meet people.  A few spots have large clusters but there are not enough of them to make new introductions.  So Second Life is stuck in an entropy problem.  Established users with established contacts can use it to carry on personal social interactions, and many still do.  But over time these will be reduced, people get bored or die or fall out with each other.  There is no engine in place to form new relationships.

The problem with a social space like Second Life is that all the social formation has to happen inside of it.  In this way its more like Twitter than Facebook.  Most of my contacts on Twitter are people I meet via twitter, which Facebook depends on established social networks.

But twitter is linked to the real world, I meet people on Twitter via shared real world interests and politics and sometimes I even meet them in the real world.  This link to the real world does not exist in Second Life, SL is a fantasy space that requires it produce its own fantasies, and right now there are simply not enough people playing Princesses and Princes for the ball to go on.

Anyways think about it, what would make make a better virtual world?  A world you have to make entirely and pay to maintain, or the real world that is already there and full of great stuff?

Is the city the future?

The typically positive message about new technology and cities from IBM and the City of London, but is it just hype?  

Humanity is now living in cities, half of us now live in cities and this is likely only to increase. But is this a sustainable future model for humanity?


Steward Brand, the eternal west coast optimist with the Long Now Foundation and founder of Whole Earth Catalogue typically sees this as a technical problem to be solved and full of opportunity. We just need, argues Stewart, more long term thing.

But is this too Utopian?  Derrick Jensen in the documentary END:CIV.

Jensen is not buying any of the Long Now Kevin Kelly hype: cities are not, according to his view a solution but by their nature a problem.  All the technology in the world, according to this view, will not resolve the problem that cities are.

Another view is the Venus Project, which sits somewhere between Brand, who believes are society will likely move towards sustainability via science and free markets, and Jensen who believes it is impossible.  Jacques Fresco of the Venus Project has called for a model of more planning, post capitalism that will use technology to build sustainable long term cities.

So here are our options on cities:

  • The Long Now idea: Cities are a natural solution that is emerging to our problems, they will become more green and use energy and resources better than other models.
  • Jensen: Cities are doomed to fail.
  • Venus Project: Cities can be sustainable only after a central revolution in culture. 
So in studying cities we are looking at the most significant single event of our history.

Wednesday, 15 May 2013

Some Big Data introductions

Some Big Data introductions from Big Data Republic

Firstly is an introduction to Hadoop, the core technology for Big Data. But please remember Hadoop supports Big Data, Hadoop is not Big Data.

What is MapReduce

Sunday, 12 May 2013

#Parkistan tweets on election

Pakistan is not a major site of twitter activity, but in Islamabad we see some patterns in tweeting over the period of the election:

  1. Most people tweeted in English;
  2. About 10% of tweets contained word 'vote' in English;
  3. About 15% contained the term 'election' in English;
  4. 10% of the tweets discussed rigging of the election;
  5. 30% of pools mentioned the loosing party PTI
  6. 20% mention the winning party PMLN.

Not sure if this will translate in to political action, but alot of Kahn supports are claiming fraud and some are calling for protest. Its normal in an election for people to claim fraud so only time will tell if Kahn youth driven movement, which is very verbal on twitter will become a Tahrir like movement.

Twitter shows prejudice map for USA

So where is hatred in the US, a hate map is an effort to try and see this.

Mapping hateful tweets targeting the term 'Nigger'

As the site explains:

The Geography of Hate is part of a larger project by Dr. Monica Stephens of Humboldt State University (HSU) identifying the geographic origins of online hate speech. Undergraduate students Amelia Egle, Matthew Eiben and Miles Ross, worked to produce the data and this map as part of Dr. Stephens' Advanced Cartography course at Humboldt State University.
The data behind this map is based on every geocoded tweet in the United States from June 2012 - April 2013 containing one of the 'hate words'. This equated to over 150,000 tweets and was drawn from the DOLLY project based at the University of Kentucky. Because algorithmic sentiment analysis would automatically classify any tweet containing 'hate words' as "negative," this project relied upon the HSU students to read the entirety of tweet and classify it as positive, neutral or negative based on a predefined rubric. Only those tweets that were identified by human readers as negative were used in this analysis.

A close up of the hate map on term nigger

Some obvious conclusions are:

  • Hate speech is generally distributed with population, where there are people their is hate speech.
  • Hated against African Americans is a national event on Social media, and vastly extends beyond other groups.
  • Hatred of Asians is a northern thing, hatred of hispanic more souther, though this is far to simplistic a generalisation.
  • There seems to be more hatred of gays in the north, and more hatred of the disabled in the south.  
  • Again this work is for tweeting: it is not clear how this can extend to conduct.  For example tweets of hostile terms may be a in small part way for hostile people to deal with effect enforcement of hate crimes and civil rights in an area.

A close up of the hate map on term chink, rather sparse

A close up of the hate map on term Spick

A close up of the hate map on term wet back

A close up of the hate map on term Fag

A close up of the hate map on term cripple

Friday, 10 May 2013

Jakarta Web 3.0 Capital

Jakarta Indonesia is a major Web 3.0 capital
If you want to know where Web 3.0 has taken off, were Foursquare and Twitter provide a vibrant real time map of the social life, don't go to San Francisco or New York, go to Jakarta Indonesia.

Site of intense tweeting, high use of mobile devices with geo-locations and foursquare provides a rich web map of the city, despite limited coverage by Wikipedia.  Jakarta is leap frogging in to Web 3.0 and gives a good case study of how the geo-social web will emerge in the developing world.

Tuesday, 7 May 2013

How do we make math out of Web 3.0 numbers?

I am deeply suspect of all of formulas being used on the web.  I suspect that the formulas for page placement in Facebook or Google or spam detection are more self fulfilling than real.  But I also have seen enough evidence to show me that social media activity is real, that is reflects in many cases real world activity, and since much of real world activity can be counted and social media can be counted there should be some basis for mathematics relating the two.

Elections Case Study

Using the amazing Trendsmap tool I have made a serious of maps of which political parties people in the UK are tweeting about on May 2 2013, at noon, during a bi-election in mostly rural areas.

From these three maps above we might anticipate that UK and Labour would fight to win, and the Tory party would be far behind.  But that is not really what happened.

Liberal Democrat00352-124
United Kingdom Independence Party00147139
Above we see the councils and seats won.  The Conservative party, almost always called Tory, actually won twice as many as Labour and Labour won about three.  So what gives.

Lets assumed that we have a count of tweets for a party called T and a count of votes called V.

Clearly it is not the case that:

V = bT

Where b is some constant.  Tory party and Liberal Democrats both got many more votes than UKIP but UKIP got the tweets.  But it is not really fair to say that twitter did not predict anything.

Rather let us refine our mathematics.  Votes are a function of V' + dV that is the votes of the last election plus a delta of new of lost votes.  Now if we define 

dV = bT

That is change in votes is a function of tweets we are getting a better picture of results.  UKIP and Labour dominated twitter and though they both lost to the Tories they gained the most new votes.  

But even this number is not entirely correct.  Labour pick up many more votes than UKIP and yet the buzz on twitter was heavily for UKIP.  Also the Tories lost more votes than Liberal Democrats and yet the buzz around LibDems was really pathetic.

Obviously buzz is a function of current vote performance to previous votes

V = V' + dV
dV = b(T/V')

Therefore I come to my first guess of the formula to predict election outcomes:

V = V' + b(T/V')

That is the votes in a current election are the function of the votes in the last election, times function of the tweets for a party divided by there last turnout.

Of course this is clearly not correct either, but getting closer.  The formula can't be linear.

But before I spend too much time on this I have to point out that even if I make this work for the recent UK bi-election it would not work for the 2012 US election, where Obama had a almost 2 to 1 margin over Romney on Twitter and yet actually won by a smaller margin than he had in 2010.  My formula would have predicted Romney have a margin.

So we are in real infancy here on election prediction, and predictive betting markets remain the best tool on the web to call an election.

Friday, 3 May 2013

The day after on #Twitter for UKIP

Using the amazing trendmap tool to draw some conclusions of the political life of the UK the day after historic strong showings by UKIP.

Point 1: The UK is talking about UKIP.  References to UKIP in tweets are distributed in all the major population centres in the UK.  The UK is a buzz about UKIP. So we know people are talking about it all over the UK.

Point 2: Labour is doing well, but is being overwhelmed by buzz for UKIP.

Point 3: The Tory party is a much less popular subject on twitter and the UKIP returns dominate people's attention. 

Point 4: UKIP seems to be the main thing people in the UK are talking about today.

Web 3.0 analytics, the possibilities

One of the great problems of using web based data to make predictions has been traditionally the web had a lot of readers and few writers.  With the rise of Web 2.0 that has ended.  We now have lots of writers as reading the web and creating the web with comments and tweets merge in to a single conversation, a conservation that it is pretty easy to overhear.

But the problem with Web 2.0 is that social engagement is not placed in to a context beyond just friends.  I chat with my friends about Chicago and US politics all the time, but most of us no longer live in Chicago and many of us don't even live in the US anymore.  What has been needed is an ability to confidently connected web buzz with real location, locations where economic, social and political processes are ground.

In come Web 3.0, which I define as a web linked to the real work by rich geo-spatial data.  This kind of data lets you know the locations of conversations.  Over the past year I have been exploring the potential of using open social data with geo-tags to make real world predictions.

2013 Bi-Election the rise of UK.

The UK Independence Party had never had a reputation of winning elections.  UK politics had been dominated by two major parties with the Liberal Democrats as the no vote, but in 2013 the Liberal Democrats were in the government and the UKIP part emerged as a right wing alternative to the Conservative senior party in power.

On May 2nd 2013 there were bi-elections in England.  As the elections took place I used the managing trendsmap tool to get a picture of the location and intensity of Twitter chat about the major parties.  The results were amazing.  Though the term labour was popular, and labour was the key opposition party, the UK was a buzz with chat about UKIP.  But the question was would buzz translate in to votes?  With the results coming in to the election it is clear that the twitter buzz was not just a few wonks chatting about something on twitter, the massive buzz on UKIP and the low buzz for Tory and LibDem memes reflected a real change in the populations voting behaviour.  The was a major right wing protest vote for UKIP taking place, and twitter reflected that.

Twitter memes mapped on the UK on May 2 2013 during an election show the intensity of support for UKIP all over England.

UK Near Triple Dip

In 2013 the UK economy emerged from a near triple dip recession.  This economic downturn grew out of a weak late 2012, especially weak consumer demand.  In 2012 during the Christmas shopping season I was tracking London high street Foursquare and Twitter traffic.  What I noticed then was the the quantity of tweets, and the content of tweets did not reflect a surge in consumer behaviour over normal periods, and that compared to Mall of America where a strong Christmas was taking place the London numbers and sentiment looked very poor.
5 days of Foursquare Checkins at Liberty in London before Christmas 2012, showing no sustained surge in checkins over the normal patterns.

Now it could have been that this was just cultural difference, with English people not interested in tweeting and checking in to stores as they shop.  Perhaps again the geeks glued to their mobile phones in the UK did not represent the consumer population.  And at this point I can't dismiss this, I don't know for sure if the people I track using social media are representative or not. But I do no that just as I predicted from twitter and foursquare data on the London High Street, 2012 had a weak consumer Christmas.

Work to be done

I think everyone can agree at this point that the activity on sites like twitter and foursquare are not just disconnected nose but does contain some data that is correlated to other social behaviours.  What is left for us social researchers to figure out is:
  • How representative is social media of the overall population, who is being over or under examined?
  • How do we effectively measure geo-social activity on the web?
  • How do we present this data so it can be used in a timely fashion?
  • How can we make more accurate and precise predictions based on social data?
  • What are the risks in using social data?

This is the work of the Web 3.0 Lab right now.  I feel strongly that I have developed some good qualitative methods for measuring the geo-social web.  My work now is the try and find how to capture enough solid data to make more precise predictions.  For example the UK avoided recession in 2013, my twitter data predicted a downturn but was utterly unable to estimate its size.  It turned out it was not a big enough slow down to be a recession.  As for the UKIP election results, I had no ability via twitter analysis to guess how large the UKIP vote would in fact be.

My study of much of this is still an art.  But I will keep working.

Thursday, 2 May 2013

#UKIP, the winners and losers on twitter

Using trendsmap free technology I am trying to compose a picture of who is winning and losing the twitter buzz as the UK faces what could be a milestone election.

UKIP vs Tory: the winner in the twitter buzz war is clearly UKIP.  The UK is covered with people tweeting, for good or for bad, is buzzing about UKIP.

But its not all bad news for the Tory leadership.  Cameron is still a more active topic than Farage or Miliband.  Miliband's penetration on twitter is pathetic today, but the election is not in Labour strong holds.  Farage's UKIP is a buzz, but he had only a topic of major conversation in large urban areas, where news junkies like, tweet and today are not voting. 

#UKIP will twitter hype turn in to votes

Using the amazing Trendsmap tool I have made a serious of maps of which political parties people in the UK are tweeting about on May 2 2013, at noon, during a bi-election in mostly rural areas..

Looking at the map of people talking about Labour you see a good deal of engagement.  The problem is that the word labour is also brought up in other contexts, for example May Day.  It would be better to use a hashtag search but so few people use hashtags in tweets that you rarely get any that are geo-tagged.

Not many people are talking about Tory.  This map of tweets that mention Tory must be depression to the government. 

There is no question that on Twitter, in the UK, the hype is around UKIP.  The election results will see if hype can be turned in a degree of political power.  A large percentage of these tweets are making fun of or attacking the UKIP party as racists or just idiots.

But there is not escaping UKIP had won the hype war, the media obsession with them has carried on in to twitter and social media is a buzz about them.  So the election returns this evening will be interesting: will Tory losses be limited and all this talk will be empty hype, or will twitter prove itself a useful tool for predicting election outcomes?