Wednesday, 28 November 2012

The Sound of dial-up Internet



Just to put things in to perspective, this was the necessary sequence that most people trying to get online had to go through just 12 years ago.

Friday, 23 November 2012

Tweeting levels in #Tahrir show major protest تويت في التحرير



Real Time Twitter density in Tahrir Square. Refresh page to see update, click on the Full Screen View to get a full screen version of tool, click See Tweets Around Here to see the last 100 tweets from the area.
Embedded code <iframe height="450" src="http://happan.in/m/?lng=31.23460292816162&lat=30.046287454189482" width="400"></iframe>
Graph above shows tweets and trend line for 4 days before 10 PM December 11 2012.  We see a clear pattern of rises and falls in a 24 hour cycle.

Graph showing tweets per hour in 1 KM readies in central Cairo around Tahrir Square. Graph shows 2 hours before 1 AM December 11, time period when an attack was reported on the Square.  The orange line is the 5 point trending line showing a clear rise in tweets suddenly in the 15 minutes before 1 AM.


UPDATE: December 11, 1 AM Cairo time. We saw a drop in tweeting about a half hour ago on reports that a group of people had attacked Tahrir Square, now we are getting reports the attack has needed and tweeting and retweeting from in and around the Square has suddenly surged to almost 1,000 geo-tagged tweets an hour in a 250 meter radius.

December 11 at 1 AM: Tweeting levels surge from RTs by Egyptians of reports coming from Tahrir Square itself
Update December 7 2012. Friday is the new Friday.  Many people have noticed that Friday has not been a major day of protests at Tahrir.  Its seems the Morsi speech may have changed that.  There are massive quantities of tweets coming from around Tahrir Square now. See more on Tuesday and Fridays in Tahrir.

Dec 7 2012 9:20 PM and geo-tagged tweeting from Tahrir is over 1.500 tweets an hour in a 250 meter radius area.


Dec 7 2012 8:45 PM and geo-tagged tweeting from Tahrir is over 500 tweets an hour in a 250 meter radius area.
Update December 6 10:20 PM, Morsi is giving a speech and it is causing tweeting to rise in Tahrir and Cairo.

Over a week of tweeting reading, moving averages and trend line. Click to enlarg
UPDATE: 1:30 PM Cairo time December 6.  We are seeing no sign of a decline in intensity of tweeting coming from central Cairo.  We are seeing very 'erratic' movement in the number of tweets per hour coming from the a KM radius of Tahrir.  This is partially due to the erratic nature of Egyptian mobile and Internet providers no doubt. I am looking in to the possibility that the jumps in tweeting indicate periods of intense change in the situation.  We definitely see something that looks like this over the past 24 hours and last week.


UPDATE: 1PM Cairo time December 6.  Despite rumours of a shut off in Internet we are still seeing a lot of tweets coming from central Cairo.  But levels are not as high as during recent protests.  Actually we have been seeing a pattern of surges on Tuesdays followed by tapering off through the week, with surges agains on Tuesday but this has only been going on few weeks.

Update: 11:20 PM Cairo time December 5 2012.  Reports that the Muslim Brotherhood is mounting attacks near Tahrir are increasing on twitter.  We are seeing somewhat subdued levels of tweeting for the evening, that would be in line with a group under physical threat, but I can not confirm that. 


UPDATE: December 5.  Today we have seen what can only be described as massive levels of tweeting with geo-tags related to Tahrir Square.  Almost 1,000 tweets an hour at one time is inline with the kind of tweeting you might see in Madrid at a protest with over 100,000 people.  Much of this seems to be discussions going on about the removal of tents from Tahrir Square by members of the Muslim Brotherhood. From reading translations of tweets around Tahrir it seems clear that a Muslim Brotherhood counter offensive it taking place, unable to stage these levels of protest the Muslim Brother hood is now accusing the Tahrir protestors of immorality and drinking.  Its starting to sound all like Occupy Wall Street.  


UPDATE Tuesday December 4, surging tweets this evening not only in Tahrir but near the Presidents Palace where another protest is taking place. See move extensive analysis here


Real time tweeting near Presidential Palace. Embed code <iframe height="650" src="http://happan.in/m/?lng=31.318631172180176&lat=30.088469796894348
" width="100%"></iframe>


At midnight Dec 4 2012 Tweeting levels in Tahrir square were reaching between 500 and 600 geo-tagged tweets an hour.  Given that only about 5% of tweets have geo-locations, and Egypt is a poor country with often inadequate mobile service, that is a lot of people.  



Image from Foursquare of recent protests posted October 21 2012


UPDATE: Midnight, 31 November, though we have seen very high tweeting today, we have not seen the same level as this Tuesday, which confirm media reports that the crowd was smaller than Tuesday's mass crowd.  Still these are large crowd of social media users. 

UPDATE 1PM Cairo time, November 29.  I have developed this analysis of tweeting elves for the central Cairo area.  This is one full KM radius around Tahrir Square.  With estimates of geo-tagged tweets per hour coming from the twitter public API.

twitter time map of Tahrir Square
The last 26 hours of tweeting level. Notice that during the day tweets average about 100 to 200 tweets per hour, but we are seeing lots of surges in tweets.  The surges are most remarkable. 

twitter time map of Tahrir Square vs Amsterdam
In this map the blue is for Central Cairo, the red is for Amsterdam.  Not surprising that overall Amsterdam produces more tweets, as more people have mobile phones and computers.  It is also not surpassing that the levels rise and fall together as they are close time zones.  What is remarkable is the spikes in Cairo well above the range of the Amsterdam.  So while normally Amsterdam has higher overall tweeting than Cairo, at certain times tweeting in Cairo rises well above Amsterdam.  This makes sense because Cairo tweeting is very event driven, with news stories on on the ground even driving tweeting while Amsterdam tweeting is more every day. 


UPDATE 2 AM Cairo time, November 28.  This picture tells the entire story, showing the tweeting levels in Tahrir Square and London at the same time.  Tahrir Square is having many times more tweeting than Soho in London!


Twitter usage in Soho London vs Tahrir Square, midnight GMT.  Tahrir is having almost 8 times the twitter activity.
UPDATE: 8 PM Cairo time, we are seeing now tweets surging up to 500 geo-tagged tweets an hour.  This indicates a crowd in the 10,000s in a city like Cairo. 


UPDATE: 6 PM local time November 27.  The call to evening prayers can be heard on Reuters live streams as a massive cloud of tweets rises above Tahrir.  Tweeting surged to over 400 geo-tagged tweets in hour in a massive demonstration. 
UPDATE: 3 PM November 27, the crowd continues to build.  We have seen some surges in tweets to up to 300 an hour, but so far the levels have returned to the 50 to 100 range.  



UPDATE: 1 PM November 27, we are starting to see a surge in tweets from the crowd in Tahrir.  We could see a crowd coming in from the north over the last half hour as well.  Interesting that as the crowd became louder the tweeting level also surged. 

UPDATE Noon Tuesday November 27 the crowd is remaining stable, in the hundreds and the tweeting levels reflect that, with only about 50 to 60 geo-tagged tweets an hour.  

UPDATE 11:30 Tuesday November 27, we will be tracking to see if the large demonstration planned for today shows up on twitter traffic.


UPDATE: 8:00 PM on November 24 and levels are well below 100 an hour.
UPDATE: 1:10 PM, we are seeing a sudden surge in tweets coming from Tahrir Square. 

UPDATE 12:45 PM November 24, twitter indicates a slower start, both levels and content of the hashtag #Tahrir


UPDATE 12:15 AM November 24, tweeting level from Tahrir Square has been dropping over the past half hour, fro over 500 geo-tagged tweets an hour to just over 150 now.

Update: At 11:45 PM Cairo time we are seeing a major surge in tweeting coming from Tahrir Square.  As November 23rd comes to an end in Egypt we are seeing over 500 geo-tagged tweets an hour being reported by Twitter for a 250 meter radius around the center of Tahrir Square, this is a major stream of tweets.  Only about 1 in 20 tweets have a geo-location, so this could mean tens of thousands of tweets an hour are coming from the area.



Tahrir Tweeting levels at 11:45 PM November 23rd 2012.


UPDATE: 6:10 PM and there are more than 300 geo-tagged tweets an hour coming from Tahrir Square.  These levels are extremely high for Cairo and indicate a large protest including most of the young educated politically active class. 6:15 PM sees the tweets going to 500 in what is clearly a major event.

UPDATE: 6:00 PM Cairo and we are seeing the level of tweets coming from Tahrir continue to rise, we are now near 141 geo-tagged tweets an hour from 250 meters around the center of the Square.  Given that only about 5% of tweets are geo-tagged that is a few thousands tweets an hour.  Given that Egypt is a poor country where few people have smart phones that is a large crowd, and must contain many of the educated young who help to mobilise the Arab Spring.





Twitter is carrying a number of reports of tear gas being used.  Two years after the start of the Arab Spring things are feeling like old times in Tahrir. The elevated tweeting levels and the content in the Twitter hashtag Tahrir indicate a very large disturbance is taking place in Cairo, perhaps on the scale of early 2011.






Mapping #BlackFriday with Web 3.0



This tool will show all the current checkins and venues in Foursquare, Yelp, Flickr and Wikipedia at Mall of America.  Click on icons to get more information about the venue and see checkins.

Tracking twitter activity from major venues in the US on Black Friday

Mall of America, North America's largest Mall by store count is seeing steady tweeting through the night. At about 1 AM local time we say about 40 geo-tagged tweets an hour coming from the Mall, indicationg thousands of shoppers are out in Mall of America for Black Friday.

Real time tweets from Macy's in New York City.  These reached over 500 geo-tagged tweets an hour doing the Macy's day Parade so it will be interesting to see if the shoppers will be out tweeting.



The Apple Store at Fifth Avenue in New York City.  Anticipate a lot of people tweeting about new Apples at this Mecca for Apple fangirls and fanboys.  I have to admit I make it a point to go when I visit New York.


Walmarts don't tweet? The massive Walmart in Albany New York is showing no real tweeting despite the presence of what we assumed to be hundreds or even thousands of shoppers, interesting to watch.


Thursday, 22 November 2012

Tweeting levels at #Macy's #Thanksgiving Parade


Real time tweets per hour at Macy's parade, we are seeing almost 500 geo-tagged tweets per hour around Macy's parade at Broadway in New York.  As we are watching the first hour of the parade levels of geo-tagged tweets are rising to over 500 an hour.  Given that only about 5% of tweets have geo-locations that means several thousand tweets an hour are likely coming from the parade right now.

Tweeting leves per hour at Macy's Parade 2012
Tweeting leves per hour at Macy's Parade 2012


Not surprisingly further north on Broadway just south of Central Park tweeting elves are far far lower.

Tweeting leves per hour well north of Macy's Parade just south of Central Park.




Wednesday, 21 November 2012

Track twitter celebrations from #Gaza



Current Tweeter level used in Gaza city center

Very high tweeting starting in Gaza City Center after ceasefire, tweets reached over 2,000 geo-tagged tweets an hour in a 250 meter radius in the center of the city.
It is interesting to note that Israel blocks geo-located tweeting services, so no such score can be obtained for Tel Aviv. 

Monday, 12 November 2012

#OpPedoChat Anon targets online pedophiles


Anonymous, never ones to sit still for every long, have launched a fairly extended operation targeting online pedophiles.  The hashtag on twitter is #OpPedoChat.

There is no question that pedophiles operate very freely on the web.  My own research over the years has time and time again found pedophiles openly sharing images and making contacts on Flickr.  Flickr clearly does almost nothing to break up the networks of adult men who repeatedly favourite images of young girls playing at pools or beaches.

I personally am concerned about witch hunts, but the level of pedophile activity on the web and the unwillingness of major companies and governments to do anything about it probably makes what Anonymous is doing necessary.

Probably only a mass exposure of pedophile sites and networks via a mass event like this can do anything.  It is extremely difficult to counter specific users one at a time.

Tweet levels from key EU #N14 #N15 protest sites

UPDATE: In are benchmark protest cities of Athens and Madrid the general strike has had little impact on Twitter. The pictures we have seen of mass protests mostly come from cities we have not been tracking. This may be due to a number of explanations:

  • That these protest are more the mainstream union support who don't use Twitter
  • That the major urban centres are 'protested out' but that protests are moving to new areas like Italy and smaller Spanish cities and in to the north of Europe.
  • The General strike is something of a dud. 
  • Much of the action on sites like Twitter and Facebook are people who don't have jobs (students and unemployed youth) so they have nothing to strike from.
It is always hard to conclude much from negative data.  But these protest don't have the kind of impact on social networks that Occupy did last year.  This may actually be that 'people whose opinions really matter' are becoming more involved, or it may indicate support for the general strike is weaker.

Real Time Tweeting level from Syntagma Square Athens, Athens tweets rarely go above 20 an hour unless something is going on.

Los niveles actuales de Twitter en las protestas
Os níveis atuais de Twitter em protestos
Τα σημερινά επίπεδα των διαδηλώσεων στο Twitter

The following dashboard can track tweeting levels at key protest sites in the UK. It was creating using the Happan.in tweeter meter.  I have noticed that Spain is the most engaged mobile twitters users in the current nations impacted by strikes.  Levels of tweeting can reach extremely high levels in Spain.  Athens has seen a recent rise in the use of twitter at protests.  Lisbon still is not a major user of twitter.  So it will be very interesting to see how the different sites twitter level rises and falls during the protest.



Plaza del Sol. Madrid Spain.

This is a high tweeting location, on a weekend 40-60 geotagged tweets an hour would be high. We anticipated about 200 or more an hour at the height of the protest.



Sunday, 11 November 2012

Wisdom of the Crowd

Odds of the election out come set by betting on the Iowa Election Market 
There are two stories about the 2012 election forming.  On one hand is the story about how Romney and his people thought for sure they had a landslide coming. This story is about the intentional ignorance and denial of ruling elites.  It is a story we are familiar with over the past 10 years, where time and time again our political and government elites have been caught by surprise by events from Iraq to Wall Street.

Gallup and other major polling services utterly failed to predict the outcome despite and extensive resources and skills.
According to this narrative the Romney failure to understand what what about to happen is another case of shell shocked experts taken by surprise by reality. 

One is about Nate Silver and the rise of Big Data science. Nate Silver, and other Data Scientists working at a few polling services and for the Obama team were able to apply difficult statistical science to form a more accurate idea of the political landscape.

According to this narrative the success is a accomplishment of a small number of elites with access to massive amounts of Big Data, much of it proprietary, and elite skills of analysis. 

But if you look at a wisdom of the crowd site like the Iowa Election Market where thousands of ordinary people place bets on the outcome of the election. These crowd sites did a far better job of predicting the the outcome than the Romney organisation or the major polling services, especially Gallup Poll who utterly failed.

The bets of ordinary users at the Iowa Election market gave a far better picture of the likely outcomes of the election than Gallup poll.
So here is a narrative that is being ignored.  The ordinary crowd betting on Iowa Election Market did a far better job calling the election than Gallup Poll.  Nate Silver did good, but he didn't do much better than ordinary crowd sources betting sites in setting the odds.

It is critical that this success of the wisdom of the crowd not be ignored as worthy praises is layered on Data Science and Nate Silver. 

Tweets from Remembrance Day Ceremony London

Update: a real example of twitter entropy at work, after the 11 am ceremony the tweeting at the monument declined while tweeting rose in other area around it as people spread out.
Seeing very high levels of tweeting from the area around the ceremony for Remembrance Day in London.  Below shows a bit further north.  We are estimating a few hundred tweets an hour within 250 meter of the center of the ceremony, with much lower levels of tweeting just outside.




These levels of tweeting show how much the use of mobile phones and twitter has become a part major events in the western world. 

Saturday, 10 November 2012

Social killed Semantic

The meme killed the fact. The web will not only tell you what ever you want to know, but the social web will tell you what ever you want to hear.
The meme killed the fact 

A great deal of the early hype around the Internet saw the web as a information distribution network.  That the web would provide accurate information to people anywhere in the world.  This viewed the internet's positive economic and social benefit in reducing educational costs, informing consumers and promoting innovation.

This view stressed the need to define the web as a system of semantic statements.  It aimed to create things like triple stores that would contain the information held by the web.  That is it would extract the truth in the web.

But the meme killed the fact.  

Meme's don't need to be true and memes that contradict can co-exist.  The social Internet is a bucket of memes and not a collection of theorems.  Unlike facts memes can contradict each other.  Meme's invite but to not demand. They even plead and pander.

Today every fact that can be imagined, every world that can be wanted, can be found as a meme on the web.  What ever you want to be true there is probably already a hashtag on twitter.  There is a community full of people who hold the it is true.  There are blogs and groups on Facebook that will convince you that what you want to be true is true, which is not hard to do.

This explains the webs relatively poor ability to see the outcome of the 2012 election.  People were getting trapped in the bubbles constructed of memes they liked and excluding memes they don't like.  Through Facebook, Twitter and blogs people got a filtered view of the world that confirmed to their world view.  For Republicans this meant the world was full of highly energetic Romney supporters who would hand the GOP a historic victory.  The election came as a strange moment of reality principle crashing in the vast fantasy that the web has become.

That is why democracy is so critical now.  Post-modernism is wrong, deeply wrong.  Science, democracy and law are now not nullified by the flood of media narratives, they are more important now than ever.  We need rational, universal discourses to prevent our culture from collapsing in to the chaotic foam of bubbles that the Internet by itself would degenerate into. 

Wednesday, 7 November 2012

Cyberspace is Dead

Our real world is more and more merging with the digital.
Reality is now the Web. Cyberspace is dead.  We are merging more and more with our digital world, bringing the two together. 

Mobile Tweeting from 2012 Election

Okay I stayed up all night watching the election to see how news impacted the level of tweeting from the ground.  I used the happen.in tweeter meter to make maps of key locations during the election.  This tool queries the number of tweets associated with a geo-location from the Twitter API, taking the most recent sample of about 50 tweets and calculating the rate at that time, for a radius of 250 meters.  Times on the screen grabs are in GMT.
The White House was a massive hub of tweeting, with almost 2,000 geo-tagged tweets coming from a 250 meter radius area in the hour when CNN called Ohio.  With about 5% of tweets geo-tagged this meant that 10,000s of tweets were coming from inside and around the White House.
Levels of tweeting from the White House are usually high, but these are extremely elevated, which should come as no surprise.  Rather it reflects the importance of social media to this President, to the media and to his supporters..


Real Time tracking of tweets from the White House Area

Twitter levels at the White House Washington DC
This reading was taken on the 25th of October when Powell endorsed Obama.  These were elevated readings but turned out to be a fraction of what we saw on election night. 

New York City's Time Square was a gathering spot for thousands of people with tweets going over 500 an hour when Ohio was called.  This reflects a crowd of several thousand people.  This is more a pure measure of popular use of tweeting to celebrate a victory, and proves that many of the White House tweets must have come from common people using smartphones. 
The story at the Romney site was much more subdued, though most people in Boston were celebrating an Obama win.  The area around Faneuil Hall where Romney was watching the results was subdued.  This reflects not only defeat, but perhaps also the less importance social media holds for Romney, the GOP and its supporters.  But the location of Romney in Boston complicates all conclusions.
Perhaps the most surprising what the limited extent of tweeting from the actual HQ sites were supporters had gathered.  Obama's Chicago HQ has a slight advantage in tweeting form location all night, but started with a significant advantage before the night began.

Not only was there no clear evidence of the enthusiasm gap between Obama's and Romney's people at the HQs, but the levels of tweeting, though elevated for those locations, were not by any means 'high'.


Maybe people didn't want to use their cellphones during a historic event.  From each site you got just under 100 geo-tagged tweets an hour at any time, which only translates in to maybe 5,000 tweets per hour.  I had imagined they would be higher, though they did rise during the event, especially at the Romney HQ were they went from 0 geo-tagged tweets an hour to almost 100. 

Tuesday, 6 November 2012

Tweeting levels at Obama and Romney HQ


Track tweeting above at Obama's HQ in Chicago.  In the middle of the night we already saw significant action there.  Remember that Chicago is a city that strongly supports Romney so you will see a lot of tweeting of either joy or regret around it by local people who may come out to celebrate.



So far Romney's HQ in Boston has been very silent.  Boston and the entire state are heavily supporting Obama so there will be a difference between what pleases people inside the Romney HQ and the people of Boston.

Our prediction from this.  Obama's HQ is showing much more tweeting, indicating energy and a strong social media strategy that is constantly running.  Romney's HQ is probably just a space to hold an event, but the lack of tweets coming from the HQ would indicate a weak social media campaign and a disheartened support group.

If you want to see how the crowd source is setting the odds track the Iowa Election Market below.


Crowdwire has a fun app counting twitter vote mentions, but since social media users tend to be younger it is giving unreliable biased numbers in favour of Obama, not that Obama will almost certainly win, but not by the twitter level.

Cities and Data

 The economist has a interest piece of the connection between cities and computer technology.  One section was on what I would call Web 3.0:

A third factor is becoming increasingly important: the production of huge quantities of data by connected devices, including smartphones. These are densely concentrated in cities, because that is where the people, machines, buildings and infrastructures that carry and contain them are packed together. They are turning cities into vast data factories. “That kind of merger between physical and digital environments presents an opportunity for us to think about the city almost like a computer in the open air,” says Assaf Biderman of the SENSEable lab. As those data are collected and analysed, and the results are recycled into urban life, they may turn cities into even more productive and attractive places.

This sums up very nicely much of my work with the Web 3.0 Lab, but being the Economist the story has a 'opptomistic' outlook.  This high connectivity also will make people in cities more vulnerable to crime and surveillance.  The often discussed issues of identity and community problems that come from what one might call 'internet addiction' stand to be more significant and more re-enforced in cities.

But that said I have become convinced for some time the the city and the Internet are connected concepts: that more and more the logic of city life and the logic of Internet community are merging. 

Sunday, 4 November 2012

November 4 Foursquare NYC

#OperationVendetta in London Nov 5 tweet tracker


Findings: Over 2,000 people showed up in front of the National Gallery for event, many of them wearing Fawkes masks.  The site of so many people in the masks lead to a lot of photography on smart phones and obviously a lot of post to the Internet including tweets.  We saw a lot of tweets from the location on the hashtags #OpVendetta #OperationVendetta and #5thNov and some of them must have come from the site.  We also so a significant rise in levels of tweeting in Trafalgar Square during the gathering.


On a Monday night in winter Trafalgar Square will normally see 20 to 50 geo-tagged tweets an hour, but during the event the number reaches closer to 200.  A significant rise, but not extremely high.  The tweeting was on scale to a many Occupy events in London.  These levels were higher than recent NUT protests and the Olympics, but Anonymous people and their supporters are in to technology.

Stream videos at Ustream
Streaming Video

Twitter level tracks for key sites in the Operation Vendetta

Demonstration is suppose to start here in Trafalgar Square about 6 PM GMT. I would anticipate a score of 97 with a few hundred tweets an hour coming from this location is anything major is happening.

It will assemble here, problem is that Parliament is a information hub for the UK, with tweets flooding in to cyberspace from here at a high rate, it might be hard to see any increase in tweets.

Operation Vendetta Facebook Page
Okay I am not entirely sure how subversive an event planned on Facebook can be, but the Operation Vendetta Facebook page has over 5,000 people confirmed to go.  Almost another 2,000 likely to attended.

The event is scheduled for November 5, 2012.  I have read different times on Twitter and Facebook, probably between 5 and 6 PM.  A flyer that has been going around London for some time said Trafalgar Square but they are not that far apart.  



This is clearly a serious effort by Anonymous UK, who ever they are.   Last month an attack on the Conservative Foundation Web Site also mentioned details about the event.  That post called on people to assemble at Trafalgar and move towards Parliament.  It could be the largest flash mob in history.  The idea seems to be to act out the closing sequences of the V for Vendetta movie, which included the exploding of Parliament.  I profoundly doubt the halls of Parliament are going to be blown up, but I suspect someone will try to play the 1812 Overture.








 Photo by Gail orenstein
    Social Media was used through out the night on mobile phones for participants to communicate  
    where the location of the march was and to check activity on Twitter and Facebook.