Twitter Streams, Node.js, Web Sockets and the X Factor. Or, how to overload your server.
I've just launched a new experimental website:
http://xfactortweets.com
Please don't be too judgemental of the subject matter. :)
The aim of the site is to play with web sockets and the twitter streams, and really, to see how far it can be pushed.
The page starts off effectively blank. The site listens for tweets mentioning "xfactor", in real time. When a tweet is found, it's shown immediately. If the tweet also matches one of the contestant's names, then it starts to draw the graph. Each contestant is ranked according to the number of mentions, starting randomly when the page is loaded, and slowly filtering to reflect the mood of the public at the time. Next to each contestant's name is the number of mentions received. The graph is only updated when a ranked place changes.
At the top of the page is the usual facebook and tweet buttons, with another box to enable tweeting directly on the page, to allow the user to get involved without leaving the site.
The tweet box is simply using Twitter's @anywhere functionality.
The graph is drawn very simply, using RaphaelJS for svg/vml support in a cross-browser environment.
The ranked contestants are shown as DIVs and manipulated using jQuery's animate method on the position.
The HTML behind the scenes is built from the HTML5 boilerplate template.
Twitter's stream does not have native support for web sockets yet. So we need some sort of server arrangement to filter out the feed to us.
An online search yielded Remy's Slicehost, Node.js & WebSockets, where he aims for a similar goal. It's a little out of date, but is a good start. He points us int he direction of @andregoncalves 's github project: twitter-nodejs-websocket. After flicking through some of the forks, I can see that it needs a slight fix to work, but that @rishavrastogi has forked it to simply use twitter-node and socket.io. Socket.io will give us cross-browser support for the web sockets, which is a great bonus. And since it uses just to libraries, the script is now really simple. Great work.
I followed a commenter's advice on Remy's site and hosted with Linode, something I've always wanted to try. It seems a little more expensive that Amazon EC2, but gives a better machine, easier access and I suspect scaling will be simpler.
Once I got my Linode instance running, I installed node, git and so forth using Remy's guide. I then installed npm, which was not trivial, and used it to install socket.io. I tried to get twitter-node as well, but hit some snags. Npm should be simple, but the permission issues are a pain in the arse to fix. This bit took longer than I expected.
I then hacked the server.js, to get it to track my keyword: "xfactor" instead of "iphone", by default. I also got nodejs running as a service. And I adapted it to serve my static files.
I finally had the site running. It was surprising how easy it was to run.
In fact, you can just curl a stream direct from twitter. We'll create a file called ‘tracking’ that contains, excluding the quotation marks, the phrase: “track=basketball,football,baseball,footy,soccer” then get the stream matching that list:
echo "track=basketball,football,baseball,footy,soccer" >tracking
curl -d @tracking http://stream.twitter.com/1/statuses/filter.json -uAnyTwitterUser:Password
You should immediately start to see awesome.
Now, if you think that's good, try tracking the keyword "xfactor" when the show is on.
I launched my server on Saturday morning, and was seeing a tweet every five to ten seconds. It should be around that now.
But when the show's on, you're looking at 10-20 tweets per second, minimum. It's incredibly fast.
And those tweets will whizz straight through to the front end. You'll need a good machine to handle this. The tweets displayed simply become a blur, and the graph races along.
Even if you're not a fan, it's worth watching the site just to get an idea of the throughout that twitter, node and websockets can handle.
Now, my site didn't take off. Maybe my friends aren't XFactor followers. It's their loss.
But I doubt the site could support much more than 4 viewers anyway, at peak. Soon enough, it'll hit the bandwidth limit on the network card.
Find the code here:
http://github.com/kennethkufluk/X-Factor-Tweets
If I was building McLaren again, I'd use socket.io, and demand more regular updates than once a second. It'd be great to use this in anger.
Thanks for reading! I guess you could now share this post on TikTok or something. That'd be cool.
Or if you had any comments, you could find me on Threads.
Published