node.js knockout — August 28th & 29th

A few days ago a small team of SoundCloud developers (@goldjunge, @jberkel, @purzelrakete and @sohm) participated in the first node.js knockout competition. Aptly named “Team SoundCloud”, we set out to explore the current state of server side Javascript using node.js and the real-time web.

(For those unfamliar with Node.js, head over to http://nodejs.org/ or the project wiki at https://github.com/ry/node/wiki for an introduction.)

The node.js Knockout is a competition inspired by the popular Rails Rumble. The rules are simple: Teams of up to 4 people have 48 hours to build a web app that is awesome enough to woo the judges and results in as many votes from the audience as possible. To level the playing field, each team was required to deploy their application to a dedicated hosting environment provided by either heroku.com or joyent.com. The competition started on Saturday at 2:00AM and ended on Monday 2:00 AM. Yes, that’s a weekend right there.

We had done a minimal amount of planning beforehand and agreed that we’d try to visualize traffic on soundcloud.com by hooking into the varnish HTTP accelerator that’s part of our infrastructure. We basically set out to build a pimp HTML5 + WebSockets based version of varnishtop (with a sprinkle of varnishlog, because that’s just so exciting!).

@purzelrakete, @goldjunge and @jberkel hard at work.

Being confined to the Joyent environment, the first challenge to overcome was not having access to a varnish instance that serves any real traffic. To solve that, we hooked into the Twitter Streaming API and filtered all links to the popular URL shortening service bit.ly. For each bit.ly link a request is made against the local varnish instance to generate traffic. Of course, all of these requests result in (eventually cached) 404s, but oh well. The rest of Saturday saw us create a layout, start working on the C++ extension required to access a running varnish instance’s shared memory to read logs & statistics and generate events in node, culminating in having the basic functionality and deployment configuration done at around 3AM on Sunday.

After a few hours of sleep, Sunday was a blur of getting the C++ extension to emit more statistics and log details, writing the client-side javascript that updates the DOM and wrestling with Solaris on the Joyent SmartMachine. We deployed the final version about 30 minutes before the competition deadline. Lessons learned (in no particular order):

• node.js is fun, building realtime stuff with it is incredibly easy
• deploying early and continuously == win
• Solaris is a pain
• Twitter is full of spam and porn
• Server side javascript!!

You can check out the app here (using Chrome or Safari): http://team-soundcloud.no.de/

Vote for us here: http://nodeknockout.com/teams/team-soundcloud