Building The Next SoundCloud

This article is also available in:

• Serbo-Croatian: Pravljenje novog SoundCloud
• Armenian: SoundCloud (ձայնամպ) ծրագրավորողների համար

The front-end team at SoundCloud has been building upon our experiences with the HTML5 widget to make the recently-released Next SoundCloud beta as solid as possible. Part of any learning also includes sharing your experiences, so here we outline the front-end architecture of the new site.

Building a single-page application

One of the core features of Next SoundCloud is continuous playback, which allows users to start listening to a sound and continue exploring without ever breaking the experience. Since this really encourages lots of navigation around the site, we also wanted to make that as fast and smooth as possible. These two factors were enough in themselves for us to decide to build Next as a single-page Javascript application. Data is drawn from our public API, but all rendering and navigation happens in the browser for near-instant navigation without the need to make round-trip requests to the server.

As a basis for this style of application, we have used the massively popular Backbone.js. What attracted us to Backbone (apart from the fact that we’re already using it for our Mobile site and the Widget), is that it doesn’t prescribe too much about how it should be used. Backbone still provides a solid basis for working with views, data models and collections, but leaves lots of questions unanswered, and this is where its flexibility and strength really lies.

For rendering the views on the front end, we use the Handlebars templating system. We evaluated several other templating engines, but settled on Handlebars for a few reasons:

• No logic is performed inside the templates, which enforces good separation of concerns.
• It can be precompiled before deployment which results both in faster rendering and a smaller payload that needs to be sent to clients (the runtime library is only 3.3kb even before gzip).
• It allows for custom helpers to be defined.

Modular code

One technique we used with the Widget which ended up being a great success was to write our code in modules and declare all dependencies explicitly.

When we write code, we write in CommonJS-style modules which are converted to AMD modules when they’re executed in the browser. There are some reasons we decided to have this conversion step, possibly best explained by seeing what each style looks like:

// CommonJS module ////////////////
var View = require('lib/view'),
    Sound = require('models/sound'),
    MyView;

MyView = module.exports = View.extend({
    // ...
});

// Equivalent AMD module //////////
define(['require', 'exports', 'module', 'lib/view', 'models/sound'],
  function () {
    var View = require('lib/view'),
      Sound = require('models/sound'),
    MyView;

  MyView = module.exports = View.extend({
      // ...
    });
  }
);

• The extra define boilerplate is tedious to write
• Duplication of module dependencies is also tedious and error-prone
• Conversion from CommonJS to AMD is easily automated, so why not?

During local development, we convert to AMD modules on-the-fly and use RequireJS to load them individually. This makes development quite frictionless as we can just save and refresh to see the updates, however it’s not so great for production since this method creates hundreds of HTTP requests. Instead, the modules are concatenated into several packages, and we drop RequireJS for the super-lightweight module loader AlmondJS.

CSS and Templates as dependencies

Since we’re already including all of our code by defining explicit dependencies, we thought it made sense to also include the CSS and template for a view in the same way. Doing this for templates was rather straight-forward since Handlebars compiles the templates into a Javascript function anyway. For the CSS, it was a new paradigm:

Views define which CSS files are needed for it to display properly, just the same as you would define the javascript modules you need to execute. Only when the view is rendered do we insert its CSS to the page. Of course, there are some common global styles, but mostly, each view has its own small CSS file which just defines the styles for that view.

When writing the code, we write in plain vanilla CSS (without help of preprocessors such as SCSS or LESS), but since they are being included by Require/Almond they need to be converted into modules as well. This is done with a build step which wraps the styles into a function which returns a <style> DOM element. Here’s an example of how it looks in essence:

Input is plain CSS

.myView {
  padding: 5px;
  color: #f0f;
}
.myView__foo {
  border: 1px solid #0f0;
}

Result is an AMD module

define("views/myView.css", [...], function (...) {
  var style = module.exports = document.createElement('style');
  style.appendChild(
    document.createTextNode('.myView { padding: 5px; color ... }');
  );
})

Views as components

A central concept in developing Next is that of treating views as independent and reusable components. Each view can include other ‘sub’ views, which can themselves include subviews and so on. The effect of this is that some views are merely composites of other views and cover an entire page, whereas others can be as small as a button, or even a label in some cases.

Keeping these views independent is very important. Each view is responsible for its own setup, events, data, and clean up. Views are ‘banned’ from modifying the behaviour or appearance of their subviews, or even making assumptions about how or where this view itself is being included. By removing these external factors, it means that each view can be included in any context with absolute minimum fuss, and we can be sure that it will work as it is supposed to.

As an example, the ‘play’ button on Next is a view. To include one anywhere on the site, all that we need to do is create an instance of the button, and tell it the id of the sound it should play. Everything else is handled internally by the button itself.

To actually create these subviews, most of the time they are created inside the template of the parent view. This is done by use of a custom Handlebars helper. Here is a snippet from a template which uses the view helper:

<div class="listenNetwork__creator"> {{view "views/user/user-badge" resource_id=user.id}}
</div>

As you can see, adding a subview is as simple as specifying the module name of the view and passing some minimal instantiation variables. What actually happens behind the scenes goes like this:

When a view is rendered, the template must return a string. When the view helper is invoked, it pushes the attributes passed to it, plus a reference to the requested view class, into a temporary object with an id, and outputs a placeholder element (we use <view data-id="xxxx">). The id is just a unique, incrementing number. After a template is rendered, the output would be a string which might look something like:

<div class="foo">
    <view data-id="123"></view>
</div>

Then we find the placeholders and replace those elements with the subview’s element which it automatically creates for itself. In essence, the code does this:

parentView.$('view').each(function () {
  var id = this.getAttribute('data-id'),
        attrs = theTemporaryObject[id],
      SubView = attrs.ViewClass,
      subView = new SubView(attrs);
  subView.render(); // repeat the process again
  $(this).replaceWith(subView.el);
});

Sharing Models between Views

So, we now have a system where there will be dozens of views on the screen at one time, many of which will be views of the same model. Take, for example, a “listen” page:

There would be a view for the play button, the title of the sound, the waveform, the time since the sound was uploaded (this dynamically updates itself, which is why it is a view), and so on. Each of these views are of the same sound model, but we wouldn’t want each to duplicate the data. Instead we need to find a way to share the model.

Also remember that each of these views has to handle the case where there is no data yet. Almost all views are instantiated only with the id of its model, so it’s quite possible that the data for that model hasn’t been loaded yet.

To solve this, we use a construct we call the instance store. This store is an object which is implicitly accessed and modified each time a constructor for a model is called. When a model is constructed for the first time, it injects itself into the store, using its id as a unique key. If the same model constructor is called with the same id, then the original instance is returned.

var s1 = new Sound({id: 123}),
    s2 = new Sound({id: 123});

s1 === s2; // true, these are the exact same object.

This works because of a surprisingly little-known feature of Javascript. If a constructor returns an object, then that is the value assigned. Therefore, if we return a reference to the instance created earlier, we get the desired behaviour. Behind the scenes, the constructor is basically doing this:

var store = {};

function Sound(attributes) {
    var id = attributes.id;

    // check if this model has already been created
    if (store[id]) {
        // if yes, return that
        return store[id];
    }
    // otherwise, store this instance
    store[id] = this;
}

This is not a particularly new pattern: it’s simply just the Factory Method Pattern wrapped up into the constructor. It could have been written as Sound.create({id: 123}), but since Javascript gives us this expressive ability, it makes sense to use it.

So, this feature means that it’s completely simple for views to share the same instance of a model without knowing anything about the other views, simply by calling the constructor with a single id. We can then use this shared instance as a localised ‘event bus’ to facilitate communication and synchronisation of the views. Usually this is in the form of listening to changes to the model’s data. If the views subscribe to the ‘change’ events which affect it, then they will be notified immediately upon change and the page can be kept up-to-date with very little effort required by the developer.

This also is how we solve the issue of there being no data on the model. On the first pass, several views might have a reference to a model which only contains an id and no other attributes. When the first view is rendered, it can detect that the view does not have enough information and so it would ask the model to fetch its data from the API. The model keeps track of this request, so that when the other views also ask it to fetch, we do nothing and avoid duplicate requests. When the data comes back from the server, the attributes of the model will be updated, causing ‘change’ events which then notify all the views.

Making full use of data

A common feature of many APIs is that when one particular resource is requested, other related resources are included in the response. For example, on SoundCloud, when you request information about a sound, included in the response is a representation of the user who created that sound.

/* api.soundcloud.com/tracks/49931 */
{
  "id": 49931,
  "title": "Hobnotropic",
  ...
  "user": {
    "id": 1433,
    "permalink": "matas",
    "username": "matas",
    "avatar_url": "http://i1.soundc..."
  }
}

Rather than let this extra data go to waste, each model is aware of which ‘sub-resources’ it can expect in its responses. These sub-resources are inserted into the instance store in case any views need to use the data. This means we can save a lot of extra trips to the API and display the views much faster.

So, for our example above, the Sound model would know that in its property “user” it has a representation of a User model. When that data is fetched, then two models are created and populated on the client side:

var sound = new Sound({id: 49931 });

sound
  .fetch()             // get the data
  .done(function () {  // and when it's done
    var user = new User({id: sound.user.id });
    user.get('username'); // 'matas' -- we already have the model data
  });

What’s important to remember is that because there’s only ever one instance of each model, even pre-existing instances are updated. Here’s the same example from above, but note when the User is created.

var sound = new Sound({id: 49931 }),
    user = new User({id: 1433 });

user.get('username'); // undefined -- we haven't fetched anything yet.
sound
  .fetch()
  .done(function () {
    user.get('username'); // 'matas' -- the previous instance is updated
  });

Letting go

Holding on to every instance of every model forever isn’t feasible, especially for the Next SoundCloud. Because of the nature of the site, it’s quite possible that a user might go for several hours without ever performing a page load. During this time, the memory consumption of the application would just continue to grow and grow. Therefore, at some point we need to flush some models out of the instance store. To decide when it is safe to do this, the instance store increments a usage counter each time an instance has been requested, and views can ‘release’ a model when it is no longer needed and the count is decremented.

Periodically, we check the store to see if there are any models with a count of zero, and they’re purged from the store, allowing the browser’s garbage collector to free up the memory. This usage count is encapsulated in the store object, but in essence it’s something like this:

var store = {},
    counts = {};

function Sound(attributes) {
  var id = attributes.id;
  if (store[id]) {
    counts[id]++;
    return store[id];
  }
  store[id] = this;
  counts[id] = 1;
}

Sound.prototype.release = function () {
  counts[this.id]--;
}

The reason for performing the cleanup on a timer, rather than whenever a usage count hits zero is so that the model stays in the store when you switch views. If you navigate to another page, there will be a single moment between cleaning up the existing views and setting up the new ones when every single model’s count is zero. The new page might actually contain views of one or more of these models, so it’d be quite wasteful to remove them instantly.

A long journey…

This has been a brief introduction into some of the methods and concepts we’re using to create Next SoundCloud, but it’s just the beginning. There are plenty more features which we have yet to build and therefore plenty more challenges to tackle. If you want join us along the way, remember that we’re always hiring!