Waveforms, Let's Talk About Them

I’ve worked at SoundCloud for over two years now, and if there’s one thing I do a lot, it’s color waveforms. Tons of them. And, I’ve done it several different ways. Today, Johannes and I are pumped to announce a new JavaScript library called Waveform.js that will assist you in your coloring efforts. But first, let’s take some time to look back and learn from past techniques.

Element Stacking

If you’ve used our API before, you’ll know that each track resource contains a waveform_url that points to a 1800 x 280 pixel PNG of that particular sound. The classic way of coloring this image was to not color it at all. Instead, you would simply stack it on top of two or three absolutely positioned DIVs whose size was adjusted given the loading or playing state of that sound.

So, what’s wrong with this technique? Well, you can’t adjust the outer color of the waveform image. So unless the area around the waveform is #E5E5E5, things are going to look subpar.

Webkit Mask

I saw the future, and it was Webkit

Webkit browsers can overcome the outer color issue with a new css function called webkit-mask-box-image. This function colors a div to the shape of a provided mask. In this case, our waveform image. Check out the example below:

.waveform {
    -webkit-mask-box-image: url(https://w1.sndcdn.com/cWHNerOLlkUq_m.png);
    background: #81D8D0;
    height: 100px;
    width: 500px;
}

<div class="waveform"</div>

It’s simple to code and damn powerful. You can combine this function with the element stacking technique above to generate nice waveforms in most environments. But what about non-webkit browsers? HTML 5 to the rescue!

Canvas

Canvas adoption is rapidly increasing and for good reason - it fucking rules. With canvas we can dynamically manipulate an area of pixels with basic JavaScript. By placing the waveform image mask onto the canvas and looping through each pixel, we can adjust the outer color accordingly by simply asking “Is this pixel non-transparent?” and changing it to a specified color.

However, there is one caveat: the Cross-Domain Origin Policy. If you’ve developed with canvas before, you know that accessing and manipulating images from different domains or origins is prohibited. [1] But, we can get around this by accessing the image via the server, and I wrote a service that does exactly that.

Wave64 exists as an endpoint you pass a SoundCloud waveform_url to from the client. It will then read that image on the server and return a Base64 encoded text blob [2] which can be manipulated by canvas. I combined this endpoint with some jQuery and wrote the $.wave64 plugin which takes a waveform url, height, width, and color and returns the colored waveform.

Problem solved? Never. Programming is a math problem you can only make harder on yourself, and as such Johannes and I sought out to simplify this even more.

Waveform.js

Damn Good Waveforms - Eric Wahlforss, CTO of SoundCloud

Johannes had the bright idea to analyze the waveform image’s peaks and store their values in an array of floating points. HUH..? Take a look at the image below:

If every column was considered a point, it’s value would be a floating decimal from 0 to 1 depending on it’s height. So, we wrote a basic server-side script using RMagick to analyze the columns, convert them into 1800 floating points, and cache the result.

Don’t believe me? Here’s Forss - Flickermood.

In addition, Johannes also wrote the new Waveform.js JavaScript library to convert this data into a canvas generated (and interpolated) waveform. Simply provide the container where it will live, an innerColor &&|| outerColor, and the provided waveform data array. Like so:

new Waveform({
  container: document.getElementById("waveform"),
  innerColor: "#81D8D0",
  data: WAVEFORM_DATA,
})

Psyched? Yeh, me too. And, not only can you pass colors to the inner and outer color, you can also provide functions. Check out a few examples of this technique on waveformjs.org, including the ability to render both the loading and playing progress right into a single Waveform.

Waveform.js is open-sourced on Github so go ahead and push, pull, and fork the hell out of it. You can report bugs and discuss features on the Github issues page.

The End?

Probably not. It would rock if the SoundCloud API provided the waveform array data right from the API so we wouldn’t even need the endpoint. I’ll keep complaining internally until this happens.

One More Thing

Johannes and I would like to thank our fellow co-workers Tomas and Robb for their wisdom when building Waveform.js. Make sure to throw some virtual high-fives their way.

[1] However, there are a few workarounds.

[2] Thanks to Max Novakovic for the technique.