Backstage Blog

• Android
• Mobile

At SoundCloud we have been building for the Android platform since 2010. Much has changed since then: the team has grown, the list of features has grown, and our audience has grown. Today, eight engineers are working full time on the official SoundCloud app, across various areas, with contributions pouring in from other parts of the organization. Due to the growing complexity and number of contributions, the app’s size has grown substantially. Currently the app consists of approximately 1200 Java source files, not counting tests, containing approximately 86000 lines of code. This doesn’t include native code, such as our playback or recording stacks.

We’re not the first to run into Android’s limits in terms of build tools. An internal limitation of Dalvik’s byte code format (DEX), which I will explain in more detail, can leave you unable to build after your codebase reaches a certain size. If you fail to anticipate this, it might happen during the most inconvenient time, such as when you are preparing for a release. Part of our job in Core Engineering at SoundCloud is to make sure our developers are happy and productive; not being able to build our app anymore makes for neither happy nor productive developers.

While there are a number of posts on this topic, I would like to describe in more detail what we have done to combat Android’s method limit, what things worked well and what didn’t work so well, what it actually means to use the dx tool’s --multi-dex switch and what you can do to improve application health with regards to size.

• Announcements
• API

At SoundCloud, we’re building an ecosystem where creativity thrives. Developers are an important part of that ecosystem. We’re continually inspired by how you use the SoundCloud API to support creators and listeners in innovative ways.

But as the ecosystem has grown, we’re dealing with an increasing number of applications that abuse creator content by violating our developer Terms of Use. To help control this type of behaviour, we’re introducing a daily rate limit on API play requests.

Beginning…

• Announcements
• API

Requests for playlists have always included the full track objects contained within. This representation may be convenient for playlists with ten or twenty tracks but can cause problems for playlists that contain hundreds or thousands of tracks. Requesting such large playlists could result in requests that take a long time to respond and that eventually timeout.

Today, we introduce two new representations for the /playlists resource: compact and id.

If you add representation=compact to a playlist request, the request will return only…

• Announcements
• API

In October 2014, Apple announced that all submissions to the App Store must include 64-bit support by June 1, 2015. The SoundCloud API for Cocoa contains 32-bit dependencies and will not be updated, because it has been discontinued. Anyone using the SoundCloud API for Cocoa will need to will need to migrate away from it if they wish to update their app after June 1.

To ease this transition we have built a sample app that demonstrates how to authorize a user via OAuth using only built-in…

• Announcements
• API

The SoundCloud API will be dropping support for offset-based pagination on March 2, 2015, in favor of linked partitioning.

To page through a JSON response, pass the linked_partitioning=1 parameter along with your request and it will return a collection, along with a next_href property if there are additional results. To fetch the next page of results, simply follow that URI. If the response does not contain a next_href property, you have reached the end of the results.

You can read more about linked partitioning in the Pagination section of our HTTP API Guide…

• Announcements
• Open Source
• Monitoring
• Go

In previous blog posts, we discussed how SoundCloud has been moving towards a microservice architecture. Soon we had hundreds of services, with many thousand instances running and changing at the same time. With our existing monitoring set-up, mostly based on StatsD and Graphite, we ran into a number of serious limitations. What we really needed was a system with the following features:

• A multi-dimensional data model, so that data can be sliced and diced at will, along dimensions like instance, service, endpoint, and method.
• Operational simplicity, so that you can spin up a monitoring server where and when you want, even on your local workstation, without setting up a distributed storage backend or reconfiguring the world.
• Scalable data collection and decentralized architecture, so that you can reliably monitor the many instances of your services, and independent teams can set up independent monitoring servers.
• Finally, a powerful query language that leverages the data model for meaningful alerting (including easy silencing) and graphing (for dashboards and for ad-hoc exploration).

All of these features existed in various systems. However, we could not identify a system that combined them all until a colleague started an ambitious pet project in 2012 that aimed to do so. Shortly thereafter, we decided to develop it into SoundCloud’s monitoring system: Prometheus was born.

• Scalding
• Hadoop

Recently we teamed up with Concurrent Inc., the backers of the data-processing framework Cascading, to do a case study of how we use Scalding for some of our data-driven products such as Search. Scalding enables us to iterate quickly, test easily, and it allows for loose coupling of some of our data-processing pipelines.

Check back for future posts about our use of other data-processing tools, and frameworks such as Spark.

• Announcements
• API

The SoundCloud API will be dropping support for Extensible Markup Language (XML) responses. XML will be phased out on the following schedule:

1. XML is currently the default response format for requests without an explicit format specified in the path (e.g. /tracks) or Accept header. This default will be changed to JSON on December 1, 2014.
2. Explicit requests for XML — specified either in the path (e.g. /tracks.xml) or an Accept: application/xml header — will continue to be supported until December 15, 2014. After that point, only JSON responses will be supported.
…

• iOS
• Mobile

When we rebuilt our iOS app, the player was the core focus. The interactive waveform was at the center of the design. It was important both that the player be fast and look good.

Initial implementation

We iterated on the waveform view until it was as responsive as possible. The initial implementation focused on replicating the design, which heavily used CoreGraphics. A single custom view calculates the current bar offset based on its time property. It then draws each of the waveform samples that are in the current visible section of the waveform. Each sample is either rendered as a filled rectangle for unplayed samples, or by adding clip paths to the context then drawing a linear CGGradient…

• iOS
• Mobile

Recently, SoundCloud launched the new iOS application which was a complete rewrite of the existing iOS application. The Mobile engineering team saw this as an opportunity to build a solid foundation for the future of SoundCloud on iOS and to experiment with new technologies and processes at the same time.

In the world of mobile, you deal with data, errors, threads and concurrency a lot. The common scenario starts with a user tapping on the screen. The application jumps off of the main UI thread…