At SoundCloud we have been building for the Android platform since 2010. Much has changed since then: the team has grown, the list of features has grown, and our audience has grown. Today, eight engineers are working full time on the official SoundCloud app, across various areas, with contributions pouring in from other parts of the organization. Due to the growing complexity and number of contributions, the app’s size has grown substantially. Currently the app consists of approximately 1200 Java source files, not counting tests, containing approximately 86000 lines of code. This doesn’t include native code, such as our playback or recording stacks.
We’re not the first to run into Android’s limits in terms of build tools. An internal limitation of Dalvik’s byte code format (DEX), which I will explain in more detail, can leave you unable to build after your codebase reaches a certain size. If you fail to anticipate this, it might happen during the most inconvenient time, such as when you are preparing for a release. Part of our job in Core Engineering at SoundCloud is to make sure our developers are happy and productive; not being able to build our app anymore makes for neither happy nor productive developers.
While there are a number of posts on this topic, I would like to describe in more detail what we have done to combat Android’s method limit, what things worked well and what didn’t work so well, what it actually means to use the dx tool’s --multi-dex switch and what you can do to improve application health with regards to size.
The article is in two parts: part one lays out the basic problem and describes a number of options that you have to stay under the method limit. Part two focuses on Android’s MultiDex mode, which you can use to grow your app beyond the method limit.
Android’s DEX format dates back to a time when applications were a lot smaller and devices were more resource constrained. To accommodate for this, the size of a DEX file’s method index is limited to 16 bits, which means if your application references more than 65,536 (2^16) methods in total, the dx tool will refuse to build your app. That number includes “Android framework methods, library methods, and methods in your own code”. 65k might sound like a lot, but when Android’s own support libraries account for about a third of that, and if on top of that you’re using one or two large libraries like Guava or Jackson, you rapidly approach that limit. To find out where you’re at, there are convenient tools to count the number of methods in your app’s DEX file.
To address a common point of confusion, the method limit described above is unrelated to “jumbo mode”, but it has a similar cause. Jumbo mode pertains to the number of strings that can be referenced in a DEX file, which by default are indexed using 16 bit wide integers. Therefore, if your application encodes more than 2^16 strings, the dx tool will fail as well. For string references however, there is a remedy: DEX supports “jumbo opcodes” which allow for 32 bit wide string references. The jumboMode flag in an Android Gradle build script enables this mode, allowing up to 2^32 strings to be referenced. However no equivalent switch exists for method references, so jumbo mode does nothing for you when your DEX file exceeds the number of methods allowed.
The following sections will outline attempts we made to address the method limit imposed on an application.
It looks like we’ve hit a wall–not being able to assemble an APK anymore is not where you want to be. There are a number of options to overcome the limit, each with drawbacks and benefits. Let’s touch on the most obvious one first: putting your app on a diet!
When we first hit the DEX method limit, we made the mistake of taking shortcuts to dodge it, such as staying on outdated but smaller versions of libraries like the Play services. Don’t do that!
Sure, old dependencies wil buy you time for the short term if you’re stuck with a broken build. However, do invest effort into making sure you don’t immediately hit the limit again. Since removing dependencies from your app isn’t something you do in an afternoon and since MultiDex sounded scary and was something we initially considered a last resort, we opted for a hack: trimming down libraries using ProGuard and use those in debug versions of the app.
If the release build is healthy in terms of method count, but the debug build isn’t, it means we’re dragging around a lot of unused stuff, so why not get rid of it in the first place? Thus was born: The Repacker!
The idea is simple: have a Gradle task that performs a debug build with a special ProGuard config that only removes, neither optimizing nor obfuscating, finds all surviving classes for a particular library we’re targeting, rolls them into a new JAR and deploys it to our artifact repository under a new version. Then change the app to depend on the trimmed down version of the library which contains only the methods it actually needs. We’ve done this successfully for Guava, Jackson and the Facebook SDK, but it works best for libraries that change infrequently.
This has several drawbacks. First, if you do this with a library like Guava, where you don’t know in advance what parts of the library you want to use, you will have to keep repacking the library as you need different parts. Second, since ProGuard sometimes accidentally removes parts that are actually in use, one might have to repack multiple times with different configurations to achieve the desired result. Lastly, we lost support for source JARs and JavaDocs, although we could have brought them back with further workarounds. While it sounded good on paper, we found this solution difficult to work with in the end, and it will not help at all if you’re exceeding the method count even after ProGuard strips away classes.
Cutting down the number of dependencies might sound obvious, but there are a few things worth pointing out.
First, we as developers are not always conscious enough of the possible repercussions of our lasting choices. Here’s an example: choosing your building blocks. Take Guava for instance. If you’re not familiar with the library, it contains a wealth of utilities that aim to close many of the gaps in Java’s language library, especially when you’re stuck with Java 6 (which you are unless you’re targeting KitKat and above). Unfortunately, the library is massive; it’s a single JAR clocking in at roughly 5MB and leaves a substantial footprint on your method allowance. It’s easy to just drop that dependency in your build script and unlock all the niceties it offers, but since it operates at a foundational level, it is bound to appear everywhere. Libraries like these are very difficult to rip out of your application later on, making these choices expensive to revisit. Moreover, you’d be surprised how much stuff a single method you use can drag on board along with it. Guava in particular is highly self-referential, so using one part of it will most likely make you unknowingly use a good chunk of its other—superficially unrelated—parts.
We didn’t have this foresight and used Guava extensively; fortunately, it was one of those 80/20 things where in the majority of our code, we’d use a minority of the library. We took two weeks to remove Guava from our code base and salvage those high-value parts that we were using frequently into a utility library that is now fully owned and maintained by us. The great thing is that in the process we shed a lot of unnecessary code, such as Guava’s workarounds for GWT and older Java versions, which are all meaningless on Android. This made the code easier to read, test, and understand. Jackson is another offender in terms of method count. Because we didn’t encapsulate it sufficiently behind APIs we own, we haven’t yet been able to replace it. In summary: before you bake something you have little or no control over into the foundations of your app, think twice. The safest decisions are those that are reversible. Especially in phases of rapid growth and uncertainty which are inherent to software start-ups.
In the next and final part, we will turn to MultiDex and what you should know about when and how to use it.