Inside a SoundCloud Microservice

If you’re a regular visitor to this blog, you might be aware that we have been transitioning to a microservices based architecture over the past four to five years, as we have shared insights into the process and the related challenges on multiple occasions. To recap, adopting a microservices architecture has allowed us to regain team autonomy by breaking up our monolithic backend into dozens of decoupled services, each encapsulating a well defined portion of our product domain. Every service is built and deployed individually, communicating with other services over the network via light-weight data interchange formats such as JSON or Thrift. What we haven’t touched on so far is how a microservice at SoundCloud looks backstage.

The answer is: it depends! A necessary property of a microservices driven architecture is that a microservice represents a unit of encapsulation. As long as there is some well-understood contract in the form of an API that other services can consume, anything else becomes an implementation detail. Consequently, moving between code bases may reveal different languages, stacks, and patterns, and whether this is a blessing or curse is a hotly debated topic for someone else to explore.

What’s covered in this article is written from the perspective of a single team—the Creators Team—and as such highlights some of the programming patterns that we unilaterally settled on.

Your Server as a Function

With that out of the way, it might be interesting to know what kinds of services our team owns. Since we’re responsible for anything powering the user-facing products that serve our creators (i.e. musicians and podcasters rather than listeners) the services we build and operate are quite varied: track management, messaging, RSS feed syndication, statistics and APIs powering our mobile app (SoundCloud Pulse) are just some examples. All our services are written in Scala and built on top of Finagle with the help of a few light-weight, shared internal libraries adding support for cross-cutting concerns such as configuration injection, telemetry, and session termination.

Microservices are often intermediate nodes in a graph of services, acting as façades where an incoming request translates to N outgoing requests upstream, the responses to which are then combined into a single response back downstream to the client. In the fictional scenario illustrated below, ServiceA needs to reach out to and collect responses from ServiceB and ServiceC, which in turn might have more dependencies such as other services, databases, and caches.

One solution to this “scatter-gather” problem based on functional programming was pioneered by Twitter and presented in the very approachable paper Your Server as a Function, which outlines the ideas that eventually materialized in Finagle. Finagle models each node as a pure function from some request to some eventual response:

class Service[-Req, +Rep] extends (Req => Future[Rep]) 

While Finagle does an excellent job at abstracting away the tedious details behind communication protocols, connection management, and concurrency, it does leave us with certain complexities when working with this abstraction. Specifically, we will explore solutions to the following problems:

• How to collect results from multiple service calls into a single data structure
• How to explicitly model failures in a service’s logic layer using Scala’s Either type
• How to bring the above together in a convenient way using monad transformers

Parallel Futures

The service signature above tells us that we’re dealing with effectful functions: they return Futures, not the things we’re actually after. This means we will have to lean on different combinators to extract the desired result. Imagine we want to fetch a track and all user comments it received into some hypothetical TrackWithComments object:

for {
  track <- fetchTrack(42)
  comments <- fetchTrackComments(42)
} yield TrackWithComments(track, comments)

Scala will desugar for-comprehensions into a combination of flatMap and map calls, which means that comments will be fetched only after we have retrieved the track. However, these two calls do not depend on each other, so we could run them in parallel instead. There are documented solutions to the problem of executing Scala futures in parallel in for-comprehensions, but they are non-intuitive and do not clearly express our intent.

Twitter’s Future implementation gives us a powerful combinator, join, to solve this problem both more elegantly and explicitly:

val res: Future[TrackWithComments] = for {
  (track, comments) <- Future.join(
    fetchTrack(42),
    fetchTrackComments(42)
   )
} yield TrackWithComments(track, comments)

These instructions explicitly encode our intent to obtain a pair of independent values: given (Future[A], Future[B]) we want Future[(A, B)]. Moreover, if either of the original futures fail, the result is a failed Future, which allows us to propagate errors as return values instead of throwing exceptions. Looking at this, you might be reminded of database joins: given two tables, the SQL join operator will take the cartesian product of their rows and return a new table where rows are tuples containing data from both sources.

In distributed systems where data crosses network boundaries frequently, failure is expected and should be embraced. We will now explore how we represent failure as a first class concept in our systems, without error handling ever getting in the way of productivity. We accomplish this by using Scala’s Either class, a return type that allows us to focus on the “happy path”.

Down The Future Stack

Futures are internally represented by the Try structure. Try[A] is a union type, which can be either successful (carrying a value of A) or failed (carrying an exception.) Exceptions are well suited to dealing with operational issues such as I/O failures, runtime platform errors, and unmet system expectations, but when it comes to representing failures in our problem domain, we avoid using exceptions. This is because logical failures in an application are best represented by proper values, and their presence should be made explicit in the type system instead of disappearing from plain view in unchecked exceptions. The upshot is that any service call in the logic layer of our systems returns a type that can either be a success or a failure:

def fetchTrack(id: Long): Future[Either[ApplicationError, Track]] = ???

where ApplicationError is a union type of possible things that can go wrong. For instance, we can define errors like the following:

sealed trait ApplicationError
case object NotFound extends ApplicationError
case object AccessDenied extends ApplicationError
...

Using Either as the return type instead of a Track directly states that any call to fetchTrack will return either a valid track, or a “not found” error, or an “access denied” error, but not all, meaning all possible outcomes are explicit in the function type, and must be dealt with accordingly. Future is already a functor so it comes with the map combinator defined, which allows us to focus on the happy path and “dive into” a successful Future:

// only executes the lambda if the Future was successful
fetchTrack(42) map { trackOrError => ??? } 

However, fetchTrack now returns an Either value, not a track. We would ideally like to map trackOrError again to continue following the happy path, but indeed this only works as of Scala 2.12. In Scala 2.11 or older (which is what our systems are deployed with), Either is not “biased” toward either of the outcomes it captures (in our case an ApplicationError or Track), so no functor instance is defined that allows us to focus on the happy path. Fortunately, we found a solution in Typelevel’s Cats library, which compresses an enormous amount of utility into a small library. Let’s check it out!

Catnip’ed

Cats is not a reference to the furry animal but is short for “categories”, a nod to the branch of mathematics that studies structure in mathematics itself. It turns out that category theory is concerned with abstractions that are extremely useful when working with strongly typed languages. Specifically, category theory provides us with “blueprints” or “recipes” for the traversal and transformation of types in ways that are provably correct. One such recipe is encoded in the Functor type class, which allows us to turn any type constructor into a functor, given some rules are obeyed. Cats already provides a functor instance for Scala’s Either type, so we can now map over Eithers:

def fetchTrack(id: Long): Future[Either[ApplicationError, Track]] = ???

fetchTrack(42) map { trackOrError => trackOrError map { track => ??? } }

The drawback of nesting effects is that we now have two levels of machinery: Future and Either, which have to be dealt with separately. This makes it cumbersome to get our hands on a track. Ideally we want a functor instance which folds both into one: if the Future succeeds, and the Either contains a “happy outcome”, then map the result. It turns out that Cats provides precisely such an abstraction: it’s called a monad transformer, and for Either it’s called EitherT. In our case specifically, since we’re always dealing with Futures and ApplicationErrors, we decided to glue these together into a compact internal library written on top of EitherT to model results of any computation in our systems. We call it Outcome.

The Final Outcome

Outcome is a set of type aliases and type converters that bind together Future and a well thought out ApplicationError hierarchy (with broad applicability to most or all our services) using Cats’ EitherT type:

type OutcomeF[A] = EitherT[Future, ApplicationError, A]

This definition allows us to specify return types in a compact fashion while maintaining our focus on the happy path:

def fetchTrack(trackId: Long): OutcomeF[Track] = ???

fetchTrack(42) map { track => track.title } // works!

We can also recover from potential errors:

fetchTrack(42) map { track => track.title } recover { 
  case NotFound => "(unknown)"
}

In fact, we have recover clauses like these in all our request handlers, so we can easily map a NotFound error to an HTTP 404 response code. But that’s not all. You may recall from above how Future.join can flip futures inside out. Using the EitherT transformer, we can do the same for outcomes:

val trackWithComments: OutcomeF[TrackWithComments] = 
        (fetchTrack(42) |@| fetchTrackComments(42)).map(TrackWithComments)

Here, |@| is the product combinator of Cats’ Cartesian type which joins outcomes into a tuple of values without stepping out of the “outcome context”. We can then conveniently map this tuple onto a properly named data structure.

Summary

We started out by exploring what makes writing microservices challenging. The frequent crossing of network boundaries requires us to deal with asynchronous, error prone operations. We furthermore pointed out that we draw a sharp distinction between operational errors and those arising in our product domain. To deal with the latter, we used Scala’s Either type, leaving us with more explicit but verbose service function calls. Finally we revealed how Cats can restore compactness of service function calls via EitherT and Cartesian, and how we capture their functionality in a convenient type alias: OutcomeF. We barely scratched the surface of Cats in this article, it provides a wealth of utility when opting for a more functional style of programming. However, we hope we could give you a glimpse at the patterns you would find in a typical Scala code base in SoundCloud’s Creators Team and hope they may turn out as useful for you as they did for us.

Until next time!

Matthias and the Creators Team