The mobile version of SoundCloud is a consumer of our own API dog food. That decision was made with the intention to deploy a self-sufficient client application that depends only on a static provider. Our early experiements showed that the attempt we made had some downsides. For example, the implementation of redirects in CORS is not behaving properly and therefore can’t be used with many of the endpoints in our API where we rely on the correct handling. Also classic XHR communication with the API is not an option due to the same origin policy implications that apply even on subdomains.
In another internal project we handled that problem with an iFrame hack for all non GET/HEAD HTTP methods. This makes sense when you’re not directly dependent on an immediate response, but it is not a feasible way to handle direct actions like commenting or favoriting. At this point we looked into different approaches to make the API accessible to the mobile app, without reimplementing any application logic from the mothership. The one we agreed on was a paradigm that we called mountable proxy. Basically, it means that we reserve a namespace in our routes that is exclusive to our API communication and can be treated like api.soundcloud.com. In our case we reserved /_api. Our frontend application is hitting that endpoint with a well-formed request that would originally go to the API subdomain. To get the correct response the proxy is applying basic rewrite rules and proxies the request to the API and returns the reponse again with some basic rewrite rules to the frontend. Thus the frontend code can be written in a way that it not has to deal with extra layers like CORS or any rewrite logic. Normal rules for XHR requests behaviors for example the handling of redirects still apply.
With Nginx being one of the most important tiers in our http stack, it was natural to lean in that direction first. Nginx serves most of our subdomains and also the static content of the mobile site. The software itself ships with a good amount of powerful modules one of it is the proxy module. It allows one to redirect and alter some of the important bits like Location header. The following snippet is what we had in production when we first went live with m.soundcloud.com:
# Only apply redirects to a specific route
location /_api/ {
# Clean the prefix
rewrite ^/_api(.*)$ $1 break;
# Override the default we want the host of the proxy here
proxy_set_header Host $proxy_host;
# the actucal redirect
proxy_pass https://api.soundcloud.com/v1/;
# Rewrite Location header
proxy_redirect https://api.soundcloud.com http://$host/_api/;
}You can find all in-depth documentation for the used directives in the proxy module wiki page.
The second iteration of the mobile app included features that are exclusive to authenticated users like the Dashboard, own Tracks, and Favorites. It also brings some more requirements like traffic over HTTPS and storage of credentials. We handle authentication against our API with OAuth2. To see in detail how it works, check out the developers documentation for authentication.
In our case we wanted to avoid exposing the client secret used to get the access token for authenticated communication with the API. For that reason we decided we want to add that secret inside of the proxy for the token endpoint. Really early in the development we started to use a server written in nodejs to provide an environment comparable to our live setup. It enabled all developers to equally contribute to the component that drives the local development. Because of that fact we looked into possible options to extend this server code to match the new requirements. The nginx part changed to use a pool of servers combined in one upstream directive and looks like this:
upstream mobile {
server xx.xx.xx.xx:xxxx weight-10 fails-5 fail_timeout-10s;
}
location / {
# Avoid double encoding for requests coming from the API
proxy_set_header Accept-Encoding "";
proxy_pass http://mobile$request_uri;
}We’re using Connect for a standard set of server functionalities and mounting the proxy into the /_api endpoint like we did before with nginx. The recently introduced pipe method on Streams helps to keep the code really small and readable. All modifications on top of that was moved to the app layer.
Through our external monitoring we observed partial downtimes after the move to the node backed cluster. Why they happened wasn’t fully clear as Nginx lacks good introspection about the up and down states of its upstream servers. As a follow-up we mirrored what we already had for our other multi-instance endpoints and put HAProxy in front of the node processes. This right away got us way better metrics and visibility. Nginx still is part of the stack as it fronts the HAProxy tier.
During that time Github released a blog post regarding how their nodeload component got optimized. One important fact is the way how nginx handles bodies and only supports HTTP/1.0. We also turned off proxy_buffering in nginx to let data being passed synchronously to the node processes.
Afer this fundamental setup change we saw a new problem. When communicating to the proxy endpoint over HTTPS and getting a 304 response from the API the socket hang up unexpectedly and HAProxy returned a 502 for this request. We synthesized the problem and isolated it to another finding from the Github blog article about the event handling of end/close on Streams and modules that inherent from Stream. As we use pipe to get the body unaltered from the response coming from the API to the response going to the client we saw that end() was called prematurely. This led to what HAProxy understands as protocol violation and we saw a lot of 502s with the reason SH, that is explained in the HAProxy documentation. This behavior is not necessarily a bug and is not yet changed in node itself. We applied a small workaround mentioned in this issue.
Even though the decision to mount the API seems like a violation of the idea to have a shippable independent client-side application, it helped to move fast and focus on the actual implementation of the authenticated areas instead of handling hacks and work around them. Both solutions — nginx and node — have worked quite well so far. With both technologies following the same paradigms in implementation they both fit well when it comes to proxy traffic with simple alterations.