Backstage Blog

RSS logo

You're browsing posts of the category Architecture

How (Not) to Build Datasets and Consume Data at Your Company

March 3rd, 2020 by George Roldugin

Two years ago, if an engineering team at SoundCloud needed data produced by another team, in many cases, that team would have had to start by building a dataset from the raw data. That meant there was no standard way to structure data into a dataset, there were errors in understanding and interpreting raw data, and there was no reuse of datasets across teams. Ownership of datasets was with the teams consuming data, even when the data itself was owned and best understood by someone else.

By agreeing on an ownership strategy, a common shape for datasets, and standard tooling, SoundCloud was able to streamline these problems away. Now the data and datasets are provided by those who know best about their data, along with the help of tooling built by those who know best about building tools. Read on to learn what datasets look like at SoundCloud and how they are built.

Read more…

How We Develop New Features Using Offsites and Clean Architecture

November 2nd, 2018 by Guilherme Endres

In 2017, our team of six engineers wanted to try out a clean architectural pattern and decided to use VIPER. In the text below, I’ll cover how the team worked on this.

Read more…

Keeping Counts In Sync

May 11th, 2018 by Lorand Kasler

Track play counts are essential for providing a good creator experience on the SoundCloud platform. They not only help creators keep track of their most popular songs, but they also give creators a better understanding of their fanbase and global impact. This post is a continuation of an earlier post that discussed what we do at SoundCloud to ensure creators get their play stats (along with their other stats), both reliably and in real time.

Read more…

How SoundCloud Uses HAProxy with Kubernetes for User-Facing Traffic

December 7th, 2017 by Matthias Rampke

A little less than two years ago, SoundCloud began the journey of replacing our homegrown deployment platform, Bazooka, with Kubernetes. Kubernetes automates deployment, scaling, and management of containerized applications.

Read more…

Leveraging frameworks to speed up our development on iOS - Part 1

October 16th, 2017 by Pedro Piñera

Growth in code bases come with exciting scalability challenges. As the size of our iOS codebase and team at SoundCloud grew, we faced challenges: long compile times and conflicts. Our productivity started to suffer as a result. We took inspiration from the work done in the backend (Building Products at SoundCloud) and applied it to mobile development. The main goal was to get back to a state where development is fun, fast, and would scale as the number of contributors grew. We modularized our iOS project by splitting it up into modules with well-defined responsibilities and public interfaces that interconnect them.

Read more…

A Better Model of Data Ownership

June 20th, 2017 by Joe Kearney

Once upon a time, we had a single monolith of software, one mothership running everything. At SoundCloud, the proliferation of microservices came from moving functionality out of the mothership. There are plenty of benefits to splitting up features in this way. We want the same benefits for our data as well, by defining ownership of datasets and ensuring that the right teams own the right datasets.

Read more…

Lessons in resilience at SoundCloud

December 7th, 2016 by Argha Chattopadhyay

Building and operating services distributed across a network is hard. Failures are inevitable. The way forward is having resiliency as a key part of design decisions.

This post talks about two key aspects of resiliency when doing RPC at scale - the circuit breaker pattern, and its power combined with client-side load balancing.

Read more…

Microservices and the monolith

August 26th, 2016 by Tom Stuart

In a previous series of blog posts, we covered our decision to move away from a monolithic architecture, replacing it with microservices, interacting synchronously with each other over HTTP, and asynchronously using events. In this post, we review our progress toward this goal, and talk about the conditions and strategy required to decommission our monolith.

Read more…

Synchronous communication for microservices: current status and learnings

July 27th, 2016 by Kristof Adriaenssens

Since we started breaking our monolith and introduced a microservices architecture we rely a lot on synchronous request-response style communication. In this blog post we’ll go over our current status and some of the lessons we learned.

Read more…

Building Products at SoundCloud—Part III: Microservices in Scala and Finagle

June 13th, 2014 by Phil Calçado

In the first two parts of this series, we talked about how SoundCloud started breaking away from a monolithic Ruby on Rails application into a microservices architecture. In this part we will talk a bit more about the platforms and languages in which we tend to write these microservices.

At the same time that we started the process of building systems outside the Mothership (our Rails monolith) we started breaking our large team of engineers into smaller teams that focused on one specific area…

Read more…