Backstage Blog

RSS logo

You're browsing posts of the category Open Source

Announcing Twinagle: Twirp and Protobuf for Finagle

June 12th, 2020 by Chris Taylor, Oren Berkowitz and Steve Conover

This article announces the release of Twinagle, an open source implementation of the Twirp protocol for Scala/Finagle.

Read more…

Periskop: Exception Monitoring Service

March 16th, 2020 by Jorge Creixell

A new approach to exception monitoring, designed for high scalability...

Read more…

Prometheus has come of age – a reflection on the development of an open-source project

July 19th, 2016 by Bjorn "Beorn" Rabenstein

On Monday this week, the Prometheus authors have released version 1.0.0 of the central component of the Prometheus monitoring and alerting system, the Prometheus server. (Other components will follow suit over the next months.) This is a major milestone for the project. Read more about it on the Prometheus blog, and check out the announcement of the CNCF, which has recently accepted Prometheus as a hosted project.

Read more…

Data pipelines with Apache Crunch and Java 8

June 1st, 2016 by David Whiting

With Java 8 now in the mainstream, Scala and Clojure are no longer the only choices to develop readable, functional code for big data technology on the JVM. In this post we see how SoundCloud is leveraging Apache Crunch and the new Crunch Lambda module to do the high-volume data processing tasks which are essential at early stages in our batch data pipeline efficiently, robustly and simply in Java 8.

Read more…

Prometheus: Monitoring at SoundCloud

January 26th, 2015 by Julius Volz, Björn Rabenstein

In previous blog posts, we discussed how SoundCloud has been moving towards a microservice architecture. Soon we had hundreds of services, with many thousand instances running and changing at the same time. With our existing monitoring set-up, mostly based on StatsD and Graphite, we ran into a number of serious limitations. What we really needed was a system with the following features:

  • A multi-dimensional data model, so that data can be sliced and diced at will, along dimensions like instance, service, endpoint, and method.
  • Operational simplicity, so that you can spin up a monitoring server where and when you want, even on your local workstation, without setting up a distributed storage backend or reconfiguring the world.
  • Scalable data collection and decentralized architecture, so that you can reliably monitor the many instances of your services, and independent teams can set up independent monitoring servers.
  • Finally, a powerful query language that leverages the data model for meaningful alerting (including easy silencing) and graphing (for dashboards and for ad-hoc exploration).

All of these features existed in various systems. However, we could not identify a system that combined them all until a colleague started an ambitious pet project in 2012 that aimed to do so. Shortly thereafter, we decided to develop it into SoundCloud’s monitoring system: Prometheus was born.

Read more…

Roshi: a CRDT system for timestamped events

May 9th, 2014 by Peter Bourgon

Let’s talk about the stream.

The SoundCloud stream represents stuff that’s relevant to you primarily via your social graph, arranged in time order, newest-first. The atom of that data model, an event, is a simple enough thing.

  • Timestamp
  • User who did the thing
  • Identifier of the thing that was done

For example,

If you followed A-Trak, you’d want to see that repost event in your stream. Easy. The difficult…

Read more…

Sponsoring CocoaPods

March 8th, 2014 by Erik Michaels-Ober

Cocoapods I’m excited to announce that SoundCloud is sponsoring the development of CocoaPods through a Travis Foundation grant. CocoaPods is an open-source dependency manager for Objective-C projects. Travis Foundation is a non-profit that pairs corporate sponsors with open-source projects to make open source even better.

At SoundCloud, our iOS team uses CocoaPods every day to manage the dependencies of our mobile apps. We hope that this sponsorship will lead to improvements that benefit the entire Mac…

Read more…

Say hello to Sketchy the spam fighter

October 7th, 2013 by Ursula Kallio

Sketchy the spam fighter Sketchy is a spam-fighting, open-source software framework developed by SoundCloud engineers Matt Weiden, Rany Keddo, and Michael Brückner. Sketchy reduces malicious user activity on web applications. You can use it to address several common issues:

  • Detect when a user submits text that contains spam content.
  • Detect when a user submits multiple texts that are nearly identical.
  • Rate-limit malicious actions that users perform repeatedly.
  • Check user signatures such as IP addresses against external blacklist APIs.
  • Collect and consolidate reports of spam from users.

Read more…

Win a trip to the Barcelona Ruby Conference

August 9th, 2013 by Erik Michaels-Ober

Barcelona Ruby Conference September MMX111

The lineup for BaRuCo 2013 looks amazing, with speakers such as Aaron Patterson, Katrina Owen, Sandi Metz, and Ruby’s inventor Yukihiro Matsumoto. The conference is currently SOLD OUT, but we have one extra ticket… and it could be yours!

If you win the ticket, SoundCloud will fly you from anywhere in the world to Barcelona, Spain and put you up in a nice Catalonian hotel.

How do you enter to win?

It’s simple. Just create a command-line interface in Ruby that uses the SoundCloud API. You can use the SoundCloud Ruby SDK, but this is not a requirement. The only…

Read more…

Evolution of SoundCloud’s Architecture

August 30th, 2012 by Sean Treadway

This is a story of how we adapted our architecture over time to accomodate growth.

Scaling is a luxury problem and surprisingly has more to do with organization than implementation. For each change we addressed the next order of magnitude of users we needed to support, starting in the thousands and now we’re designing for the hundreds of millions.  We identify our bottlenecks and addressed them as simply as possible by introducing clear integration points in our infrastructure to divide and…

Read more…