SoundCloud for Developers

Discover, connect and build

Backstage Blog RSS

  • May 9th, 2014 Announcements Go Open Source Roshi: a CRDT system for timestamped events By Peter Bourgon

    Let's talk about the stream.

    The SoundCloud stream represents stuff that's relevant to you primarily via your social graph, arranged in time order, newest-first. The atom of that data model, an event, is a simple enough thing.

    • Timestamp
    • User who did the thing
    • Identifier of the thing that was done

    For example,

    If you followed A-Trak, you'd want to see that repost event in your stream. Easy. The difficult thing about time-ordered events is scale, and there are basically two strategies for building a large-scale time-ordered event system.

    Data models

    Fan out on write means everybody gets an inbox.

    Fan out on write

    That's how it works today: we use Cassandra, and give each user a row in a column family. When A-Trak reposts Skrillex, we fan-out that event to all of A-Trak's followers, and make a bunch of inserts. Reads are fast, which is great. But writes carry perverse incentives: the more followers you have, the longer it takes to persist all of your updates. Storage requirements are also quadratic against user growth and follower count (i.e. affiliation density). And mutations, e.g. changes in the social graph, become costly or unfeasible to implement at the data layer. It works, but it's unwieldy in a lot of dimensions.

    At some point, those caveats and restrictions started affecting our ability to iterate on the stream. To keep up with product ideas, we needed to address the infrastructure. And rather than tackling each problem in isolation, we thought about changing the model.

    The alternative is fan in on read.

    Fan in on read

    When A-Trak reposts Skrillex, it's a single append to A-Trak's outbox. When users view their streams, the system will read the most recent events from the outboxes of everyone they follow, and perform a merge. Writes are fast, storage is minimal, and since streams are generated at read time, they naturally represent the present reality. (It also opens up a lot of possibilities for elegant implementations of product features and experiments.)

    Of course, reads are difficult. If you follow thousands of users, making thousands of simultaneous reads, time-sorting, merging, and cutting within a typical request-response deadline isn't trivial. As far as we know, nobody operating at our scale builds timelines via fan-in-on-read. And we presume that's due at least in part to the challenges of reads.

    Yet we saw potential here. Storage reduction was actually huge: we projected a complete fan-in-on-read data size for all users on the order of a hundred gigabytes. At that size, it's feasible to keep the data set in memory, distributed among commodity servers. The problem then becomes coördination: how do you reliably and correctly populate that data system (writes), and materialize views from up to thousands of sources by hard deadlines (reads)?

    Enter the CRDT

    If you're into so-called AP data systems, you've probably run into the term CRDT recently. CRDTs are conflict-free replicated data types: data structures for distributed systems. The tl;dr on CRDTs is that by constraining your operations to only those which are associative, commutative, and idempotent, you sidestep a lot of the complexity in distributed programming. (See: ACID 2.0 and/or CALM theorem.) That, in turn, makes it straightforward to guarantee eventual consistency in the face of failure.

    With a bit of thinking, we were able to map a fan-in-on-read stream product to a data model that could be implemented with a specific type of CRDT. We were then able to focus on performance, optimizing our reads without becoming overwhelmed by incidental complexity imposed by the consistency model.

    Roshi

    The result of our work is Roshi, a distributed storage system for time-series events. It implements what we believe is a novel CRDT set type, closely resembling a LWW-element-set with inline garbage collection. At its core, it uses the Redis ZSET sorted set to store state, and orchestrates self-repairing reads and writes on top, in a stateless operational layer. We spent a long while optimizing the read path to support our latency and QPS requirements, and we're confident that Roshi will accommodate our exponential growth for years. It took about six developer months to build, and we're in the process of rolling it out now.

    Roshi is fully open-source, and all the gory technical details are in the repository, so please do check it out. I hope it's easy to grok: at the time of writing, it's 5000 lines of Go, of which 2300 are tests. And we intend to keep the codebase lean, explicitly not adding features that are outside of the tightly defined problem domain.

    Open-sourcing our work naturally serves the immediate goal of providing usable software to the community. We hope that Roshi may be a good fit for problems in your organizations, and we look forward to collaborating with anyone who's interested in contributing. Open-sourcing also serves another, perhaps more interesting goal, which is advancing a broader discussion about software development. The obvious reaction to Roshi is to ask why we didn't implement it with an existing, proven data system like Cassandra. But we too often underestimate the costs of doing that: costs like mapping your domain to the generic language of the system, learning the subtleties of the implementation, operating it at scale, and dealing with bugs that your likely novel use cases may reveal. There are even second-degree costs: when software engineering is reduced to plumbing together generic systems, software engineers lose their sense of ownership, which is the foundation of craftsmanship and software quality.

    Given a well-defined problem, a specific solution may be far less costly than a generic version: there's a smaller domain translation, a much smaller surface area, and less operational friction. We hope that Roshi stands in evidence for the case that the practice of software engineering can be a more thoughtful and crafted process. Software that is "invented here" can, in the right circumstances, deliver outstanding business value.

    Roshi was a team effort. I'm deeply indebted to the amazing work of Tomás Senart, Björn Rabenstein, and Johan Uhle, without whom Roshi would have never been possible.

  • May 1st, 2014 Announcements JavaScript SDKs Introducing JavaScript SDK version 2 By Erik Michaels-Ober

    SoundCloud is pleased to introduce a new major version of the SoundCloud JavaScript SDK. In version 2, we've rewritten much of the internal code, resulting in better performance for your JavaScript applications and support for more streaming standards, such as HTTP Live Streaming.

    You can test the new version by pointing your JavaScript applications to http://connect.soundcloud.com/sdk-2.0.0.js.

    We've also created a guide to help you upgrade from version 1 to version 2.

    JavaScript SDK version 1 is now deprecated and will be permanently replaced by version 2 on July 1, 2014.

    On June 17, 2014, we will temporarily replace version 1 with version 2 between 10:00 and 11:00 UTC. We will do this again on June 24, 2014, between 18:00 and 19:00 UTC. These two upgrade tests will give you an opportunity to understand the impact of this change on your applications. To ensure a seamless transition for your users, we strongly encourage you to upgrade and perform internal tests in advance of these dates.

    To receive notices before, during, and after these tests, follow @SoundCloudDev on Twitter.

    If you have any questions about this upgrade, please feel free to email api-support@soundcloud.com.

  • April 27th, 2014 Contests Search and Discovery Irrational Fun: Find Yourself at Berlin Buzzwords By Erik Michaels-Ober

    We were counting down the days until Berlin Buzzwords on May 25, when we realised that it would be great if you came too! With that in mind, we've created a contest. One lucky winner will receive a free ticket to Berlin Buzzwords, including travel expenses and accommodation. Here are the details about how to apply.


    The ratio of a circle's circumference to its diameter, represented by the Greek letter π, is an irrational number—it never terminates or repeats. Your goal is to find the SoundCloud logo in π.

    We have provided a 14 pixel by 6 pixel, greyscale reference image:

    Here is the same image at 60X magnification:

    Each of the 10 shades of grey in this image can be mapped to a number:

    Color RGB Hex Number
      ffffff 0
      f0f0f0 1
      ebebeb 2
      d0d0d0 3
      c1c1c1 4
      a8a8a8 5
      878787 6
      535353 7
      333333 8
      000000 9

    Applying this mapping to the reference image produces the following 84-digit bitmap:

    0 0 0 0 0 3 3 9 9 9 6 0 0 0
    0 0 0 3 4 9 5 9 9 9 9 4 0 0
    0 1 2 7 5 9 5 9 9 9 9 7 1 0
    3 9 5 9 5 9 5 9 9 9 9 9 9 3
    6 9 5 9 5 9 5 9 9 9 9 9 9 6
    1 8 5 9 5 9 5 9 9 9 9 9 8 1
    

    Your challenge is to write an algorithm that finds the 10 sequences that most closely approximate the reference image. Each result should include the sequence and its position (after the decimal point) in π.

    Here is an example result set:

    Rank Image Sequence Offset
    1 082201638940102393659252475011295776958920 282336494898427768768699465405437965994582 297,640,119
    2 310015049916890341096198545241549627773525 291969856827158758552799587406476458977970 792,987,187
    3 212021479960265330123798820231693768599316 147634474729776147987653958935291919768971 972,165,010
    4 000053743536020032496985553181128909983810 344939134894807349584729687746183109884672 981,165,566
    5 142204297650171312983445842322141909755200 787408739757838589593329762648444919386594 789,652,974
    6 300011495970664010077917573663456957498854 662995598898697947677549686339433357728071 197,342,990
    7 313560984264300011495970664010077917573663 456957498854662995598898697947677549686339 197,342,978
    8 870402479996214234001557832923050859979903 649788689695954439755933903629798966788984 75,975,342
    9 208503759370245135490877462200175839969750 453766432680245845143285985661373828688970 343,577,393
    10 300544295771128716836756973814286978997516 282647269986574856578306678421894769876141 950,462,734

    Entries will be judged against the follow criteria:

    • Code quality
    • Runtime performance
    • Visual closeness to the reference image (subjective)


    You should run your algorithm against the following data set (approximately 1 GB), which contains the first 1,000,000,000 (billion) digits of π.

    Please send your submission, including a link to the source-code repository, to e@soundcloud.com on or before May 5, 2014 23:59 UTC. Your repository should contain a README that includes instructions about how to set up and run your code. Entries are subject to the terms and conditions.


    UPDATE

    Congratulations to Tomasz Pewiński, who submitted the winning entry.

    We’d also like to acknowledge entries by Dan Oved and Martin Kühl, which were also excellent. Thanks to everyone else who participated in the contest. We hope it was fun!

  • April 11th, 2014 Announcements Welcome to SoundCloud's redesigned developer site By Erik Michaels-Ober

    We've taken some time to bring all our developer resources together into a single site. In doing so, we've reorganized the layout to make things easier to find and also given the site a fresh new look.

    We hope you like it!

    If you have any feedback about the new design, follow @SoundCloudDev on Twitter and let us know.

  • April 11th, 2014 Announcements Security Security update: Heartbleed vulnerability in OpenSSL By Astera Schneeweisz

    HeartbleedOn Monday, April 7th, 2014, a major security vulnerability in OpenSSL was made public. The vulnerability was filed as CVE-2014-0160 and later dubbed “Heartbleed”, because the bug lies within OpenSSL’s heartbeat extension, which is used for keepalive monitoring. As a result of the bug, process memory can be read out remotely by an attacker—potentially including certificates, keys, credentials, tokens, or other sensitive data processed by the server.

    OpenSSL works as a cryptographic library that allows for authenticity and confidentiality across the entire Internet. Because the reported Heartbleed bug affects a vast number of internet services using OpenSSL to secure their services (such as HTTPS, SMTP, IMAPS, and POP3), a patched OpenSSL version was released by the maintainers within hours. Linux and UNIX distributions, which depend on the OpenSSL implementation, received patches by their respective upstream maintainers.

    SoundCloud too uses OpenSSL in many of our services to increase the security and privacy of our users. We therefore moved quickly to patch the vulnerability, and did so within hours of the patch being made available. We’ve also been in close communications with our vendors and service providers, to ensure that they have applied the appropriate fixes as well. We have confirmed that our implementations of OpenSSL are no longer vulnerable to this bug.

    Because we consider our users’ security and privacy of the utmost importance, we have further taken the precautionary measures to rotate SSL certificates and keys, and expire authentication tokens, such as session cookies, remember tokens, and OAuth access and refresh tokens. This means that users will be signed out of their SoundCloud accounts. Along with top security researchers and responsible companies, we have also recommended to our users that they change their passwords on all accounts (not just SoundCloud) that they have signed-in with in the past week. Developers of API clients that check our SSL fingerprints will need to update them.

    While the Heartbleed bug marks a sad day for the Internet as a whole, with SoundCloud’s rollout of Perfect Forward Secrecy (PFS) support last year, we ensured that the impact of an attack with the purpose of stealing private keys and reading previously encrypted traffic is minimized. In the same spirit we will also strive to find more such opportunities in the future and preemptively provide our users with the highest possible level of safety.

    For more details about this bug, go to heartbleed.com. To use tools to check your services, go to Filippo.io or FiloSottile/Heartbleed on GitHub.