Connect the world of sound

Backstage Blog RSS

  • October 7th, 2013 Hacker Time Open Source Say hello to Sketchy the spam fighter By Ursula Kallio

    Sketchy the spam fighterSketchy is a spam-fighting, open-source software framework developed by SoundCloud engineers Matt Weiden, Rany Keddo, and Michael Brückner. Sketchy reduces malicious user activity on web applications. You can use it to address several common issues:

    • Detect when a user submits text that contains spam content.
    • Detect when a user submits multiple texts that are nearly identical.
    • Rate-limit malicious actions that users perform repeatedly.
    • Check user signatures such as IP addresses against external blacklist APIs.
    • Collect and consolidate reports of spam from users.

    We created Sketchy so that we could fight spam around the clock. Any startup that reaches the scale of SoundCloud has to deal with spammers on a daily basis. We decided to release Sketchy as open-source software so other companies don’t have to reinvent the wheel. Since we’ve released it, Sketchy has been featured on GigaOM.

    Get started with Sketchy by understanding its main concepts, reading additional documentation, and playing around with the Sketchy example project. As you become familiar with Sketchy, you can contribute to Sketchy via the sketchy-core repository or by emailing the Trust, Safety & Security Team.

    We’d love to know what you think and look forward to your feedback!

  • September 20th, 2013 Architecture Clojure Building Clojure Services at Scale By Duana Stanley

    SoundCloud has a service-oriented architecture, which allows us to use different languages for different services. With concurrency and scaling in mind, we started to build some services in Clojure due to its interoperability with the JVM, the availability of good quality libraries, and we just plain like it as a language.

    How do you build distributed, robust, and scalable micro-services in Clojure? Read what Joseph Wilk, an engineer and Clojure enthusiast at SoundCloud, has to say.

  • September 9th, 2013 JavaScript Writing your own Karma adapter By Misha Reyzlin

    Background

    When we started to work on the new version of our mobile web app, we knew we wanted to run unit tests on a wide variety of clients, mobile devices, PhantomJS, and on Chrome when running locally. Because we practice continuous integration, we knew we also wanted Git hooks and proper results formatting.

    We chose Karma runner, which is a project from the Angular JS team that provides developers with a “productive testing environment”. One of the advantages that Karma runner offers over other similar projects is its ability to use any testing framework. At SoundCloud, we aim to have the same toolset across various JavaScript projects, and our unit test framework of choice is Tyrtle.

    Using Tyrtle

    You can write your own Karma adapter by using the Tyrtle example that follows. The idea is to tie your tests to the Karma API. The pieces of information that you need are the number of tests, test suites or modules, the results of each test (with possible assertion or execution errors, or both), and a hook to let Karma know that the runner ran all of the tests.

    You also need to provide a start function that configures the unit test framework, loads the test files, and starts the tests.

    The basic template for an adapter is as follows:

    (function (win) {
    /**
     * Returned function is used to kick off tests
     */
    function createStartFn(karma) {
      return function () {
      };
    }
    
    /**
     * Returned function is used for logging by Karma
     */
    function createDumpFn(karma, serialize) {
      // inside you could use a custom `serialize` function
      // to modify or attach messages or hook into logging
      return function () {
        karma.info({ dump: [].slice.call(arguments) });
      };
    }
    
    win.__karma__.start = createStartFn(window.__karma__);
    win.dump = createDumpFn(win.__karma__, function (value) {
      return value;
    });
    })(window);
    

    Next, create a renderer/reporter for the unit test framework that will pass the data to Karma. Tyrtle has a renderer that can render HTML, XML for CI, or print to any other type of output.

    To pass the data to Karma, implement the methods that follow:

    /**
     * Tyrtle renderer
     * @interface
     */
    function Renderer () {}
    Renderer.prototype.beforeRun  = function (tyrtle) {};
    Renderer.prototype.afterRun   = function (tyrtle) {};
    Renderer.prototype.afterTest  = function (test, module) {};
    

    The createStartFn function creates a renderer object, with a Karma runner instance available within the start-function’s scope.

    Create a parameter named karma:

    function TyrtleKarmaRenderer (karma) {
      this.karma = karma;
    }
    

    Tell karma what the total number of tests is:

    /**
     * Invoked before all tests are run; reports complete number of tests
     * @param  {Object} tyrtle Instance of Tyrtle unit tests runner
     */
    TyrtleKarmaRenderer.prototype.beforeRun = function (tyrtle) {
      this.karma.info({
        // count number of tests in each of the modules
        total: tyrtle.modules.reduce(function(memo, currentModule) {
          return memo + currentModule.tests.length;
        }, 0)
      });
    };
    

    After each test, pass the resulting data to Karma:

    /**
     * Invoked after each test, used to provide Karma with feedback
     * for each of the tests
     * @param  {Object} test current test object
     * @param  {Object} module instance of Tyrtle module
     *                  to which this test belongs
     */
    TyrtleKarmaRenderer.prototype.afterTest = function (test, module) {
      this.karma.result({
        description: test.name,
        suite: [module.name + "#"] || [],
        success: test.status === Tyrtle.PASS,
        log: [test.statusMessage] || [],
        time: test.runTime
      });
    };
    

    Next, inform Karma that the tests have all finished running:

    /**
     * Invoked after all the tests are finished running
     * with unit tests runner as a first parameter.
     * `window.__coverage__` is provided by Karma.
     * This function notifies Karma that the unit tests runner is done.
     */
    TyrtleKarmaRenderer.prototype.afterRun = function (/* tyrtle */) {
      this.karma.complete({
        coverage: window.__coverage__
      });
    };
    

    You now have a renderer constructor. Next, turn your attention to the createStartFn function. You need to configure and initialize the unit test framework that returns a function, which requires a list of test files that are served from the Karma server and starts the actual runner.

    Karma serves the files that are required for testing from a path that Karma creates and timestamps the files to avoid caching issues in browsers. Karma makes each path available as a key in a hash named __karma__.files. This makes Karma a bit tricky to configure when using an AMD-loader such as require.js. To understand how to use AMD with Karma, go to:

    http://karma-runner.github.io/0.8/plus/RequireJS.html

    Here is the final createStartFn function:

    /**
     * Creates instance of Tyrtle to run the tests.
     *
     * Returned start function is invoked by Karma runner when Karma is
     * ready (connected with a browser and loaded all the required files)
     *
     * When invoked, the start function will AMD require the list of test
     * files (saved by Karma in window.__karma__.files) and set them
     * as test modules for Tyrtle and then invoke Tyrtle runner to kick
     * off tests.
     *
     * @param  {Object} karma Karma runner instance
     * @return {Function}     start function
     */
    function createStartFn(karma) {
      var runner = new Tyrtle({});
    
      Tyrtle.setRenderer(new TyrtleKarmaRenderer(karma));
      return function () {
        var testFiles = Object.keys(window.__karma__.files)
          .filter(function (file) {
            return (/-test\.js$/).test(file);
          })
          .map(function (testFile) {
            return testFile.replace('/base/public/', '').replace('.js', '');
          });
        require(testFiles, function (testModules) {
          // test files can return a single module, or an array of them.
          testFiles.forEach(function (testFile) {
            var testModule = require(testFile);
            if (!Array.isArray(testModule)) {
              testModule = [testModule];
            }
            testModule.forEach(function (aModule, index) {
              aModule.setAMDName(testFile, index);
              runner.module(aModule);
            });
          });
          runner.run();
        });
      };
    }
    

    To find more examples of how this all fits together, see the scripts test-main.js (the RequireJS configuration to work with Karma) and karma.conf.js. Also, there are many adapter implementations such as Mocha, NodeUnit, and QUnit on the Karma GitHub page.

    Ursula Kallio contributed to the writing of this post.

  • August 12th, 2013 Android Mobile Responsive Android applications with sane code By Duana Stanley

    A common problem in Android development is that you need to jump off the main UI thread to retrieve data from an IO-based source. At SoundCloud, we use Netflix’s RxJava to simplify asynchronous flows in Android. In an interview given for the Google Developers series “root access Berlin”,  Mustafa Sezgin—who heads up our Mobile Team—explains why we chose RxJava, what it gives us, and he walks through an example-usage pattern:

    If you are interested in more details, be sure to catch Matthias Käppler—from our Android Team—give an in-depth talk at Droidcon London, October 24-27, 2013.

  • August 9th, 2013 Contests Open Source Ruby Win a trip to the Barcelona Ruby Conference By Erik Michaels-Ober

    Barcelona Ruby Conference September MMX111

    The lineup for BaRuCo 2013 looks amazing, with speakers such as Aaron Patterson, Katrina Owen, Sandi Metz, and Ruby’s inventor Yukihiro Matsumoto. The conference is currently SOLD OUT, but we have one extra ticket… and it could be yours!

    If you win the ticket, SoundCloud will fly you from anywhere in the world to Barcelona, Spain and put you up in a nice Catalonian hotel.

    How do you enter to win?

    It’s simple. Just create a command-line interface in Ruby that uses the SoundCloud API. You can use the SoundCloud Ruby SDK, but this is not a requirement. The only constraints are that it is:

    1. a command-line interface,
    2. written in Ruby,
    3. integrated with SoundCloud,
    4. and totally awesome!

    The author of the most creative CLI that uses SoundCloud will win the BaRuCo ticket.

    To submit an entry, push your code to GitHub and send an email that includes a link to the repo to e@soundcloud.com by 12:59:59 UTC on August 26th, 2013. We will announce the winner on August 30th.

    You must be at least 18 years old to enter and allowed to travel to Spain (sorry Edward Snowden).

    Entries will be judged by a team of SoundCloud staff. If you have any questions, email me.