SoundCloud for Developers

Discover, connect and build

Backstage Blog RSS

You're browsing posts of the category Mobile

  • August 3rd, 2016 React Native Mobile React Native at SoundCloud By Jan Monschke & Peter Minarik

    About a year ago we faced an interesting question at SoundCloud: can we build SoundCloud Pulse — our app for creators — with React Native? Is a five-month-old technology mature enough to become part of SoundCloud’s tech stack?

    Read more...

  • March 21st, 2016 Android Mobile Open-sourcing LightCycle for Android By Guillaume Lung

    Last week, we open-sourced LightCycle, an Android library that helps break logic out of Activity and Fragment classes into small, self-contained components called LightCycles.

    Components that typically need to be aware of Activity and Fragment lifecycle events include presenters, UI tracking code, input processors and more. We’ve been using LightCycle extensively in the SoundCloud Music & Audio and SoundCloud Pulse apps over the last year.

    Read more...

  • October 6th, 2015 Android Mobile "Congratulations, you have a lot of code!" Remedying Android’s method limit - Part 2 By Matthias Käppler

    In part one we described how running into Android’s method limit may leave you unable to build, and offered strategies you can employ to make your app fit into a single DEX file. In this part we share an alternative option: using multiple DEX files.

    Read more...

  • September 21st, 2015 Android Mobile "Congratulations, you have a lot of code!" Remedying Android’s method limit - Part 1 By Matthias Käppler

    At SoundCloud we have been building for the Android platform since 2010. Much has changed since then: the team has grown, the list of features has grown, and our audience has grown. Today, eight engineers are working full time on the official SoundCloud app, across various areas, with contributions pouring in from other parts of the organization. Due to the growing complexity and number of contributions, the app’s size has grown substantially. Currently the app consists of approximately 1200 Java source files, not counting tests, containing approximately 86000 lines of code. This doesn’t include native code, such as our playback or recording stacks.

    We’re not the first to run into Android’s limits in terms of build tools. An internal limitation of Dalvik’s byte code format (DEX), which I will explain in more detail, can leave you unable to build after your codebase reaches a certain size. If you fail to anticipate this, it might happen during the most inconvenient time, such as when you are preparing for a release. Part of our job in Core Engineering at SoundCloud is to make sure our developers are happy and productive; not being able to build our app anymore makes for neither happy nor productive developers.

    While there are a number of posts on this topic, I would like to describe in more detail what we have done to combat Android’s method limit, what things worked well and what didn’t work so well, what it actually means to use the dx tool’s --multi-dex switch and what you can do to improve application health with regards to size.

    Read more...

  • September 15th, 2014 iOS Mobile Building the new SoundCloud iOS application — Part II: Waveform rendering By Richard Howell

    When we rebuilt our iOS app, the player was the core focus. The interactive waveform was at the center of the design. It was important both that the player be fast and look good.

    Initial implementation

    We iterated on the waveform view until it was as responsive as possible. The initial implementation focused on replicating the design, which heavily used CoreGraphics. A single custom view calculates the current bar offset based on its time property. It then draws each of the waveform samples that are in the current visible section of the waveform. Each sample is either rendered as a filled rectangle for unplayed samples, or by adding clip paths to the context then drawing a linear CGGradient for the played samples.

    To handle progress updates, we used the callback block on AVPlayer:

    [player addPeriodicTimeObserverForInterval:CMTimeMake(1, 60)
                                         queue:dispatch_get_main_queue()
                                    usingBlock:^(CMTime time) {
        [waveformView setPlaybackTime:CMTimeGetSeconds(time)];
    }];
    

    This generates 60 callbacks per second to try and achieve the highest possible frame rate. This solution worked perfectly during testing on the simulator. Unfortunately, testing on an iPhone 4 did not produce similar results:

    CPU rendering instruments profile

    The preceding trace displays an initial frame rate of 17 FPS that drops to 10 FPS over time. As time progresses, there are more played waveform samples, which are expensive gradient fills and take more CPU time than unplayed samples. The CPU profiler displays that 90% of CPU time is spent in the drawRect: method of our waveform view.

    Reducing waveform renders

    We needed a new approach that did not require redrawing the entire waveform 60 times a second. The waveform samples never need to change in size, but only in tint color. This makes it possible to render the waveform once, and apply it as an alpha mask to a layer filled with a gradient on the left half, and white on the right half.

    Waveform mask with background colors

    This increases the FPS to the maximum of 60 with approximately 50% GPU usage. The CPU usage dropped to approximately 10%.

    Using system animations

    We can now stop using the AVPlayer timer callback for waveform progress and use a standard UIView animation instead:

    [UIView animateWithDuration:CMTimeGetSeconds(player.currentItem.duration)
                          delay:0
                        options:UIViewAnimationOptionCurveLinear
                     animations:^{
                        self.waveformView.progress = 1;
                    }
                    completion:nil];
    

    This drops CPU usage down to approximately 0%, without noticeably increasing the GPU usage. This would seem to be an obvious win, unfortunately it introduces some visual glitches. Watching the animation on longer tracks shows an amplitude phasing effect on the waveform samples.

    Pixel-aligned waveform rendering Interpolated waveform rendering
    Waveform on pixel boundary Waveform inbetween pixels

    While the mask is animated across the waveform view, the animation will not only be drawn at integer pixel positions. When the mask is at a fractional pixel position, it causes the edges of the mask to be interpolated across each of the neighboring pixels. This reduces the alpha of the mask at the edge of each sample. This lower alpha reduces the amplitude of the visible waveform samples. The overall effect is that as the mask is translated across the view, the waveform samples animate between a sharp dark state and a lighter blurry state.

    We need a way to tell CoreAnimation to only move the mask to pixel-aligned positions. This is possible by using a keyframe animation that is set to discrete calculation mode, which does not interpolate between each keyframe.

    A high-level API was added for keyframe animations in iOS7, but the CoreAnimation interface in this case is a bit clearer:

    - (CAAnimation *)keyframeAnimationFrom:(CGFloat)start to:(CGFloat)end
    {
        CAKeyframeAnimation *animation =
            [CAKeyframeAnimation animationWithKeyPath:@"position.x"];
    
        CGFloat scale = [[UIScreen mainScreen] scale];
        CGFloat increment = copysign(1, end - start) / scale;
        NSUInteger numberOfSteps = ABS((end - start) / increment);
    
        NSMutableArray *positions =
            [NSMutableArray arrayWithCapacity:numberOfSteps];
        for (NSUInteger i = 0; i < numberOfSteps; i++) {
            [positions addObject:@(start + i * increment)];
        }
    
        animation.values = positions;
        animation.calculationMode = kCAAnimationDiscrete;
        animation.removedOnCompletion = YES;
        return animation;
    }
    

    Adding this animation to the mask layer produces the same animation, but only at pixel-aligned positions. This has a performance benefit; animating longer tracks produces animations with the FPS reduced to the maximum possible FPS for the number of animation keyframes, which reduces the device utilization accordingly. In this case, the distance of waveform animation is 680 pixels. If the played track is 10 minutes long, the FPS is 680 animation pixels over 600 seconds ≈ 1 pixel position per second. CoreAnimation schedules the animation frames intelligently, so that we only get a 1 FPS animation with correspondingly reduced 1% device utilization.

    Increasing up-front cost for cheaper animations

    The CPU performance is at an acceptable level, but 50% GPU usage is still high. Most of this is due to the high cost of masking, which requires a texture blend on the GPU. As we fetch a waveforms samples asynchronously for each track, initial render time for the view is less important than performance during playback. Instead of masking the waveform samples, they can be drawn twice: once in the unplayed state, once in the played state with gradient applied. That way, each waveform is contained in separate views that clip their subviews. To adjust the waveform position within these windows, the bounds origin can be adjusted for each subview to slide over its content, similar to how a UIScrollView works.

    Clip view hierarchy

    You can offset the bounds origin of the left clip view by its width. The result is a seamless view that looks exactly the same as the masking effect. Because we are only moving views around in the hierarchy, redrawing does not need to occur and the same keyframe animations can be applied to each of the clip views. After applying these changes, the GPU utilization drops to less than 20% for 60 FPS animation. This was considered fast enough, and is our final iteration of the waveform renderer.

    Conclusion

    Understanding the performance cost of drawing techniques on iOS can be tricky. The best way to achieve acceptable performance is to start simple, profile often, and iterate until you hit your target. We significantly improved performance by profiling and identifying the bottleneck at each step, and most of the drawing code from our initial naïve implementation survived to the final performant iteration.

    We obtained the biggest win by shifting work from the CPU to the GPU. Any CoreGraphics drawing done using drawRect: uses the CPU to fill the layers contents. This is often unavoidable, but if the content seldom changes, the layer contents can be cached and manipulated by CoreAnimation on the GPU. After the drawing reduces to manipulating UIView properties, nearly all of the work can be performed using animations, thereby reducing the amount of CPU view state updates.

    It is important to consider GPU usage, but this is harder to understand intuitively. Profiling with Instruments provides helpful insight, especially the OpenGL ES Driver template, which shows animation FPS and the percentage utilization of the GPU. The main tricks here are to reduce blending and masking, ideally using opaque layers wherever possible. The simulator option "Color Blended Layers" can be useful to identify where you have unnecessary overdraw. For more details see the Apple documentation and WWDC videos.