Most SoundCloud backend engineers have good feelings about the old backend engineering take-home challenge. It’s commonly been characterized as a fun, thought-provoking coding problem.
However, most of us took a significant amount of time to complete it. In fact, reports of having worked on it for 12 hours or more, over a timespan of two or more weeks, are not uncommon to hear. Candidates coming through the hiring process reported in feedback to their recruiters that they were straining under the time burden, and we clearly saw this reflected in the form of a significant dropoff at this stage in our hiring pipeline data.
This makes intuitive sense: Many great candidates have good jobs and busy personal lives. We want to talk to as many qualified candidates as possible, but to do that, we need to minimize the chances that our interview process itself gets in the way.
As a result, engineering management challenged engineers and recruiters to improve the situation. We set out to patch the obvious problems, but we ended up comprehensively rethinking the backend engineering take-home challenge.
The ability to compose, change, and critique programs is only one dimension of an engineer’s job — but it’s an essential one. Similar to an orchestra or acting audition, we use the take-home test to get an initial sense of how a candidate performs. We judge the performance with respect to what we believe it takes to be successful as an engineer in this organization.
The form a “code tryout” should take is a hotly debated topic, in our view, partly because contextual factors heavily constrain the format possibilities. It’s clear to us that there’s no one ideal answer that fits all engineering organizations.
The take-home challenge format has historically been a natural fit for SoundCloud for a couple of key reasons:
In addition:
After much discussion, we guessed we could take the same problem, change the format a little, put some modest constraints on how the solution could be solved, and still get as good of an indication of a candidate’s likelihood to succeed in the onsite interview as we did with the old challenge.
Here’s what we changed:
This plan was motivated by a goal to reduce the burden on the candidate. But we quickly realized that reducing the burden on reviewers is almost as important. Efficient and consistent review means we can get back to candidates with a response more quickly — but it also opens up opportunities for us to be fairer.
Each submission is evaluated by two people independently. They then meet up and file a “consensus review” signed off by both of them. But code review is a human activity, and reviewers will often end up in disagreement over a submission. When reviews are quick, it’s easy to pull in tiebreaker reviewers and still promptly get back to the candidate with a response.
Inspired by Slack engineering’s pull request-based challenge, we decided to add a small PR challenge of our own to the backend engineering take-home challenge.
PR review allows us to get some insight into how candidates give critique, and the additional code problem allows us to probe a couple other technical areas. In exchange for a modest amount of extra time to complete the code challenge, candidates get an important new channel through which they can show reviewers how they approach these important aspects of the day-to-day work in engineering.
Please see part 2 of this series for the recruiting take on the coding challenge reform, along with a report on both quantitative and qualitative results.
We found the series of posts on technical interviewing from Slack Engineering to be particularly helpful:
These prompted us to create our own form of a PR-based challenge.
Much of SoundCloud engineering, recruiting, and management contributed to this in some way. Big thanks go out to:
We would also like to thank Slack Engineering for their time and advice as we built our version of the PR challenge, in particular:
You know the drill: You have 60 minutes, a web browser coding environment or whiteboard, the problem is straight out of an Algorithms and Data Structures class, and the interviewer mostly sits there in awkward silence. Some companies see such high-pressure conditions as a good thing. We don’t.
↩