Technical Interview Reform, Part 1: Rethinking the Backend Engineering Interview Take-Home Challenge

Most SoundCloud backend engineers have good feelings about the old backend engineering take-home challenge. It’s commonly been characterized as a fun, thought-provoking coding problem.

However, most of us took a significant amount of time to complete it. In fact, reports of having worked on it for 12 hours or more, over a timespan of two or more weeks, are not uncommon to hear. Candidates coming through the hiring process reported in feedback to their recruiters that they were straining under the time burden, and we clearly saw this reflected in the form of a significant dropoff at this stage in our hiring pipeline data.

This makes intuitive sense: Many great candidates have good jobs and busy personal lives. We want to talk to as many qualified candidates as possible, but to do that, we need to minimize the chances that our interview process itself gets in the way.

As a result, engineering management challenged engineers and recruiters to improve the situation. We set out to patch the obvious problems, but we ended up comprehensively rethinking the backend engineering take-home challenge.

The Tryout

The ability to compose, change, and critique programs is only one dimension of an engineer’s job — but it’s an essential one. Similar to an orchestra or acting audition, we use the take-home test to get an initial sense of how a candidate performs. We judge the performance with respect to what we believe it takes to be successful as an engineer in this organization.

The form a “code tryout” should take is a hotly debated topic, in our view, partly because contextual factors heavily constrain the format possibilities. It’s clear to us that there’s no one ideal answer that fits all engineering organizations.

The take-home challenge format has historically been a natural fit for SoundCloud for a couple of key reasons:

  • We hire engineers from all over the world, so the decision to bring a candidate onsite means there’s a significant time investment to be made by both the candidate and SoundCloud. While the onsite is certainly no guarantee of an offer, a key goal is to maximize the odds that candidates brought onsite will excel and get an offer.
  • This format presents a unique opportunity — and in the very first stage of a candidacy — to eliminate unwanted bias in evaluating candidate performance. We treat the take-home challenge as a kind of anonymous audition, and we do our best to ensure reviewers do not have any knowledge of irrelevant personal attributes of a candidate when judging submissions.

In addition:

  • We want our initial evaluation of a candidate to be based on what they can do given more ideal conditions than short onsite coding interviews allow for.1 This means having enough time to really concentrate and produce work they’re proud of. Candidates often report that this is a refreshing change when compared to industry norms.
  • SoundCloud engineers who review submissions are more confident in their ability to be fair and consistent if they are able to compare and contrast work that is technically consistent from candidate to candidate over time. In part because the programming problems are consistent, experienced reviewers can productively “train up” those who are new to the review process.

Same Coding Problem, but Faster to Complete and Review

After much discussion, we guessed we could take the same problem, change the format a little, put some modest constraints on how the solution could be solved, and still get as good of an indication of a candidate’s likelihood to succeed in the onsite interview as we did with the old challenge.

Here’s what we changed:

  • Instead of building from scratch, it’s now a refactoring challenge plus a small feature extension. It comes complete with a tester program that passes “out of the box.” The candidate’s first goal is to keep that tester passing while making the code a lot better.
  • Partly as a consequence, we offer the choice of selecting from six languages to solve the problem, with an emphasis on languages that are popular in the industry and academia. In the past, candidates could choose any language they wanted, but with a refactoring exercise, this is no longer possible.
  • We decided to restrict solutions to using only the standard library for the chosen language.
  • We revised instructions and READMEs to be more thorough and clearer in what our expectations are.

This plan was motivated by a goal to reduce the burden on the candidate. But we quickly realized that reducing the burden on reviewers is almost as important. Efficient and consistent review means we can get back to candidates with a response more quickly — but it also opens up opportunities for us to be fairer.

Each submission is evaluated by two people independently. They then meet up and file a “consensus review” signed off by both of them. But code review is a human activity, and reviewers will often end up in disagreement over a submission. When reviews are quick, it’s easy to pull in tiebreaker reviewers and still promptly get back to the candidate with a response.

Adding Pull Request Review

Inspired by Slack engineering’s pull request-based challenge, we decided to add a small PR challenge of our own to the backend engineering take-home challenge.

PR review allows us to get some insight into how candidates give critique, and the additional code problem allows us to probe a couple other technical areas. In exchange for a modest amount of extra time to complete the code challenge, candidates get an important new channel through which they can show reviewers how they approach these important aspects of the day-to-day work in engineering.

Results

Please see part 2 of this series for the recruiting take on the coding challenge reform, along with a report on both quantitative and qualitative results.

Further Reading

General

Slack Engineering

We found the series of posts on technical interviewing from Slack Engineering to be particularly helpful:

These prompted us to create our own form of a PR-based challenge.

SoundCloud History

Thank You

Much of SoundCloud engineering, recruiting, and management contributed to this in some way. Big thanks go out to:

  • Anya Voronova
  • Ben Stahl
  • Matthias Käppler

We would also like to thank Slack Engineering for their time and advice as we built our version of the PR challenge, in particular:

  • Chase Rutherford-Jenkins
  • Maude Lemaire
  • Will Kimeria

  1. You know the drill: You have 60 minutes, a web browser coding environment or whiteboard, the problem is straight out of an Algorithms and Data Structures class, and the interviewer mostly sits there in awkward silence. Some companies see such high-pressure conditions as a good thing. We don’t.