Please also see Part 1: Rethinking the Backend Engineering Interview Take-Home Challenge
Among the engineering groups at SoundCloud, backend engineering has always had the highest number of open roles — at times reaching 10 positions. To fulfill headcount plans in the current fast-paced market, recruitment needs to move quickly and put efficient processes in place.
At SoundCloud, we have designed a three-step process to enable us to move candidates through the pipeline quickly: recruitment screening, challenge, and final onsite round. Each step has to provide a clear signal as to if the candidate is a good match for the role we’re trying to fill.
However, in the past, we’d experienced challenges with high dropout and time-to-hire rates, as well as with the low ratio of challenges passed to offers made. Having analyzed the data, we identified the code challenge step as the bottleneck from the recruitment perspective.
According to the candidate feedback, it took anywhere from 10 to 20 hours to complete the backend code challenge task. The turnaround between a candidate receiving the test and sending it back was three weeks on average, and it took around a week for engineers to review a submission.
This is a big time investment to ask from candidates, and it is hard for them to commit to such a time-consuming task while having a full-time job. It is even less inclusive of people with families and kids or more junior candidates. As a result, the dropout rates during the challenge steps were high. All these factors contributed to the lengthy process and not always the best candidate experience.
On a positive note, people who invested the time to complete the challenge were definitely motivated to join SoundCloud. :)
To initiate the change in the process, the recruitment team worked closely with the members of the leadership and engineering teams. We analyzed the data, proposed a possible solution, and issued an internal RFC (“request for comments”). Recruitment and engineering collaborated on developing an efficient process and constantly iterating based on the candidate feedback.
To onboard challenge reviewers and interviewers, we introduced challenge review squads. A squad consists of more experienced and less experienced reviewers who pair and learn from each other. The squads change every 6–8 weeks so that everyone can participate and get onboarded to become reviewers of the new challenge.
In addition, engineers are able to better manage their time with this process in place: they commit for 6–8 weeks, and when that time has passed, they can be free of the challenge reviews for the rest of the year if they want to.
The new backend challenge was implemented in October 2019, and the candidate feedback has been great: The time commitment is now 5–6 hours. The new PR review tasks received a positive response as well: Candidates often find this part of the challenge interesting, and they consider it a refreshing change from the norms they encounter in technical interviewing.
The turnaround time between the candidate receiving the challenge and sending back the solution went down to an average of seven days (a decrease of 60 percent). The new challenge also provides stronger signals: Conversion rates (challenge completed/offer made) increased from 35 percent to 50 percent.
As an interesting side effect, the new backend code challenge positively contributed to the company’s focus on diversity in hiring: Since implementing it in October 2019, 30 percent of the offers we’ve made have been to non-male engineers.
After seeing such significant positive results of the new backend challenge process, we are now working on making similar changes in other engineering coding interviews.
An upcoming bonus post will outline changes we’ve had to make and challenges we’ve faced adjusting our technical interviewing process in the COVID-19 era.