Apple introduced automated UI testing in Xcode 7. This was a great addition for developers because this native support promised, among other things, an improvement in the flakiness notoriously associated with automation tests. As many of us developers have experienced, tests can sometimes fail even when there has been no modification to the test or underlying feature code.
Flakiness can often come from external factors. One such source of flakiness is live data, e.g. data loaded from a web service API. Data from the web service often drives the state of the UI. A test might assume the UI is in a certain state, e.g. the like button shows as “liked” or that a playlist has 10 songs. But when the data from the web service changes, the test fails.
Such occurrences are relatively common, and there are many other scenarios, such as network connectivity issues, in which these false negatives can occur. In these situations, the functionality of the app is not broken; it’s just that our assumptions about the data are no longer valid. What’s more, debugging the cause of these failures can be a time-consuming affair.
In this blog post, I’ll discuss how we at SoundCloud implemented a simple system for automatically recording and replaying the network responses that occur during the course of an Xcode UI automation test in order to remove any flakiness that may be introduced from the existence of live data coming from a web service.