Apple introduced automated UI testing in Xcode 7. This was a great addition for developers because this native support promised, amongst other things, an improvement in the flakiness notoriously associated with automation tests. As many of us developers will have experienced, tests can sometimes fail even when there has been no modifications to the test or underlying feature code.
Flakiness can often come from external factors. One such source of flakiness is 'live data', e.g. data loaded from a web service API. Data from the web service often drives the state of the UI. A test might assume that the UI is in a certain state, e.g. the like button shows as ‘liked’ or that a playlist has 10 songs. Then when the data from the web service changes, the test fails.
Such occurrences are relatively common and there are many other scenarios in which these false negatives can occur, such as network connectivity issues. In these situations, the functionality of the app is not broken, just our assumptions about the data are no longer valid. What’s more, debugging the cause of these failures can be a time consuming affair.
In this blog post I'll discuss how we at SoundCloud implemented a simple system for automatically recording and replaying the network responses that occur during the course of an Xcode UI automation test in order to remove any flakiness that may be introduced from the existence of live data coming from a web service.