Why Is Random Order a Big Deal for Test Quality? 

Have you ever seen a unit test pass in isolation, but fail when run in a suite? Or vice versa, pass in a suite, but fail when run in isolation?

Besides the other improvements for test-centric workflows, Xcode 10 brought a rarity: a new XCTest feature for unit tests! Randomizing test order will help us flush out mistakes in the design of our tests.

Improve your test writing “Flow.”

Sign up to get my test-oriented code snippets.

Get Access

Let’s start by examining a symptom… Have you ever seen naming like this in a test suite?

This is a dirty trick to get the top test to run first, before any other tests in the suite. It relies on the fact that currently, tests within a suite run in lexicographic order. 01 comes early in ASCII, so test01RunThisTestFirst will run first.

Why Do We Do These Awful Hacks?

What drives us to do this in the first place? This happens because that first test performs some kind of side effect we want in place for the other tests. Unit tests don’t run in a complete vacuum. There may be shared mutable state hiding in:

• a cross-cutting concern, such as analytics
• the file system
• UserDefaults
• a local database
• a remote database

When one test runs, it may leave side effects behind. The temptation is strong to use these side effects as the starting point for the next test.

How Does Test Ordering Bite Us?

Make no mistake: adding characters to test names to force them into a particular order is a hack. One test should not influence another test. Unit tests should be FIRST, where the R stands for Repeatable. “They must not depend upon any assumed initial state, they must not leave any residue behind that would prevent them from being re-run.”

This is important because tests are not a fixed system. We add new tests. We delete tests that no longer add value. We run them individually. To avoid problems as our test suites grow and change, it’s important that the test cases inside remain independent.

Simon Whitaker knows the pain that dependencies between tests cause:

For as long as we’ve had XCTest (and OCUnit before that), tests have always been run in the same order. testA comes before testB.

Implicit test case order means that we can create dependencies between tests, and not even know it.

When I’ve encountered the problem described in Simon’s tweet, it usually hasn’t been because someone deliberately named the tests to control their order. Instead, it’s been entirely by accident. And you don’t find the problem until you run a test in isolation that you’ve always run as part of a suite. (Or, the test works in isolation so you check it in, only to have it fail, or cause another test to fail.)

How the Xcode Feature Helped Us

Xcode 10 added an important new option to XCTest: randomized test order. You can enable it by editing your scheme. Keyboard tip: Instead of mousing to the scheme selector, clicking, and selecting “Edit Scheme…” just press ⌘<

In the scheme editor, select Test in the left column. Then select the Info tab.

For each test bundle, click the Options button. Then select the “Randomize execution order” option.

This will randomize two orderings when running tests:

• Test suites
• Test cases within each suite

Let’s do this for every test bundle within every scheme!

Where Xcode Still Falls Short

Let’s assume we’ve been relying on the implicit test order without knowing it. There’s a chance that running the tests within a suite in random order will cause a hidden problem to manifest. Keep an eye open for tests that fail unexpectedly.

But then what? With careful diagnosis, we might be able to take a stab at a solution. But the next time we run the tests, they’ll be in a different order. This could mask the failure. Then how do we know if we corrected the problem? We don’t! Not unless we can rerun the tests in the same order.

RSpec gives us an example of doing this right. When you tell RSpec to run tests in random order, it prints out the seed it used for its random number generator. You can then specify that same seed to get tests in the same order.

Until we get some way to lock down the randomized order, we won’t know whether we’ve eliminated any problems this feature reveals!

Conclusion: File a Radar

The current inability to rerun tests in the same order keeps this new feature from being as useful as it could be. When Andrew Ebling and I took this problem to the Apple Labs at WWDC, the Apple engineer asked us to file a radar. And not just one. They said, the more reports they get of a given issue, the higher it goes in their list of priorities.

So… here’s my Open Radar entry. If you care about seeing this XCTest feature become truly useful, please file a duplicate! Sadly, this is still unresolved even in Xcode 13. Let’s keep banging on their door until they give us something useful.

Have you filed any Radars for other XCTest features you’d like to see? Please share below, so that we can dup your tickets!