Are Slow Tests Killing Your Feedback Loop? 

Hit ⌘U to run your tests. …Once the tests start executing, how long do you have to wait for the results?

If you waited more than a few seconds, you may have a problem. Because one sure-fire way to discourage test-driven development on your team is to have unit tests take 30 seconds or more.

When I was taking water safety training, one of the things we had to do was jump into the water fully clothed. You don’t normally notice the feel of your own clothing, so it’s surprising to feel how waterlogged clothes hinder every basic swimming movement. The training is to learn how to remove that clothing in the water, to restore normal swimming. It’s like shedding a heavy straitjacket; suddenly you can move again!

Many of us are tied down by our slow testing experiences. Slow tests mean it takes longer to get results. Taking longer to get results means we won’t run our tests as often. Not running our tests as often means we’re undercutting the core benefit of test-driven development—namely, getting feedback frequently.

So let’s restore normal movement by shedding those heavy tests and setting them aside. But which tests are “heavy”? And what do we do with them?

What Is a Slow Test?

Disclosure: The book links below are affiliate links. If you buy anything, I earn a commission, at no extra cost to you.

In his excellent book Working Effectively with Legacy Code, Michael Feathers lists various characteristics of tests that can make them unsuitable. But he summarizes it all in a quick definition:

A unit test that takes 1/10th of a second to run is a slow unit test.

As Michael says: “Yes, I’m serious.” Because before you know it, you’ll have 10,000 tests. If they all took a tenth of a second, we’re talking about sitting around for 15 minutes just waiting for that feedback.

How Do You Identify Slow Tests?

So how do we find these slow tests? Combing through Xcode’s logs takes too long. I think there’s an open need for a script that will filter the test output to show the slow tests. But until someone writes that script, there are a couple of tools that can help.

Using AppCode

AppCode provides a way to take test results and “Show Statistics”:

The statistics are arranged by test suite. We can sort them by time:

Here, we can see the most expensive test suites in OCHamcrest. The first row weighs in at 0.413 seconds, with only 6 tests! That’s worth drilling into with a double-click:

Now we can clearly identify two tests that consume a lot of time.

So with AppCode, sort test statistics by “Time elapsed” and drill down to find the worst offenders.

Using xctool

Another way to find slow tests is by using xctool, the xcodebuild replacement. xctool color-codes its output, not only to make failures more visible, but to show what’s taking the most time. Green is fast. Yellow, not so fast. Red may need attention.

Here’s part of the OCHamcrest tests, run through xctool. I just scrolled and kept my eyes open for yellow or red:

As you can see, this also leads to the same two time-consuming tests.

What About Networking Tests?

Tests that do any network communication also need to be corralled. These tests may not seem that slow if you run them from work on your corporate intranet. But try them from home, and you may get different results due to timeouts.

So the problem is not just that networking tests are slow. They’re also inconsistent.

A quick way to identify them is to run your tests, and make sure you’re all green. Then unplug your Ethernet cable, turn off your wi-fi, and run your tests again. Anything that fails without a network connection needs to be moved aside (which I’ll describe below).

A bonus of removing networking tests: not only are your tests faster, but you can run them from an airplane, without purchasing any wi-fi!

What Do We Do with Slow Tests?

So now that we’ve identified tests that slow us down, what do we do with them?

Move them to a new test target. We’ll keep them, but not run them as often.

You might name this test target “SlowTests”. Or, because it tends to collect networking tests, “NetworkingTests” or “AcceptanceTests”.

With slow tests moved aside, we should now be getting faster feedback from the main set of unit tests. This is important for TDD and for test-enabled refactoring, where “compile and test” is a frequent step.

If You Don’t Use CI, Now Is the Time

The danger with having more than one test target is that I will run the fast target quite often, but forget to run the slow target before pushing my changes.

If you haven’t yet set up Continuous Integration on your project, this is a good time to do so. With Continuous Integration in place, you can let a robot do the tedious work of running all tests, slow and fast. When anything breaks, it will identify the offending commit and raise the alarms. With some social engineering, this will actually motivate people to run all test targets before pushing.

Keep Those Tests Snappy!

So let’s identify our slow tests and do something about them — either find a way to make them faster, or move them to a separate test target. Doing so will keep our main test target fast. And fast feedback is a marvelous thing! It’s the heart of TDD and of refactoring in general.

Which of your tests come in slow? What makes them slow? What will you do about them? You can let us know by clicking here.