Are Slow Tests Killing Your Feedback Loop? 

 April 28, 2015

by Jon Reid


Hit ⌘U to run your tests. …Once the tests start executing, how long do you have to wait for the results?

If you waited more than a few seconds, you may have a problem. Because one sure-fire way to discourage test-driven development on your team is to have unit tests take 30 seconds or more.

When I was taking water safety training, one of the things we had to do was jump into the water fully clothed. You don’t normally notice the feel of your own clothing, so it’s surprising to feel how waterlogged clothes hinder every basic swimming movement. The training is to learn how to remove that clothing in the water, to restore normal swimming. It’s like shedding a heavy straitjacket; suddenly you can move again!

Many of us are tied down by our slow testing experiences. Slow tests mean it takes longer to get results. Taking longer to get results means we won’t run our tests as often. Not running our tests as often means we’re undercutting the core benefit of test-driven development—namely, getting feedback frequently.

So let’s restore normal movement by shedding those heavy tests and setting them aside. But which tests are “heavy”? And what do we do with them?

What Is a Slow Test?

Disclosure: The book links below are affiliate links. If you buy anything, I earn a commission, at no extra cost to you.

In his excellent book Working Effectively with Legacy Code, Michael Feathers lists various characteristics of tests that can make them unsuitable. But he summarizes it all in a quick definition:

A unit test that takes 1/10th of a second to run is a slow unit test.

As Michael says: “Yes, I’m serious.” Because before you know it, you’ll have 10,000 tests. If they all took a tenth of a second, we’re talking about sitting around for 15 minutes just waiting for that feedback.

"A unit test that takes 1/10th of a second to run is a slow unit test." — Michael Feathers

Click to Tweet

How Do You Identify Slow Tests?

So how do we find these slow tests? Combing through Xcode’s logs takes too long. I think there’s an open need for a script that will filter the test output to show the slow tests. But until someone writes that script, there are a couple of tools that can help.

Using AppCode

AppCode provides a way to take test results and “Show Statistics”:

AppCode test option "Show Statistics"

The statistics are arranged by test suite. We can sort them by time:

AppCode showing time elapsed per suite

Here, we can see the most expensive test suites in OCHamcrest. The first row weighs in at 0.413 seconds, with only 6 tests! That’s worth drilling into with a double-click:

AppCode showing time per test

Now we can clearly identify two tests that consume a lot of time.

So with AppCode, sort test statistics by “Time elapsed” and drill down to find the worst offenders.

Using xctool

Another way to find slow tests is by using xctool, the xcodebuild replacement. xctool color-codes its output, not only to make failures more visible, but to show what’s taking the most time. Green is fast. Yellow, not so fast. Red may need attention.

Here’s part of the OCHamcrest tests, run through xctool. I just scrolled and kept my eyes open for yellow or red:

xctool showing times

As you can see, this also leads to the same two time-consuming tests.

What About Networking Tests?

Tests that do any network communication also need to be corralled. These tests may not seem that slow if you run them from work on your corporate intranet. But try them from home, and you may get different results due to timeouts.

So the problem is not just that networking tests are slow. They’re also inconsistent.

A quick way to identify them is to run your tests, and make sure you’re all green. Then unplug your Ethernet cable, turn off your wi-fi, and run your tests again. Anything that fails without a network connection needs to be moved aside (which I’ll describe below).

A bonus of removing networking tests: not only are your tests faster, but you can run them from an airplane, without purchasing any wi-fi!

What Do We Do with Slow Tests?

So now that we’ve identified tests that slow us down, what do we do with them?

Move them to a new test target. We’ll keep them, but not run them as often.

You might name this test target “SlowTests”. Or, because it tends to collect networking tests, “NetworkingTests” or “AcceptanceTests”.

With slow tests moved aside, we should now be getting faster feedback from the main set of unit tests. This is important for TDD and for test-enabled refactoring, where “compile and test” is a frequent step.

If You Don’t Use CI, Now Is the Time

The danger with having more than one test target is that I will run the fast target quite often, but forget to run the slow target before pushing my changes.

If you haven’t yet set up Continuous Integration on your project, this is a good time to do so. With Continuous Integration in place, you can let a robot do the tedious work of running all tests, slow and fast. When anything breaks, it will identify the offending commit and raise the alarms. With some social engineering, this will actually motivate people to run all test targets before pushing.

Keep Those Tests Snappy!

So let’s identify our slow tests and do something about them — either find a way to make them faster, or move them to a separate test target. Doing so will keep our main test target fast. And fast feedback is a marvelous thing! It’s the heart of TDD and of refactoring in general.

Which of your tests come in slow? What makes them slow? What will you do about them? You can let us know by clicking here.

Jon Reid

About the author

Programming was fun when I was a kid. But working in Silicon Valley, I saw poor code lead to fear, with real human costs. Looking for ways to make my life better, I learned about Extreme Programming, including unit testing, test-driven development (TDD), and refactoring. Programming became fun again! I've now been doing TDD in Apple environments for 20 years. I'm committed to software crafting as a discipline, hoping we can all reach greater effectiveness and joy. Now a coach with Industrial Logic!

  • hello,
    congratulations for the work they were doing.

    I have a doubt, best in the blocks and put the assert inside or at the end of the method?

  • Any thoughts on decreasing test time on iOS app? The tests I wrote seem fast, but it always takes several seconds before they start running even on very small projects.

    This has been a hinderance for doing TDD on iOS apps. When I was a ruby developer this was usually not an issue since tests would start almost instantly as long as you were not loading too many third party libs.

    • Thijs, I’m afraid for iOS tests, Xcode has to copy the built app to the simulator, launch the app, then inject the test bundle — all before a single test is run. The situation is worse in Swift (for now) because the standard Swift libraries are also copied over.

      So I’m afraid we’re stuck with that. One way to avoid it is to write tests only against a framework, not an app. The problem with this is that it eliminates a whole class of of unit tests that depend on UIKit running within an app (like performing button taps). At some point I get frustrated by this limitation and go back to testing against an app.

      Compile time improvements help. We’re still stuck with the whole “launch in the simulator” process, but compilation speed can also have an impact.

  • {"email":"Email address invalid","url":"Website address invalid","required":"Required field missing"}