Xcode Unit Testing: The Good, the Bad, the Ugly 

Before Xcode 4, I recommended adding third-party unit testing frameworks such as Google Toolbox for Mac (GTM) and GHUnit. But with Xcode 4, the out-of-the-box tools are mostly sufficient. I say "mostly," because it's still a mix of the good, the bad, and the ugly. But, mostly good! Read on for a rundown of the pros and cons…

Good: Creating new projects

Starting off a new project with Xcode unit testing used to be complex and error-prone, requiring arcane project settings. The third-party solutions were considerably simpler.

But with Xcode 4, the "New Project" templates include a simple checkbox:

Just check "Include Unit Tests" and you're good to go. It doesn't come any simpler than that!

Good: Adding test files

To add a new test file, just ⌘N to create a new file, and choose the template for test cases. Or, reveal the File Template Library and drag the template directly into the Navigator area. Either way, just make sure the new file goes in your test target, not your primary target:

In a later post, I describe how Apple's "Objective-C test case class" template is less than ideal, and what to do about it.

Good: Test management

For iOS unit tests, Apple used to make a confusing distinction between "logic tests" and "application tests." In fact, their documentation still reflects this. The former went into a bundle that ran against your application's classes, while the latter went into a specialized app that could only run on a device. So you had to separate your tests depending on whether they required the iOS runtime—for example, most tests of view controllers went in the "application tests" category.

So to create a new test case, you had to ask yourself, "What kind of test is this? Does it need the iOS runtime, or can it run on its own?" Depending on your answer, you had to create your test in a file targeted to either your logic tests or your application tests.

This is the biggest reason I used to recommend third-party frameworks for iOS unit testing. Both GTM and GHUnit run all your tests directly in the simulator, with no need to corral them into separate families.

Thankfully, Xcode 4 follows suit: All tests can now be run in the simulator, in one shot. Although Apple still needs to update their documentation, there's no need to distinguish between "logic tests" and "application tests"—just write your test!

Good: Target management

GTM and GHUnit require separate testing targets that are essentially copies of your application, but with testing code added. What this means is that any changes you make to your primary target must be repeated in your testing target. Add a new file? It has to go in both targets.

With Xcode 4, iOS unit tests are now much more like Cocoa unit tests: You have your application target, of course. But your test code goes into a test target that isn't a copy of your application. Rather, it's a bundle containing only your test code. To execute tests, Xcode launches your application, then injects the test bundle into the running application! This

• lets you manage your primary target without error-prone duplication;
• reduces build time because you're no longer compiling the same code twice;
• increases confidence because you know the tests running against the real application, not a copy.

Good: Test execution

When tests execute in a separate testing application, you have to switch your active target to the testing side, and do a build (which also executes the tests in most setups). There's a dance switching back-and-forth between your testing application and your real application.

Xcode 4 treats test execution as an integrated action. As the image at the top of this post shows, you can click and hold the Run button to show a pop-up menu of actions, and select Test. But why mouse around when there's an even faster way? Just ⌘U. Think U for unit tests.

Bad: Continuous integration

So far so good. But there's more to unit tests than manual execution. How do you invoke them from the command line, for automatic execution in a continuous integration system like Hudson/Jenkins? This is important for making sure your unit tests always pass, and identifying test breaks as soon as possible.

Here I have bad news. But there is a glimmer of hope.

How do you execute iOS unit tests from the command line? The short answer is: Apple doesn't support it at this time. You'd think there would be an argument to xcodebuild, like "test" as one of the available build actions. No such luck.

So what do we do? There are two options:

• Create a separate testing application. But this defeats the benefits of Xcode 4's test bundles, so you might as well stick with GTM or GHUnit.
• On your continuous integration machine, follow these instructions to alter a script in the simulator SDK.

I have yet to try the latter approach. If you've done it, please share your experiences in the comments!

Ugly: Coverage analysis

Coverage analysis an important tool for measuring your unit test code:

• You can quickly spot any areas that your unit tests haven't covered.
• If the coverage percentage goes down, you know some code was written without unit test verification.

Here too, I have bad news. But there's a workaround.

The bad news is not about Xcode 4 per se, but about the new LLVM compiler. If you're still using GCC, then you can measure code coverage the same as before. But for projects that have shifted to LLVM, there is no coverage tool at this time.

The workaround is fairly simple: Create a new configuration that uses GCC instead of LLVM, and use it when measuring coverage. In a later post, I will explain how to set this up.

Conclusion: Thumbs-up

Xcode unit testing for iOS has come a long way with Xcode 4.

It seems Apple hasn't figured out that there's more to unit tests than just running them. How do we run them continuously? How do we measure them?

But for day-to-day development, Xcode 4's tight integration makes unit testing easier than it's ever been for iOS developers. If you have an established setup using GTM or GHUnit, you'll need to weigh the pros and cons above to decide whether you want to switch. If you use GTM or GHUnit's extended test assertions, just stay put. But if you're starting fresh, I'd go with the Xcode 4 out-of-the-box experience.

Question: What has your experience been with iOS unit tests on Xcode? Leave a comment below.