Xcode Unit Testing: The Pros & Cons of Xcode 4

July 30, 2011 — 16 Comments
Xcode unit testing is now fully integrated into the UI!

Unit testing is now fully integrated into the UI!

Xcode unit testing has come a long way for iOS development. Before Xcode 4, I recommended adding third-party unit testing frameworks such as Google Toolbox for Mac (GTM) and GHUnit. But with Xcode 4, the out-of-the-box tools are mostly sufficient. I say “mostly,” because it’s still a mix of the good, the bad, and the ugly. But mostly good, read on for a rundown of the pros and cons…

Good: Creating new projects

Starting off a new project with Xcode unit testing used to be complex and error-prone, requiring arcane project settings. The third-party solutions were considerably simpler.

But with Xcode 4, the “New Project” templates include a simple checkbox:

Just click the checkbox to enable Xcode unit testing!

Just check “Include Unit Tests” and you’re good to go. It doesn’t come any simpler than that!

Good: Adding test files

To add a new test file, just ⌘N to create a new file, and choose the template for test cases. Or, reveal the File Template Library and drag the template directly into the Navigator area. Either way, just make sure the new file goes in your test target, not your primary target:

Select the unit test target

In a later post, I describe how Apple’s “Objective-C test case class” template is less than ideal, and what to do about it.

Good: Test management

There’s no need to distinguish between “logic tests” and “application tests” — just write your test!

For iOS unit tests, Apple used to make a confusing distinction between “logic tests” and “application tests.” In fact, their documentation still reflects this. The former went into a bundle that ran against your application’s classes, while the latter went into a specialized app that could only run on a device. So you had to separate your tests depending on whether they required the iOS runtime — for example, most tests of view controllers went in the “application tests” category.

So to create a new test case, you had to ask yourself, “What kind of test is this? Does it need the iOS runtime, or can it run on its own?” Depending on your answer, you had to create your test in a file targeted to either your logic tests or your application tests.

This is the biggest reason I used to recommend third-party frameworks for iOS unit testing. Both GTM and GHUnit run all your tests directly in the simulator, with no need to corral them into separate families.

Thankfully, Xcode 4 follows suit: All tests can now be run in the simulator, in one shot. Although Apple still needs to update their documentation, there’s no need to distinguish between “logic tests” and “application tests” — just write your test!

Good: Target management

GTM and GHUnit require separate testing targets that are essentially copies of your application, but with testing code added. What this means is that any changes you make to your primary target must be repeated in your testing target. Add a new file? It has to go in both targets.

With Xcode 4, iOS unit tests are now much more like Cocoa unit tests: You have your application target, of course. But your test code goes into a test target that isn’t a copy of your application. Rather, it’s a bundle containing only your test code. To execute tests, Xcode launches your application, then injects the test bundle into the running application! This

  • lets you manage your primary target without error-prone duplication;
  • reduces build time because you’re no longer compiling the same code twice;
  • increases confidence because you know the tests running against the real application, not a copy.

Good: Test execution

Why mouse around when there’s an even faster way? Just ⌘U.

When tests execute in a separate testing application, you have to switch your active target to the testing side, and do a build (which also executes the tests in most setups). There’s a dance switching back-and-forth between your testing application and your real application.

Xcode 4 treats test execution as an integrated action. As the image at the top of this post shows, you can click and hold the Run button to show a pop-up menu of actions, and select Test. But why mouse around when there’s an even faster way? Just ⌘U. Think U for unit tests.

Bad: Continuous integration

So far so good. But there’s more to unit tests than manual execution. How do you invoke them from the command line, for automatic execution in a continuous integration system like Hudson/Jenkins? This is important for making sure your unit tests always pass, and identifying test breaks as soon as possible.

Here I have bad news. But there is a glimmer of hope.

How do you execute iOS unit tests from the command line? The short answer is: Apple doesn’t support it at this time. You’d think there would be an argument to xcodebuild, like “test” as one of the available build actions. No such luck.

So what do we do? There are two options:

  • Create a separate testing application. But this defeats the benefits of Xcode 4’s test bundles, so you might as well stick with GTM or GHUnit.
  • On your continuous integration machine, follow these instructions to alter a script in the simulator SDK.

I have yet to try the latter approach. If you’ve done it, please share your experiences in the comments!

Ugly: Coverage analysis

Coverage analysis an important tool for measuring your unit test code:

  • You can quickly spot any areas that your unit tests haven’t covered.
  • If the coverage percentage goes down, you know some code was written without unit test verification.

Here too, I have bad news. But there’s a workaround.

The bad news is not about Xcode 4 per se, but about the new LLVM compiler. If you’re still using GCC, then you can measure code coverage the same as before. But for projects that have shifted to LLVM, there is no coverage tool at this time.

The workaround is fairly simple: Create a new configuration that uses GCC instead of LLVM, and use it when measuring coverage. In a later post, I will explain how to set this up.

Conclusion: Thumbs-up

Xcode unit testing for iOS has come a long way with Xcode 4.

It seems Apple hasn’t figured out that there’s more to unit tests than just running them. How do we run them continuously? How do we measure them?

But for day-to-day development, Xcode 4’s tight integration makes unit testing easier than it’s ever been for iOS developers. If you have an established setup using GTM or GHUnit, you’ll need to weigh the pros and cons above to decide whether you want to switch. If you use GTM or GHUnit’s extended test assertions, just stay put. But if you’re starting fresh, I’d go with the Xcode 4 out-of-the-box experience.

Questions? Comments? What has your experience been with iOS unit tests on Xcode 4? Please share in the comments.

Jon Reid

Posts Twitter Facebook Google+

I'm passionate about not just improving our code, but improving the way we code.

16 responses to Xcode Unit Testing: The Pros & Cons of Xcode 4

  1. Hey Jon,
    For xcodebuild you should be able to use the -target footarget argument to build your unit test target.

    • That builds the unit test target, but doesn’t run the tests. Try it, and you’ll see the following:
      RunPlatformUnitTests:95: warning: Skipping tests; the iPhoneSimulator platform does not currently support application-hosted tests (TEST_HOST set).
      I just tried the technique of hacking that script, and it worked quite well! The only failures I had were in testing whether something was first responder, which was already skirting the edge of possibility.

      • Ah. I only have “logic” tests in my set of SenTests so I don’t run into that issue, but then I have a very jury-rigged set up where my logic tests are SenTests while my UI tests are Google Toolbox tests. This is of course very suboptimal because it is difficult to run them often. I have a script that runs both sets of tests, but that’s not such a good setup.
        The fact that you get that message about the iPhoneSimulator shows that support is still immature.

  2. Hey Jon, nice write up. Just an FYI code coverage does exist for LLVM. Check out:
    http://code.google.com/p/coverstory/wiki/UsingCoverstory
    To see how to set it up (note this is with 4.2).

    • That’s great news, Dave — thanks!
      I’ve been using lcov, but can see ways in which CoverStory might complement it, or be more suitable for some people. That’ll be for another post…

  3. Good article. I am fresh men, try it at first.

  4. Hi John,

    You mentioned that there’s no need to distinguish between ‘application’ and ‘logic’ tests any more, but there’s still a difference in how tests are run depending on the bundle loader and test host settings. How do you run your tests?

    Thanks

  5. Hi Jon,

    I have read back and forth about application vs logic tests. I have so far had success running kiwi specs from the terminal by adding TEST_HOST=” to my build script. Do you know of or have you ever encountered any troubles with this approach?

Leave a Reply

*

Text formatting is available via select HTML.

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong> 

Have you Subscribed yet? Don't miss a post!