WWDC20: What’s New in Unit Testing

WWDC20 has wrapped up. What’s new this year for unit testing? Let’s look at the changes Apple has made for the Xcode 12 beta, and also some changes in the current version of Xcode.

Hello, I’m Jon Reid from QualityCoding.org. Every summer, Apple has their Worldwide Developer’s Conference, part of which is announcing updates to their developer tools. Every year, they announce new features in their testing tools. But most years, those new features are mostly about UI testing. So they don’t have much impact on my unit testing workflow.

But not this year! The XCTest team put a lot of unit testing improvements into the Xcode 12 beta. They also snuck some important changes into Xcode 11.4 a little earlier this year, which we can use right away. So let’s get into what’s new for unit testing, my Top 8. (17-minute video)

8: Execution Time Allowance

Let’s start with one that shouldn’t have an impact on your unit tests, but may still be useful. That’s the ability to set a time limit on the execution time of any test. The shortest time we can set is one minute. Hopefully, you don’t have any non-UI tests that run that long! But in the session “Get your test results faster” one example was of a test that makes a real network call.

Now unit tests should never do real networking. But I have used XCTest to write a few integration tests that talk to a real server. This can be helpful if your mobile team has no real relationship with the server team.

But one minute should still be way more time than you need. One way to set the timeout is to do so directly inside the test.

XCTestCase has a new property executionTimeAllowance. We can set it to a time interval in seconds, which it raises to the nearest minute. So in this example, the test should fail if it takes more than 1 minute.

But setting this value has no effect by itself. The new timeout mechanism is off by default. To turn it on, you have to use Xcode’s Test Plans. So if you don’t already have a test plan, you’ll have to make one by editing the scheme. Then select the test plan, go into the Configurations, and turn “Test Timeouts” on.

Now let’s run this to see what it looks like. I’ll skip to the end. …And there we have it: test failed with “Test exceeded the execution time allowance of 1 minute.”

Me, if a project is set up on a build server, I wouldn’t do this way. Instead, I’d change the test script. For Xcode 12, the command line tool xcodebuild has a new argument maximum test, -default-test-execution-time-allowance. We can set the time to 60 seconds.

But again, setting that time by itself doesn’t do anything. You also have to enable timeouts at all, by setting -test-timeouts-enabled to YES. Then xcodebuild will apply this limit to all tests, and will fail any test that runs longer.

Now Sean, who gave this WWDC presentation, cautions me that the purpose of this feature is to make sure tests complete, and not so much that they’re fast. But in a big team, I usually don’t run all tests locally. I fire off a build and walk away, so I’m not paying much attention to test time. So I think setting a 1-minute timeout on unit tests could be helpful. It would be better to write build scripts to alert you about test duration. But as a poor-man’s version, this is an easy start.

7: #filePath

Number seven isn’t exactly an improvement, but it’s an important change to be aware of. I often extract helper functions as custom assertions. Here’s a simple example, running on Xcode 11.5. We have two tests that grab some count from production code. Each test calls a helper, verifyCount.

When we run tests, Xcode 11.5 shows that both tests fail, which you can see on the left annotations with the red X marks. But the failure annotation is inside the helper. It’s hard to tell what’s going on. We can easily fix this by adding arguments to the helper function to capture the location of the call site.

I do this with a code snippet called testhelper. If you’d like to get the test-oriented code snippets I use, subscribe to qualitycoding.org.

Once we have the file name and line number of the call site, we pass those along to the assertion, like this:

file: file, line: line

Now when we rerun tests, the failure annotations move from the test helper to each test case. This makes it much easier to see what’s going on.

But here’s the thing. This special #file symbol expands to the full path of the file. I had no idea that #file was used outside of tests, but apparently there’s a lot of implicit use in the toolchain. This causes code bloat and security leaks. So Swift shortened #file from an absolute file path to a shorter, relative file path. This makes the content shorter, but causes a warning in Xcode 12. So they added a new symbol #filePath which has the old meaning of expanding to the absolute file path. So this is a change in the Swift language. Once we adopt it with Xcode 12, we’ll have to change all our test helpers to use #filePath instead of #file.

I really wish they’d done it the other way around. But it’s not a hard change to make, because where else are you using #file? We can just do a global change.

So, wherever you have test helpers that use #file, be prepared to update those for Xcode 12.

6: XCTSkip

Before we look at XCTSkip, I want to jump back to WWDC19 to review something.

Back in 2019, Xcode 11 added XCTUnwrap. This is a function that tries to unwrap an optional value and return that value. If the value is nil, it throws an exception. This makes it easier to have code that sets up the objects you need for a particular test—without having to do a guard, an XCTFail, and a return.

Now back to WWDC20. Xcode 11.4 added XCTSkip. This is an exception to show that you decided to skip a test for some reason.

There are helper functions XCTSkipIf and XCTSkipUnless which throw this exception. You can also make your own helpers which throw the exception.

Before, Xcode would show the test diamond as either not run, or passing, or failing. Whenever I needed to skip a test at runtime, I would have a guard clause at the top that returns. But this would show as a passing test.

Xcode 11.4 actually cooks in support for a new test result, a skipped test. This shows up in the logs. Even better, it displays as a gray annotation inside of Xcode. Ahh! As you can see, XCTSkip can have an optional message. I recommend always adding a message, because you might skip a test for various reasons. So describe the reason for the skip.

5: Throwing setUp and tearDown

In the xUnit style of testing frameworks, a test can fail for two reasons. The first reason is if an assertion fails. The second is if something throws an exception.

XCTest belongs to the larger xUnit family. But lately, Apple is leaning more toward exceptions thrown specifically by test code. We get this by declaring tests as throwing functions. And we’re starting to accumulate throwing helpers like XCTUnwrap and XCTSkip.

In Swift, you can’t just throw an exception from anywhere. We have to mark the function with throws. So it makes sense to extend this ability to setUp and tearDown as well. Xcode 11.4 adds setUpWithError and tearDownWithError. XCTest calls setUpWithError, then your test method, then tearDownWithError. If anything goes wrong on either end, just throw an exception.

This is especially useful if you have tests that set up their object graph by reading a file. For example, you might have some complicated JSON responses saved. If something goes wrong reading them during setUpWithError, throw an exception. XCTest will pull the rip cord, and fail the test.

4: Call Stack Breadcrumbs on Failures

With more built-in functions throwing exceptions, it’s likely that you’ll also write test helpers that throw. But if a test helper throws an exception, how can we see that in the test that calls it?

Apple solves this problem in Xcode 12. When there’s an exception, it marks that spot with the red X of failure. But it also walks backward in the call stack, leaving gray annotations. These serve as breadcrumbs. I don’t know what Apple calls them, but I’m calling them “call stack breadcrumbs.”

With this, we can see a thrown exception right inside the test case method. But that’s not all! Apple also extended this mechanism to include assertion failures. So remember before where we had to capture the #filePath and pass it down to any assertions inside a helper? Without those arguments, Xcode 12 will show breadcrumbs for the distant assertion failure.

I still prefer how things look when you capture the file name and line number of the call site. But you may like it this way. It’s certainly easier and more concise not to have to pass file and line arguments to every assertion.

3: Standalone Test Helpers in Objective-C

If you still have any Objective-C tests, this one’s for you. Back in the day, the original testing framework for Objective-C was called SenTestingKit. This was a third-party library. Eventually, Apple licensed it, and integrated it into Xcode 4.

In Objective-C, the only way to for assertions to capture the file name and line number is with C preprocessor macros. These macros packaged up information about the assertion, and called an instance method.

Because the underlying call was to an instance method, we ended up using assertions only in subclasses of the test case class. This was a limitation of the framework which made it difficult to write test helpers.

But when we got the Swift version of XCTest, assertions became static functions. They weren’t restricted to test case subclasses, so you can assert anywhere. This makes it much easier to write and reuse standalone test helpers without subclassing.

Now with Xcode 12, the Objective-C assertions also get to grow up and become independent. They can live anywhere. Capturing the file name and line number of the call site is still a challenge. Part of the reason I wrote OCHamcrest was to do all that for you, so you could write custom assertions without writing preprocessor macros, and get the file and line.

But! Now that Xcode has call stack breadcrumbs, we don’t have to do fancy tricks to get good reporting. Our test helpers can just assert what they want. Xcode will walk back along the call stack and add annotations. Look at that! So without any tricks, you can see the failure message right next to the call site. If you write Objective-C tests, this is good news.

2: XCTIssue for Custom Failures

For folks who like writing third-party testing libraries, this one’s important. Apple is deprecating recordFailure. This is the XCTestCase method that takes a failure description, file path, and line number. It also takes a boolean flag indicating whether the failure is from an assertion or an exception.

That set of arguments goes way back, even before recordFailure. Third party tools had to put any diagnostics into the failure description, as one long string.

But now we have a better way. Xcode 12 adds record(_ issue:) which takes a new type, XCTIssue. It has several properties.

First, instead of a boolean flag with two values, XCTIssue represents the failure type as an enumeration. Currently, the values include:

• assertionFailure for assertions
• thrownError for exceptions thrown by test code
• uncaughtException for other exceptions
• performanceRegression
• system. This is for UI testing failures like not launching the app, or not finding a particular UI element.

Moving on, XCTIssue has not one, but two description strings: a compactDescription, and a detailedDescription.

Of course XCTIssue has file name and line number. But those are now packaged into a new type XCTSourceCodeContext. This type also includes the test code call stack, for those lovely breadcrumbs.

XCTIssue can have an associatedError. And it has an array of XCTAttachment to attach arbitrary data, like images.

So an XCTIssue can have all sorts of things in it. Then we attach it to a particular test case using record(_ issue:). That’s how you stick stuff in. To get stuff out, we can use XCTestObservation. Instead of getting the old failure description, file name, and line number, it now gets an XCTIssue.

This is a fantastic new playground. We should see improvements to existing third-party testing tools. But it also opens the door for new testing tools. This is super-exciting!

1: Offer to Switch Schemes to Run Tests

Say you’re working in a workspace. Here, we have BigApp. Part of the code is in two frameworks, FeatureOne and FeatureTwo. And you go to the tests for FeatureTwo… but where are the test diamonds? And you go crazy for a while, until you realize your current scheme doesn’t include those tests. This is what is looks like in Xcode 11.5.

Now let’s open the same workspace in Xcode 12. This time, even though we’re in the wrong scheme, the test diamonds show up for FeatureTwo. Click a diamond, and Xcode asks if we want to switch schemes. Yes, please! Boom.

Wrap-up

So those are my top 8 observations of what’s new in unit testing for WWDC20. Which ones are your favorites? Are there any that I missed? Please leave a comment below to share your thoughts.

Also, my book iOS Unit Testing by Example is now published, so go check that out at PragProg.com. Thanks for watching!