What’s new for unit testing from WWDC21? Xcode 13 promises async, but many of us can’t use that for a couple of years. Instead, I want to look at new unit testing features we can use right away in Xcode 12.5.
(Watch the 10-minute video above, or keep reading below…)
Why Focus on Xcode 12.5?
When Apple announces “What’s new in testing” at their annual developer’s conference, I brace myself for UI testing features. I’ll be honest with you: I’m not interested in UI testing. I want test feedback that’s more precise and a heck of a lot faster. That’s why I focus on unit testing and wrote a book about it.
So when Apple does announce anything new for unit testing, I’m pleasantly surprised. They did it for WWDC20, and they did it again at WWDC21. They’re also establishing a pattern. When they have unit testing features that run on the existing operating systems, they bundle those in the last major release of Xcode before WWDC. And any features that require OS changes go into the beta for release in the fall.
Of course, the main thing everyone looks forward to is async/await. Xcode 13 will give us a much easier way to write async tests. But it’s not useful until you have async production code, and that requires iOS 15. Most of us have to support customers on older versions of iOS. So, I probably won’t be able to use the new async tests for another 2 years.
But Apple did add unit testing features to the current shipping version of Xcode. So let’s look at 6 new features we can use now in Xcode 12.5.
1. XCTAssertIdentical
Sometimes, you want to test that two objects are not just equal, but that they are the same object. For example, you may want to check that a property points to a particular view controller. Not just that it’s equal, but that it’s identical.
In the past, I’ve done this in Objective-C by comparing pointers with ==. In Swift, you compare objects for identity with ===. And I’ve asserted this comparison using XCTAssertTrue. But unlike XCTAssertEqual, XCTAssertTrue doesn’t have anything useful to say about mismatches. All it knows is that the condition failed. It doesn’t say why.
So I add a descriptive failure message. The result usually looks like this:
XCTAssertTrue(
actual === expected,
"Expected \(expected), but was \(actual)"
)But Xcode 12.5 provides new assertions: XCTAssertIdentical and XCTAssertNotIdentical. Like XCTAssertEqual, they take two objects as arguments. And if the assertion fails, it will report both arguments.
XCTAssertIdentical(actual, expected)So in general, by using these new assertions, we get more information. It’s always helpful to use the highest level assertion that matches your needs.
Both assertions are available for Swift, and also for Objective-C.
See what’s new for unit testing in Xcode 12.5, including XCTAssertIdentical
2. Clean Out Phantom Test Results
Do you ever see phantom test failures? This happens when a test fails, so you make some kind of change to the production code. But the test code doesn’t pick up the results of this change right away. The old failure annotations are still loitering around. It can be confusing, especially if you click on individual test diamonds to run tests separately. The old failures are still there, and this confusion slows us down.
So Xcode 12.5 adds a new menu item. In the Product menu, we now have “Clean Test Results”. Select it to make old test results go away. You can then concentrate on the tests you’re running without being thrown off by phantom failures from the past.
3. Lines-of-Code Count for Test Coverage
When I measure code coverage, I’m not that interested in the percentage. What I’m curious about is the number of lines that are definitely not covered. I also want to know how that number is trending over time.
I go into this in my book iOS Unit Testing by Example. What I’ve done up until now is use the command-line tool cloc to count the number of executable lines of code. I then multiply that by the inverse of the percentage to get the number of lines that were not covered.
We no longer need that extra step of installing cloc from Homebrew and writing a script to run it across our files. Xcode 12.5’s code coverage report now includes not just the percentage, but also that lines-of-code count. We still need to track how things are trending, but getting good information is that much easier now.
4. Generic Test Suites for Reuse
In general, test suites don’t have to be reusable. Once or twice in my career, I’ve written an abstract test suite as a superclass, where the subclasses provide some sort of factory method. Then all the test cases in that suite get repeated for the specific instances. (Since we don’t have abstract base classes in either Swift or Objective-C, this did mean it ran the tests for the parent suite, which was a little weird.)
Maybe you’ve wanted to repeat the same tests for a different object. For example, you may want to run the same test cases over different types that implement a protocol. Maybe you tried defining a generic XCTestCase, hoping to reuse the test suite across a few types. If you tried this before, you know that it didn’t work. XCTest uses introspection and follows specific rules to gather test cases and test suites. It didn’t pick up anything generic.
But Xcode 12.5 adds support for generic test suites. Here’s how it works. Write a generic subclass of XCTestCase. The generic class defines a generic factory method. Since we don’t have a proper type at this generic level, have the factory method return a generic optional, with the generic version returning nil.
class AbstractSuite<T>: XCTestCase {
func makeSpecificObject() -> T? { nil }
// Various tests…
}For this suite, don’t build your System Under Test in setUp(). Instead, write your tests to use factory methods. In this example, a test can call makeSpecificObject(). Or maybe the test calls a more complex helper, and that helper calls makeSpecificObject() to build the System Under Test.
Then you can define specialized versions, like this:
class OneClassTests: AbstractSuite<OneClass> {
override func makeSpecificObject() -> OneClass? {
// Make instance of OneClass
}
}
class AnotherClassTests: AbstractSuite<AnotherClass> {
override func makeSpecificObject() -> AnotherClass? {
// Make instance of AnotherClass
}
}This specializes the AbstractSuite for two versions. So it will run all the test cases for OneClass, then for AnotherClass. They can also specialize the same type, but vary the way they build their objects. And unlike the days when I did this with a quasi-abstract base class, XCTest will not run any tests for the top-level generic suite.
Thank you to Apple engineer Stuart Montgomery for explaining this feature to me.
5. Tests for watchOS
This one’s quick but important. If you write watchOS apps, you can now write tests for them. I suppose folks have been surviving by writing tests only for frameworks that contain no watchOS code. Hey Apple, what if… you made testability a first-class requirement for everything? (cough SwiftUI cough)
Anyway, this is good news for watchOS developers! You can write UI tests as well as unit tests. (But remember, unit tests are your friend.)
6. XCTExpectFailure
Have you ever had a test fail in your build system, and the fix wasn’t obvious? What do you do? Well, Xcode 12.5 introduces XCTExpectFailure for marking a test that is known to fail. The test still executes, it still fails, but the failure doesn’t affect the overall test outcome.
XCTExpectFailure is a way to make an exception for a failing test, saying, “Don’t count this one.”
What’s the difference between this and XCTSkip? A skipped test throws an exception as soon as it hits the skip. So it doesn’t continue from that point. But XCTExpectFailure says, “Keep going. This test should fail. And if it doesn’t, let’s count the unexpected pass as a failure to get your attention.”
But this situation should be rare. I can’t see myself ever using XCTExpectFailure. But it did give me an idea for a way to use XCTSkip to prevent cascading failure reports, so that’s good! I’ll explain that another time.
Wrap-Up
…So that’s what’s new for unit testing from WWDC21, at least for Xcode 12.5. And unlike async/await, these are features we can start using in our tests right away.
Which new unit testing feature excites you the most? Please leave a comment to share your reactions.
