Every WWDC, I hope for improvements to unit testing—but have learned to expect disappointment. So at WWDC 2018, I was surprised to have my low expectations thwarted! Xcode 10 brings changes that will improve my test-centric workflow.
For several years, Apple's changes for test support have underwhelmed me. They focused on:
- Xcode Bots (…I tried them. I gave up.)
- Performance tests (…My code is not performance-critical.)
- UI tests (…I don't write any.)
With Test-Driven Development, unit tests run locally are my primary tool. The features above may have helped some people, but I wasn't one of them.
So what did WWDC 2018 bring me? Here's what I see on the horizon with Xcode 10.
1. Parallel Testing
You may be used to stepping away for a break while tests run locally. And I hope you have a Continuous Integration system that runs tests before merging to master.
The tests I write are usually microtests. When each test is fast enough, you can run thousands of tests in a few seconds. At WWDC, I got to meet Andrew Ebling. He tried converting his fast-running tests to run in parallel, but saw no benefit. Apparently, the cost of spinning up multiple simulators isn't recovered if the tests themselves execute too quickly.
But microtests aren't the only kind of tests I write. I also lean on snapshot tests of views. Apple has their own snapshot system, part of their UI test framework. But the tests I use have nothing to do with UI-driven testing. They don't navigate to a view in a running app. Instead, they programmatically create the views or view controllers, and render them.
Such snapshot tests are much faster than UI tests. Yet one such test takes hundreds of times longer than a microtest. And we have lots of them.
These snapshot tests were bogging down the microtests. To improve TDD turnaround time, I moved snapshot tests into a separate test bundle. But we still need to run them, especially before merging to master. This looks like a case where parallel testing will pay off! And they should help speed up UI tests, which was what Apple demonstrated. I expect significant improvements in test time with Xcode 10.
2. Randomized Test Order
At last, a new feature of XCTest to help with unit testing! (Well, that's not totally true. The Swift version of XCTest has brought some significant improvements.)
Randomizing test execution order is the first sign I've seen of any new unit testing features in XCTest. I hope it is the first of many. It deserves its own blog post.
I'm glad to finally be included in the target audience for a testing feature!
3. Swift Compile Time
Ironically, the biggest improvement that Xcode 10 brings to testing isn't a testing feature. The greatest hindrance to TDD has been Swift compile time. It's been pretty abysmal for me. Remember, fast feedback is essential for TDD to flow.
In desperation, we learned to play tricks with whole-module-compiling. This improved build time, at the cost of sacrificing incremental builds.
So it was a huge relief to see Apple work on Swift compile time. Things should be so much better with Xcode 10! We'll be able to leave that whole-module-compiling hack behind. I want to make a little change in test code, build it (and nothing else), and see a test fail.
Every improvement Apple is making to Swift compile time is a big deal for TDD, because TDD works best in small steps.
The biggest improvement Xcode 10 brings to testing is Swift compile time.
Looking Forward to Using Xcode 10 This Fall
Xcode 10 promises to bring big improvements to my daily test-driven workflow. I look forward to using it this fall, when Apple releases iOS 12.
What did you get from this year's WWDC? How will these testing benefits affect your projects? Please share in the comments below!
Thanks for this post. Objective-C compile time is much faster, than in Swift. I think it is because in Swift every declared class or struct is available within the whole module.
So, complication time improvements will help a lot.
I think there are probably ways we can structure Swift code to reduce build dependencies. I think I’ll wait until I can start using Xcode 10, though… I’m really looking forward to the speed increase! (When I say “Use Xcode 10,” I really mean “Use a version of AppCode that can talk to Xcode 10.”)
My unit tests are running faster in Xcode 10, and I don’t think it’s either of the two features Apple mentioned.
It seems like the simulator can launch faster, so the non-logical tests run faster than before (~4.6 seconds for 30 tests instead of ~7.35 seconds in Xcode 9.4.1)
BUT I still think the fastest is to use isolated (non-UIKit) logical tests, since you don’t need to wait for the app to launch.
For those you need to statically link to all source files under test, and your file/path logic won’t mirror the structure of the iPhone app directory.
That’s good to hear, Paul, thanks for sharing.
I do have some success putting sections of code inside a framework. As long as those tests don’t require a living app, I get that benefit of test-without-launch.