I try to avoid asynchronous tests because they’re slow. But sometimes they’re worth having. When you do have them, it’s worth spending extra effort to make them as fast as possible.
My mistake was that I only focused on the speed of the happy path. Today, I learned a way to improve the speed of failure scenarios, at least around testing network requests.
Test speed matters a great deal. In test-driven development (TDD), you want to repeatedly take small, verified steps. Fast feedback is crucial to the process.
But even apart from TDD, test speed is important to keep your whole team moving forward. When someone submits a pull request, your build system builds your system and runs its tests. The time this requires puts pull request on hold. And the more simultaneous pull requests you have, the worse things get.
Let me rephrase that positively: small wins in test speed can accumulate into big wins for your entire team.
So let’s look at an asynchronous test that does real networking. If you have any tests like this, there may be an opportunity to improve their speed.
[This post is part of the series TDD Sample App: The Complete Collection …So Far]
Networking Tests Aren’t Unit Tests
Before we start, a reminder: The kind of test we’re going to look at doesn’t belong in your unit test target.
And when I say “unit test,” I’m not particular about whether they’re testing isolated units. I use the term “unit test” quite loosely. What I mean is “fast, reliable tests.” They can be testing complex systems. As long as they’re fast, I’m happy.
Networking tests are fine but belong in a separate test target. Even when they’re valuable, I don’t want networking tests clogging up my TDD workflow.
Fast Happy Path, But Slow Failure
I have code to authenticate calls to the Marvel Comics API. I want acceptance tests to verify that my code actually works. Here’s what I came up with for the Objective-C version:
- (void)testValidCallToMarvel_ShouldGetHTTPStatusCode200
{
NSString *validQueryMissingAuthentication = @"https://gateway.marvel.com/v1/public/characters?nameStartsWith=Spider";
NSURL *validQueryURL = [NSURL URLWithString:
[validQueryMissingAuthentication stringByAppendingString:[QCOMarvelAuthentication URLParameters]]];
__block NSHTTPURLResponse *httpResponse;
[self startGETRequestToURL:validQueryURL
withCompletionHandler:^(NSData *data, NSURLResponse *response, NSError *error) {
if (error)
XCTFail(@"%@", error);
httpResponse = (NSHTTPURLResponse *)response;
}];
assertWithTimeout(5, thatEventually(@(httpResponse.statusCode)), is(@200));
}This uses the OCHamcrest expression “assert with a timeout (in seconds) that eventually (some value) will satisfy this predicate.” There are similar ways to express this in other test support libraries, from Kiwi to Quick & Nimble.
For the Swift version, I used XCTest’s built-in expectation API:
func testValidCallToMarvel_ShouldGetHTTPStatusCode200() {
let queryWithoutAuth = "https://gateway.marvel.com/v1/public/characters?nameStartsWith=Spider"
let fullQuery = queryWithoutAuth + MarvelAuthentication.init().urlParameters()
guard let validQueryUrl = URL(string: fullQuery) else {
XCTFail("Invalid URL '\(fullQuery)'")
return
}
let promise = expectation(description: "Status code: 200")
startDataTask(with: validQueryUrl) { data, response, error in
if error != nil {
XCTFail("Error: \(error!.localizedDescription)")
return
}
let statusCode = (response as! HTTPURLResponse).statusCode
if statusCode == 200 {
promise.fulfill()
} else {
XCTFail("Status code: \(statusCode)")
}
}
self.waitForExpectations(timeout: 5, handler: nil)
}I don’t like the wordiness of the expectation approach. But because the Swift language is still a moving target, I’m being extra vigilant to avoid dependencies.
While trying to get away from this wordiness, I made a surprising discovery:
- When the test passes, it takes about 1 second. That’s good. But…
- When the test fails, it takes the full 5-second timeout.
I’m not counting the first XCTFail which I added for completeness in the Swift version. Once the data task starts, any failure will occupy the entire time allowed.
My failure paths take 5 times as long as the happy path.
This happens in the OCHamcrest version. It even happens in the XCTestExpectation version, despite my liberal sprinkling of XCTFail statements.
When Does a Networking Test Finish?
In some asynchronous tests, there’s no clear trigger to complete the test. The loop will continue until the condition is satisfied, or a timeout occurs. This is the case when you’re trying to observe an indirect result—for example, observing changes in the UI.
But for a networking call, there’s a clear, unambiguous trigger. The code issues a network request. Whenever the completion handler is called, we’re done.
In a test, determining success or failure isn’t the responsibility of the completion handler. That comes later. The completion handler’s job is to move the test along to the verification phase.
In a test, determining success or failure isn't the responsibility of the completion handler.
Fast Failure with XCTestExpectation
Let’s first apply this to the XCTestExpectation approach. Originally, the expectation was:
let promise = expectation(description: "Status code: 200")And this expectation was fulfilled in the completion handler in the following statement:
if statusCode == 200 {
promise.fulfill()
}But now we want the expectation to be fulfilled in either success or failure. So really, the expectation is that the completion handler was invoked. The Act section of the test becomes:
var httpResponse: HTTPURLResponse?
var responseError: Error?
let promise = expectation(description: "Completion handler invoked")
startDataTask(with: validQueryUrl) { data, response, error in
httpResponse = response as? HTTPURLResponse
responseError = error
promise.fulfill()
}
self.waitForExpectations(timeout: 5, handler: nil)Since the completion handler may be invoked with either a response or an error, we capture both. Then the Assert section of the test becomes:
XCTAssertNil(responseError)
XCTAssertEqual(httpResponse?.statusCode, 200)If I synthesize an error in the test code (such as using the wrong domain name), I get a description of the resulting responseError. If I synthesize an error in the production code (creating authentication parameters), I get a status code other than 200. I can look up status codes in the Marvel documentation to determine why the test failed.
Except for the network latency, everything is now significantly faster.
Fast Failure with OCHamcrest
Applying that same logic back to the OCHamcrest version, here’s what we get for the Act and Assert sections:
__block BOOL completionHandlerInvoked;
__block NSHTTPURLResponse *httpResponse;
__block NSError *responseError;
[self startGETRequestToURL:validQueryURL
withCompletionHandler:^(NSData *data, NSURLResponse *response, NSError *error) {
httpResponse = (NSHTTPURLResponse *)response;
responseError = error;
completionHandlerInvoked = YES;
}];
assertWithTimeout(5, thatEventually(@(completionHandlerInvoked)), is(@YES));
assertThat(responseError, is(nilValue()));
assertThat(@(httpResponse.statusCode), is(@200));An explicit trigger is introduced in the completionHandlerInvoked flag. It looks a little funny to see assertWithTimeout in the Act portion. But it’s now serving as the mechanism to re-synchronize an asynchronous test.
I introduce errors in both test code and production code to check the failure speed. It’s so much faster!
The Job of a Test Completion Handler
Let me summarize my new understanding. In any situation where a test is doing actual networking by providing a completion handler…
- Using OCHamcrest, I used to have the completion handler capture the response. An assertWithTimeout would verify the success state.
- Using XCTestExpectation, I used to have the completion handler determine success or failure. Only a success state would fulfill the expectation.
These are functionally equivalent, and slow for test failures. Instead, the job of a test-supplied completion handler is to:
- Capture arguments we want to test
- Trigger the escape flag
Once the timeout loop has been escaped, the rest of the test uses the captured arguments to determine success or failure.
For a single slow test, the difference between 1 second and 5 seconds isn’t much. But if you have 100 of them…
Look through your asynchronous tests. Can you find any that should adopt this approach?
[This post is part of the series TDD Sample App: The Complete Collection …So Far]
Watch out – if your assertion times out but the promise gets fulfilled after your test case completes, the process will abend: https://jeremywsherman.com/blog/2016/03/19/xctestexpectation-gotchas/
Thank you, Jeremy. Yikes!