A published paper about test-driven development (TDD) included the following diagram. Unfortunately, it gets some things wrong…
A tweet from Nat Pryce sparked the discussion:
Grumpy request to academics: if you're going to publish ideas about how to improve TDD, get the original process right! pic.twitter.com/FaSU8CF6ol
— Nat Pryce (@natpryce) September 7, 2017
First, let me say I’m happy to see more studies on TDD. The thrust of this particular study is that TDD can be soft on negative tests. That is, maybe the code works for good data, but it’ll break on bad data.
TDD is a development discipline, so I’m all for learning more from traditional testing disciplines. I certainly don’t want to discourage folks from doing studies and writing papers.
But. Let’s first make sure we’re doing proper TDD, shall we? Otherwise any studies, especially studies about efficacy, may be flawed.
The Importance of a Failing Test
The center of this diagram is “Add a new Test Case” and “Execute all Test Cases.” Curiously, if all tests pass, try adding another test case.
But when you add a new test case in TDD, you expect it to fail. If a new test passes, surprise! That could mean one of several things:
- The test isn't written correctly. This is the usual cause. Time to dig deeper. (See the tweet below.)
- The production code got ahead of the tests. In other words, we violated the 3 Laws of TDD. Conduct a mutation test. That is, see if you can temporarily change the production code so that the test fails.
- The test adds valuable documentation. Then keep it. This isn't TDD, but that's okay. …So are there other tests that aren’t as clear, which we can delete?
Actual process: Add new test case > Run tests > Test passes > OMG NO! THIS SHOULD DEFINITELY FAIL! *spend hours trying to make it fail* pic.twitter.com/mr1GnuxIFh
— Bonnie Aumann (@bonniea) September 7, 2017
The Centrality of Refactoring
The location of the “Code Refactoring” node is a major problem. In the diagram, it looks like refactoring occurs after you’ve finished creating your suite of tests.
But refactoring isn’t something you do at the end. It’s step 3 of the TDD Waltz. Each part of the waltz is an action with fast feedback.
I tell my students, “Refactoring is the secret sauce of TDD.” It’s what the tests empower: the ability to fearlessly change the design of the code. But it’s a continuous process, not one shot at the end.
And where the diagram is careful to include “Execute all Test Cases” after “Add a new Test Case” and “Make minimal code changes,” it strangely omits it from refactoring.
(And don’t forget to refactor test code, too.)
Incorrect TDD Aside: What About Testing for Robustness?
Like I said, I want to see more studies on TDD, not less. I applaud the focus of this particular paper: we should give thought to tests for robustness, not just basic functionality.
Part of having more robust tests is learning to question assumptions about input. You can see me do this at the end of my screencast on JSON parsing.
At the same time, robust code is important mainly for handling input from external sources from outside a boundary. Making bulletproof code may be wasted effort. Context matters.
There’s also a question of improving our tooling:
- I have yet to play with SwiftCheck property-based testing. Have you tried it?
- Is there a Swift version of AutoFixture?
- There’s an automated mutation testing for Swift called Muter. Have you tried it?
So through discipline or tooling, we can make our code more robust. But we can do this without sacrificing the basic principles of TDD.
Have you encountered malformed TDD? Have you TDD’d something that turned out not to be robust? Or can you share any information about new tools? Please leave a comment below.
Have you encountered malformed TDD?
Thanks for bringing this to my attention! Good material for TDD/testing discussions in the future. Also disn’t know about property testing at all; can you explain that ins a future blog post?
Sorry for the typos, was too sloppy proofing the comment ?
Haha, no worries about typos, Christian. (If you like, I can clean them up.)
I’m afraid I have no experience with property-based testing yet. Just like anything else, I’m sure there are ways to write useful and not-so-useful property tests. That’s why I’m curious to learn from anyone who has already used SwiftCheck.
I have since written one property-based test. The test constructs random input that fits a certain “shape.” In my case, I think I needed to test currency conversion. The test then exercises the system using that not-quite-random input, and knows what the output should be.
Since it’s only one such test, I didn’t use SwiftCheck. I just wrote it long-hand in XCTest.