Incorrect TDD: What Can We Learn From It? 

A published paper about test-driven development (TDD) included the following diagram. Unfortunately, it gets some things wrong…

A tweet from Nat Pryce sparked the discussion:

First, let me say I’m happy to see more studies on TDD. The thrust of this particular study is that TDD can be soft on negative tests. That is, maybe the code works for good data, but it’ll break on bad data.

TDD is a development discipline, so I’m all for learning more from traditional testing disciplines. I certainly don’t want to discourage folks from doing studies and writing papers.

But. Let’s first make sure we’re doing proper TDD, shall we? Otherwise any studies, especially studies about efficacy, may be flawed.

The Importance of a Failing Test

The center of this diagram is “Add a new Test Case” and “Execute all Test Cases.” Curiously, if all tests pass, try adding another test case.

But when you add a new test case in TDD, you expect it to fail. If a new test passes, surprise! That could mean one of several things:

• The test isn't written correctly. This is the usual cause. Time to dig deeper. (See the tweet below.)
• The production code got ahead of the tests. In other words, we violated the 3 Laws of TDD. Conduct a mutation test. That is, see if you can temporarily change the production code so that the test fails.
• The test adds valuable documentation. Then keep it. This isn't TDD, but that's okay. …So are there other tests that aren’t as clear, which we can delete?

The Centrality of Refactoring

The location of the “Code Refactoring” node is a major problem. In the diagram, it looks like refactoring occurs after you’ve finished creating your suite of tests.

But refactoring isn’t something you do at the end. It’s step 3 of the TDD Waltz. Each part of the waltz is an action with fast feedback.

I tell my students, “Refactoring is the secret sauce of TDD.” It’s what the tests empower: the ability to fearlessly change the design of the code. But it’s a continuous process, not one shot at the end.

And where the diagram is careful to include “Execute all Test Cases” after “Add a new Test Case” and “Make minimal code changes,” it strangely omits it from refactoring.

(And don’t forget to refactor test code, too.)

Incorrect TDD Aside: What About Testing for Robustness?

Like I said, I want to see more studies on TDD, not less. I applaud the focus of this particular paper: we should give thought to tests for robustness, not just basic functionality.

Part of having more robust tests is learning to question assumptions about input. You can see me do this at the end of my screencast on JSON parsing.

At the same time, robust code is important mainly for handling input from external sources from outside a boundary. Making bulletproof code may be wasted effort. Context matters.

There’s also a question of improving our tooling:

• I have yet to play with SwiftCheck property-based testing. Have you tried it?
• Is there a Swift version of AutoFixture?
• There’s an automated mutation testing for Swift called Muter. Have you tried it?

So through discipline or tooling, we can make our code more robust. But we can do this without sacrificing the basic principles of TDD.

Have you encountered malformed TDD? Have you TDD’d something that turned out not to be robust? Or can you share any information about new tools? Please leave a comment below.