As mentioned over two months ago, I’ll be giving two talks this weekend at the Voices That Matter: iPhone Developers Fall conference. I’m feeling good about both of the talks that I’ve worked on, though I definitely think the Unit Testing talk on Saturday is better polished. That’s probably at least in part because of the work put in by everyone’s favourite Apple Employee, who gave me a great base for the talk.
One of the (myriad) problems I’m currently facing is related to unit testing, in a project I’ve recently started working on. The existing code base works well but, how can we put this delicately, it smells. There aren’t any important bugs, but when you come to extend some functionality or add a new feature you find yourself trying to piece together code from numerous classes in different libraries, before you can tell the story of how the bit you’re trying to change works. You then load the shotgun with the lead shot of enhanced featurity, which is liberally spattered across the code base.
There are various refactoring exercises that can be brought to bear on code such as this, or there could be if it weren’t for the unit tests. Some of the tests fail. Sometimes you look and find a hard-coded path to a file that can quickly be changed to get us going again. Sometimes you see a bit of expected value drift. Every so often, you find that the class under test and the test suite bear no resemblance to one another. Why are these tests failing, why did nobody notice, and what is the behaviour of the code under test?
The fact is usually that at some point, someone needed to make a change to the application and, believing it to be a “quick fix”, decided it wasn’t worth doing TDD for the new code. Whether or not that was a good decision, the new code had to interface with the old code somewhere and it changed the behaviour of that old code. Tests started failing. The developer decided that the test should be ignored. Worst. Decision. Ever.
Ignoring a test means leaving it in the code, but marking it up in some way such that the test runner either doesn’t execute the test or doesn’t care if it fails. Concrete example: in NUnit, you can set the [Ignore] attribute on a test method and it won’t get run. You can’t actually do this in OCUnit, though you can get per-class ignorance by just removing the class from the test bundle target.
The problem with ignored tests is that they don’t look much different from non-ignored tests (sometimes, they can look identical, if the ignorance is provided by a parameters file rather than the source code). They give the impression that the code is under test. They give the impression that the code should work in the way specified in the test. They give the impression, if you run the tests and they fail, that you broke something. Turning off a test is worse than useless – just delete the thing. That way it’s obvious that the code it was testing is no longer under test. And don’t worry, it’s not gone forever – you could always retrieve the deleted code from version control. You never will, but you could.
“But,” pipe up the authors of the xUnit ignore features, “that’s not what the ignore feature is for. You write tests for code that doesn’t exist yet, and set them ignored so they don’t break the build. You get the utility of annotating your code with tests, without the red bar.”
Unfortunately it doesn’t work like that. I write the tests now, and set them to be ignored. Some time later, the feature gets cancelled. The ignored tests hang around, bearing less and less resemblance to the real application.
Or I write the tests now, and set them to be ignored. Some time later, somebody else implements the feature, writing their own tests (or not). The ignored tests hang around, looking uncomfortably like they might be testing the feature.
Now using unit tests as a documentation and communication tool is useful and beneficial, and I’ll talk about that at the weekend. But only commit tests for code that either got written with the tests, or will be written imminently. If it looks like a test is no longer relevant, rewrite it or delete it. But do not tell your test runner to ignore it, because your developers probably won’t.
