Java News from Monday, November 7, 2005

Cenqua has released Clover 1.3.11, a $250 payware unit test coverage tool. 1.3.11 is a bug fix release. Clover modifies the source code to enable it to follow which statements are executed when, and keeps a running count of how many times each statement is executed during the test suite. Any statement that executes zero times is not being tested. I use Clover with Ant, but there are also versions for NetBeans, Eclipse, IntelliJ IDEA, and Oracle JDeveloper 10g. Clover can generate test coverage reports in XML, HTML, PDF, or via a Swing Viewer. Java 1.2 or later is required.

As usual I tested this release on XOM. Some hard to find bugs had recently been reported in XML canonicalization, so I looked there. Overall code coverage of this package was 99%+, but there were a couple of uncovered lines and branches, so I wrote some tests to try to reach them. This didn't find the bug I was looking for, but it did show me a different bug that was completely unexpected. Bottom line: if it isn't tested, it is buggy.

Generally I avoid mock objects and writing tests to the non-published interfaces. I much prefer to write tests that use the real code path rather than some faked code path. When you mock the input so you're really just testing one method and nothing else, then you're only verifying that the method behaves under the conditions you think will occur. When you use the real path into a method with the real objects, you're much more likely to be testing what will actually happen in running code. Half the time the bugs aren't inside the method you're testing. They can be outside the method, but reveal themselves when testing the method. They can also be violations of your assumptions about preconditions and postconditions for the method.

Too many test suites are too unitary, and thus tend to miss bugs that would be caught by more complete, wide-reaching tests. I'm not at all convinced that unit tests and integration tests are or should be as separate as developers often think. For example, a very non-unitary XSLT conformance test suite test (the entire test suite runs in one method) recently found a nasty bug in XOM that could violate well-formedness. A standard test-first unit test would never have found the bug because it only arose when two documents were parsed in sequence. A typical unit test would only have parsed and tested one doucment at a time, then torn down and rebuilt the parser and its associated data structures before parsing the next one. However, the bug only occurred when parsers were reused, on the second or later parses, never on the first. There were some other conditions that needed to be met before the bug was exposed as well, so even a specific test for parser reuse could have missed it; but testing a thousand different documents in a row in one test method showed the bug.