Java News from Sunday, December 12, 2004

Ivan Moore has released Jester 1.3.6, an interesting unit test tester tool, sadly hobbled by CLASSPATH problems, a poor user interface, and bad error reporting. The basic idea of Jester is that it mutates your source code, for instance by changing an == to a !=, runs all the tests, and sees if any of them report a failure. If none of them do, then the line of code tested is either unnecessary or untested. The results of a Jester run are often eye opening. It's an extremely valuable tool, if you can actually get it to work.

Unfortunately, Jester is quite difficult to configure. First you have to make a copy of your source files, because Jester isn't smart enough to make its own copies before mutating them. This means you also need to make sure any paths are adjusted to the new location of the source files. You can't just copy your existing source tree and build system to a new directory because Jester insists that the source code has to be placed in the same directories as the compiled .class files. Then the CLASSPATH needs to be set up just so. It must be set through an environment variable, not via the -classpath command line argument or the ext directory. And even then the TestTester fails more often than not with no indication to the user of why the program could run. A typical message is "Sun Dec 14 08:09:11 EST 2003 running command 'java jester.TestRunnerImpl nu.xom.tests.XOMTests' resulted in '[..................FAILED]'". Yes, I know it failed. Would it be too much to ask that Jester tell me why it failed? Could it not find the test class? Was some other class missing from the CLASSPATH? Was the source code in the wrong place? What was the exception? What was the stack trace? Anything at all that would help me debug the problem?

Finally, Jester is quite slow. The slowness isn't necessarily poor programming to the Jester code. It's just that it has to make hundreds of modifications in a file and rerun the complete test suite for each one. It's normally helpful to custom tune a test suite jmust for Jester that only contains the fastest tests, and puts the ones most likely to fail right up front. Even doing that, checking just one large file from XOM took 30 minutes on my dual 2.5GHz G5 PowerMac. As a rough estimate, the time to run Jester is approximately N*T/10.0, where N is the number of lines of code and T is the time normally required to run the unit test suite.

Still, despite all this, Jester may be worth the hassle. Jester tends to locate problems other code coverage tools do not. It determines not only whether the tests are executing a particular line of code, but also whether they're actualy testing it. Most code coverage tools are fairly good at finding dead code that can't be reached through the public API. Jester is the only one I've seen that detects pointless code: code that is executed but doesn't actually need to be. The slowness and difficulty of Jester make it hard to believe you could use this on every line of an even moderately sized project. However, it may well be worth your time to use it on the core classes of your application.