What's New in JUnit 4

Elliotte Rusty Harold

Tuesday, September 27, 2005

elharo@metalab.unc.edu

http://www.cafeaulait.org/

Unit Testing is Good for your Health

If I write too much code without tests, I feel physical symptoms, like tightness in my chest. One day it'll be a heart attack, but for now it's garden-variety anxiety. Of course, I'm a bit of a nut.

--J. B. Rainsberger on the junit mailing list, Friday, April 29, 2005

Benefits of Test Driven Development are Well Known

Faster development; faster time to market
More robust, error free code
YAGNI: Writing tests first avoids writing unneeded code.
Easier to find and fix bugs
Easier to add features or change behavior; less worry about unintentionally introducing bugs
Makes refactoring/optimization possible: any change that doesn't break a test suite is de facto acceptable.

TDD does not require any particular framework

JUnit is the preeminent testing frameworK:
Never in the field of software development was so much owed by so many to so few lines of code.
--Martin Fowler
JUnit was developed for Java 1.1
Based on reflection
But there are alternatives: TestNG, JTiger, Artima SuiteRunner, etc.
However JUnit has the broadest adoption
And by far the broadest tool support; Eclipse, IntelliJ, NetBeans, Maven, Ant, XDevelop, BlueJ, PMD, etc.
This has limited uptake of other, more modern tools

Enter JUnit 4

The theme of JUnit 4 is to encourage more developers to write more tests by further simplifying JUnit. To this end, we have followed NUnit's lead in identifying tests by annotation instead of subclassing and naming conventions.

--Kent Beck on the junit mailing list, June 14, 2005

JUnit 4 in one slide

A completely redesigned unit testing framework for Java
Simpler, easier-to-use, easier-to-learn
Based on annotations rather than naming conventions
More flexible initialization and cleanup
Timeouts
Exception tests
Requires Java 5

A Typical JUnit 3 Test Case

import junit.framework.TestCase;

public class AdditionTest extends TestCase {

  private int x = 1;
  private int y = 1;
  
  public void testAddition() {
    int z = x + y;
    assertEquals(2, z);
  }

}

A JUnit 4 Test Case

import org.junit.Test;
import junit.framework.TestCase;

public class AdditionTest extends TestCase {

  private int x = 1;
  private int y = 1;
  
  @Test public void testAddition() {
    int z = x + y;
    assertEquals(2, z);
  }

}

Don't Have to Follow Naming Conventions

import org.junit.Test;
import junit.framework.TestCase;

public class AdditionTest extends TestCase {

  private int x = 1;
  private int y = 1;
  
  @Test public void additionTest() {
    int z = x + y;
    assertEquals(2, z);
  }

}

import org.junit.Test;
import junit.framework.TestCase;

public class AdditionTest extends TestCase {

  private int x = 1;
  private int y = 1;
  
  @Test public void addition() {
    int z = x + y;
    assertEquals(2, z);
  }

}

Don't Have to Extend TestCase

Do have to import org.junit.Assert:

import org.junit.Assert;

public class AdditionTest {

  private int x = 1;
  private int y = 1;
  
  @Test public void addition() {
    int z = x + y;
    Assert.assertEquals(2, z);
  }

}

Makes testing protected methods much easier

Static imports make this even nicer

import static org.junit.Assert.*;
import org.junit.Test;

public class AdditionTest {

  private int x = 1;
  private int y = 1;
  
  @Test public void addition() {
    int z = x + y;
    assertEquals(2, z);
  }

}

Parameterized Test Cases

import org.junit.*;
import static org.junit.Assert.*;

public class AdditionTest {

  @Parameters public static int[][] data 
    = new int[][] {
       {0, 0, 0}, 
       {1, 1, 0}, 
       {2, 1, 1}, 
       {3, 2, 1}, 
       {4, 3, 1}, 
       {5, 5, 0}, 
       {6, 8, -2}
      };
		
  @Parameter(0) public int expected;
  @Parameter(1) public int input1; 
  @Parameter(2) public int input2; 
		
  @Test public void testAddition() {
    assertEquals(expected, add(input1, input2));
  }

  private int add(int m, int n) {
    return m+n;
  }
  
}

Is testAddition really a unit test any more?

Skipping a Test

Tests that can't run because:
- Network is unreachable
- Wrong platform
- Wrong VM version
- Other test prerequisite failures
Just a proposal

    @Test public void setInternalDTDSubsetWithRelativeURLAndCrimson() 
      throws ParsingException, IOException {
        
        XMLReader crimson;
        try {
            crimson = XMLReaderFactory.createXMLReader(
              "org.apache.crimson.parser.XMLReaderImpl");
        } 
        catch (SAXException ex) {
            skip();
        }
        Builder builder = new Builder(crimson);
        Document doc = builder.build("data/outer21.xml");
        String subset = doc.getDocType().getInternalDTDSubset();
        assertEquals(subset, subset.indexOf("file:/"), -1);

    }

Initialization

JUnit 3 test runners invoke the setUp() method before running each test.

For example, from XOM:

protected void setUp() {
        
    System.setErr(new PrintStream(new ByteArrayOutputStream()));
        
    inputDir = new File("data");
    inputDir = new File(inputDir, "xslt");
    inputDir = new File(inputDir, "input");
        
}

Each setUp() method is supposed to invoke its superclass's setUp() method, but this is often forgotten

@Before

@Before annotation identifies a method to run before each test:

@Before protected void initialize() {
        
    System.setErr(new PrintStream(new ByteArrayOutputStream()));
        
    inputDir = new File("data");
    inputDir = new File(inputDir, "xslt");
    inputDir = new File(inputDir, "input");
        
}

Method name is irrelevant

Multiple @Before

@Before protected void findTestDataDirectory() {
    inputDir = new File("data");
    inputDir = new File(inputDir, "xslt");
    inputDir = new File(inputDir, "input");
}
    
@Before protected void redirectStderr() {
    System.setErr(new PrintStream(new ByteArrayOutputStream()));
}

@Before in superclasses are run automatically
@Before in superclasses are run before those in subclasses
Order is otherwise unspecified
Overridden methods are not run.

Cleanup

JUnit 3 test runners invoke the tearDown() method after running each test.

For example, from XOM:

protected void tearDown() {
  doc = null;
  System.gc();   
}

Each tearDown() method should invoke its superclass's tearDown() method, but this is often forgotten

@After

@After annotation identifies a method to run after each test:

@After protected void disposeDocument() {
  doc = null;
  System.gc();   
}

Method name is irrelevant
Multiple methods can be annotated @After, each of which is run after each test method.
@After in superclasses are run automatically
@After in superclasses are run after those in subclasses
Order is otherwise unspecified
Overridden methods are not run.

Class scope initialization and cleanup

A method annotated @BeforeClass will run exactly once before the test methods in that class run.
A method annotated with @AfterClass will run exactly once after all the tests in the class have been run.

Redirect System.err before the tests begin so the output is not cluttered with expected error messages:

// This class tests a lot of error conditions, which
// Xalan annoyingly logs to System.err. 
private PrintStream systemErr;
    
@BeforeClass protected void redirectStderr() {
    systemErr = System.err; // Hold on to the original value
    System.setErr(new PrintStream(new ByteArrayOutputStream()));
}

Restore it after the tests end:

@AfterClass protected void tearDown() {
    // restore the original value
    System.setErr(systemErr);
}

Use judiciously; has the potential to violate the independence of the tests, introduce unexpected coupling, and make tests order dependent

Testing Exceptions

The old way: try-fail-catch

public void testDivisionByZero() {
    
    try {
        int n = 2 / 0;
        fail("Divided by zero!");
    }
    catch (ArithmeticException success) {
        assertNotNull(success.getMessage());
    }
    
}

The new way: @Test(expected=class)

@Test(expected=ArithmeticException.class) public void divideByZero() {
    int n = 2 / 0;
}

Old way is still necessary for testing exception messages and such

Ignoring Tests

@Ignore comments out a test:

// Java doesn't yet support the UTF-32BE and UTF32LE encodings
    @Ignore public void testUTF32BE() 
      throws ParsingException, IOException, XIncludeException {
      
        File input = new File(
          "data/xinclude/input/UTF32BE.xml"
        );
        Document doc = builder.build(input);
        Document result = XIncluder.resolve(doc);
        Document expectedResult = builder.build(
          new File(outputDir, "UTF32BE.xml")
        );
        assertEquals(expectedResult, result);
                
    }

Test runner indicates that tests were skipped:

$ java -classpath .:junit.jar org.junit.runner.JUnitCore nu.xom.tests.XIncludeTest
JUnit version 4.0rc1
.....I..
Time: 1.149

OK (7 tests)

Timeouts

timeout parameter to Test annotation sets maximum time test may take
Useful for simple benchmarking, avoiding slow network servers, excesivly long-running tests
Measured in milliseconds
Test that fails if it takes longer than half a second to find all the elements in a document:
```
@Test(timeout=500) public void retrieveAllElementsInDocument() {
  doc.query("//*");
} 
```

Asserting Array Equality

public static void assertEquals(String message, Object[] expected, Object[] actual) public static void assertEquals(String message, Object[] expected, Object[] actual)

Compares array length; and, if that matches, compares element by element
Handles null
Only works for object arrays, not primitive type arrays

Building and running JUnit 4

No binaries yet
Check out Version4 tag from CVS

org.junit.runner.JUnitCore is the text interface:

$ java -classpath .:junit.jar org.junit.runner.JUnitCore TestA TestB TestC...
JUnit version 4.0rc1

Time: 0.003

OK (0 tests)

Compatible

Forwards compatible: you can run your old test suites in JUnit 4
Backwards compatible: your old test runners can run the new test suites
To enable your JUnit 4 tests to run in JUnit 3 environments, wrap them in a JUnit4TestAdapter

Add this method to your JUnit 4 test class:

public static junit.framework.Test suite() {
  return new JUnit4TestAdapter(TestClassName.class);    
}

Implementation Details

A lot of code is now in the org.junit package.
junit.framework.AssertionFailedError has been replaced by java.lang.AssertionError

Failures

No longer any distinction between errors (unexpected exceptions) and failures (anticipated with asserts)

package org.junit.runner;

public class Failure {

    protected Throwable fThrownException;

    public Failure(Throwable thrownException);
    
    public Throwable getException();
    public String    getTrace();
    public String    getMessage();
    public String    getTestHeader();
    public boolean   isTestFailure();
    public String    toString();
 
}

TestListener

package org.junit.runner;

public interface TestListener {

    void testRunStarted(int testCount);
    void testRunFinished(Runner runner); //TODO: Something is wrong here... 
                                         //Should take an event anyway  
    void testStarted(Object test, String name);
    void testFinished(Object test, String name);
    void testFailure(Failure failure);
    void testIgnored(Object method);

}

XMLTestListener Example

import java.io.*;
import java.lang.reflect.Method;
import org.junit.runner.*;

public class XMLTestListener implements TestListener {

    private Writer out;
    
    public XMLTestListener(OutputStream out) {
        try {
            this.out = new BufferedWriter(new OutputStreamWriter(out, "UTF-8"));
        }
        catch (UnsupportedEncodingException e) {
            throw new RuntimeException("Broken VM does not support UTF-8");
        }
    }
    
    public void testRunStarted(int testCount) {
        try {
            out.write("<?xml version='1.0' encoding='UTF-8'?>\r\n");
            out.write("<testsuite>\r\n");
            out.write("  <properties>\r\n");
            // enumerate properties????
            out.write("  </properties>\r\n");
        }
        catch (IOException ex) {
            throw new RuntimeException(ex);
        }
    }

    public void testRunFinished(Runner runner) {

        double time = runner.getRunTime() / 1000.0; // seconds
        
        try {
            out.write("</testsuite>\r\n");
            out.flush();
        }
        catch (IOException ex) {
            throw new RuntimeException(ex);
        }
        
    }

    public void testStarted(Object test, String name) {

        try {
            out.write("  <testcase classname='" 
                    + test.getClass().getName()
                    + "' name='" + name + ">");
        }
        catch (IOException ex) {
            throw new RuntimeException(ex);
        }
        
    }

    public void testFinished(Object test, String name) {

        try {
            out.write("  </test>\r\n");
        }
        catch (IOException ex) {
            throw new RuntimeException(ex);
        }        
    }

    public void testFailure(Failure failure) {

        try {
            out.write("  <failure>");
            out.write(escape(failure.getMessage()));
            out.write("  </failure>\r\n");
        }
        catch (IOException ex) {
            throw new RuntimeException(ex);
        }
        
    }

    private String escape(String message) {

        String result = message.replaceAll("&", "&amp;");
        result = result.replaceAll("<", "&lt;");
        result.replaceAll(">", "&gt;");
        return result;
    }

    public void testIgnored(Method method) {

        // Why a Method instead of a Ignored????
        try {
            out.write("  <ignored>");
            out.write(escape(method.getName()));
            out.write("  </ignored>\r\n");
        }
        catch (IOException ex) {
            throw new RuntimeException(ex);
        }
        
    }

}

Runner

package org.junit.runner;

public class Runner implements TestNotifier {

    public void run(Class... testClasses);
    public void run(junit.framework.Test test);

    public int           getRunCount();
    public int           getFailureCount() ;
    public long          getRunTime();
    public List<Failure> getFailures();
    public int           getIgnoreCount();
    public boolean       wasSuccessful();
    
    public void addListener(TestListener listener);
    private List<TestListener> getListeners();
    
    public void fireTestStarted(Object test, String name);
    public void fireTestFailure(Failure failure);
    public void fireTestIgnored(Method method);
    public void fireTestFinished(Object test, String name) ;

}

Using Runner to Drive a TestListener

runner.addListener(new XMLTestListener(System.out));

What's Missing

GUI test runner (Use an IDE)
Suite methods (use varargs)
Categories and filters
Documentation

To Learn More

This presentation: http://www.cafeaulait.org/slides/sdbestpractices2005/junit4/
junit@yahoogroups.com: http://groups.yahoo.com/group/junit/
CVS http://cvs.sourceforge.net/viewcvs.py/junit/junit/?only_with_tag=Version4

Index | Cafe con Leche