Functional testing
Integration testing
Exploratory testing
Acceptance testing
System testing
Fuzz testing
Characterization testing
Unit testing
Glass box: testing the nonpublic API
White box: testing the public (and protected) API
Grey box: testing internal components
Black box: testing the external interface; e.g. the GUI
Before
After
During
First
Last
Continuous
Often a developer will want to make some optimizations to a piece of code on which a lot of user-interface behavior depends. Sure, the new table sorter will be ten times faster and more memory efficient, but are you sure the changes won't affect the report generator? Run the tests and find out. A developer is much more likely to run a test that has encapsulated how to set up and test a report than he is to launch the application by hand and try to track down who knows how to do that particular test.
Write a test for a feature
Code the simplest thing that can possibly work
Run the test
Test passed ? goto 1 : goto 2
Have a new feature? Write a test.
Want to change the API? Write a test.
Find a bug? Write a test.
de facto standard unit testing framework
Bundled in most IDEs
http://www.junit.org/
Designed for whitebox unit testing but the framework works pretty well for other forms:
Glassbox, greybox, blackbox
Integration
User Acceptance
etc.
Useful to have one complete test suite the programmers can run with a pass fail result.
Represents a rational number such as 2/3 or 17/5 or -5
What are the basic operations the class needs?
What are the Java utility methods the class needs?
Add
Subtract
Multiply
Divide
Absolute Value
compareTo
Reduce
Reciprocal
Numerator
Denominator
Do we always store fraction in reduced form or not?
This will have a large impact on internal design
Constructor
equals()
hashCode()
toString
toDouble()
parseFraction()
Test methods have names like testFoo
Test results are verified by assertEquals()
Test classes extend junit.framework.TestCase
import junit.framework.TestCase;
public class FractionTest extends TestCase {
public void testAddition() {
Fraction half = new Fraction(1, 2);
Fraction fourth = new Fraction(1, 4);
Fraction actual = half.add(fourth);
Fraction expected = new Fraction(3, 4);
assertEquals(expected, actual);
}
}
public class Fraction {
public Fraction(int numerator, int denominator) {
}
public Fraction add(Fraction fourth) {
return null;
}
public boolean equals(Object o) {
return true;
}
}
public void testAddition2() {
Fraction half = new Fraction(1, 2);
Fraction fourth = new Fraction(1, 2);
Fraction actual = half.add(fourth);
Fraction expected = new Fraction(1, 1);
assertEquals(expected, actual);
}Do not remove test cases unless they're wrong.
More tests are better than fewer.
Do not write new model code unless a test fails first.
public void testNotEquals() {
Fraction half = new Fraction(1, 2);
Fraction fourth = new Fraction(1, 4);
assertFalse(half.equals(fourth));
}
public class Fraction {
private int numerator;
private int denominator;
public Fraction(int numerator, int denominator) {
this.numerator = numerator;
this.denominator = denominator;
}
public Fraction add(Fraction f) {
int denominator = this.denominator * f.denominator;
int numerator = this.numerator * f.denominator + f.numerator * this.denominator;
return new Fraction(numerator, denominator);
}
public boolean equals(Object o) {
Fraction f = (Fraction) o;
return this.numerator / (double) this.denominator
== f.numerator / (double) f.denominator;
}
}For simplicity put junit.jar from the lib directory into a new empty directory.
Put this code into a file named FractionTest.java in that same directory:
import junit.framework.TestCase;
public class FractionTest extends TestCase {
public void testAddition() {
Fraction half = new Fraction(1, 2);
Fraction fourth = new Fraction(1, 4);
Fraction actual = half.add(fourth);
Fraction expected = new Fraction(3, 4);
assertEquals(expected, actual);
}
}Compiling:
$ javac -classpath junit.jar FractionTest.java
Running:
$ java -classpath .:junit.jar junit.swingui.TestRunner FractionTest
Add two tests of your choice
Each test should be a separate test method.
There should now be three tests.
Can anyone find a bug?
Entire test suite should run before each checkin.
All tests are pass-fail
No new features allowed until all tests are passing
Continuous integration
One-button build process
Faster development; faster time to market
More robust, error free code
YAGNI: Writing tests first avoids writing unneeded code.
Easier to find and fix bugs
Easier to add features or change behavior; less worry about unintentionally introducing bugs
Makes refactoring/optimization possible: any change that doesn't break a test suite is de facto acceptable.
Enables experimental programming
Run automatically without any human participation beyond launching the test suite (and maybe not even that)
Results should be pass/fail
Test failures need to be blindingly obvious
Test must be reproducible and consistent
Tests must be fast to enough to run after each change and certainly before every checkin
Failure to adhere to spec? Write a test.
Find a bug? Write a test.
Unspecified behavior? Write a test.
Unit testing requires that each test be completely independent of all other tests.
Not a requirement of all types of tests (functional, integration, user acceptance, etc.) but JUnit is primarily a unit testing framework.
No state shared between methods
All instance data reinitialized before each test run.
Each test is run with a clean, new object.
Test adding two elements to a list.
private LinkedList list = new LinkedList();
public void testAdd() {
String s = "Brooklyn";
list.add(s);
Object o = list.getFirst();
assertEquals(s, o);
}
public void testSizeAfterAddingOneElementIsOne() {
String s = "New York";
list.add(s);
assertEquals(1, list.size());
}Both tests pass!
Order in which tests are run doesn't matter
Expensive initialization slows down tests.
One of these tests will fail:
private static LinkedList list = new LinkedList();
public void testAdd() {
String s = "Brooklyn";
list.add(s);
Object o = list.getFirst();
assertEquals(s, o);
}
public void testSizeAfterAddingOneElementIsOne() {
String s = "New York";
list.add(s);
assertEquals(1, list.size());
}Sometimes the setup can be more complex than you want to put in a constructor or a field initializer
Sometimes you want to reinitialize static data
Sometimes you just want to share setup code between different methods.
setUp method is run before each test:
protected void setUp() {
list = new LinkedList();
super.setUp();
}setUp() should call super.setUp()
Move the shared code into fields.
Write a setUp() method that initializes these fields.
import junit.framework.TestCase;
public class FractionTest extends TestCase {
private Fraction half;
private Fraction fourth;
protected void setUp() {
half = new Fraction(2, 4);
fourth = new Fraction(1, 4);
}
public void testAddition() {
Fraction actual = half.add(fourth);
Fraction expected = new Fraction(3, 4);
assertEquals(expected, actual);
}
public void testSubtraction() {
Fraction actual = half.subtract(fourth);
Fraction expected = fourth;
assertEquals(expected, actual);
}
public void testAddNumberToItself() {
Fraction actual = half.add(half);
Fraction expected = new Fraction(1, 1);
assertEquals(expected, actual);
}
}tearDown method:
protected void tearDown() {
half = null;
fourth = null;
System.gc();
}Rarely necessary
A no-args constructor that calls the superclass no-args constructor
A one-arg constructor that takes a String name and calls the superclass constructor that takes a String name
Add a constructor to your test class.
public FractionTest() {}
public FractionTest(String name) {
super(name);
}First argument to each assertFoo method can be a string that's printed if the assertion fails:
public void testSubtraction() {
Fraction actual = half.subtract(fourth);
Fraction expected = fourth;
assertEquals("Could not subtract one fourth from two fourths",
expected, actual);
}Add assertion messages to each test.
import junit.framework.TestCase;
public class FractionTest extends TestCase {
private Fraction half;
private Fraction fourth;
protected void setUp() {
half = new Fraction(2, 4);
fourth = new Fraction(1, 4);
}
public void testAddition() {
Fraction actual = half.add(fourth);
Fraction expected = new Fraction(3, 4);
assertEquals("Could not add 1/2 to 1/4", expected, actual);
}
public void testSubtraction() {
Fraction actual = half.subtract(fourth);
Fraction expected = fourth;
assertEquals("Could not subtract 1/4 from 1/2",expected, actual);
}
public void testAddNumberToItself() {
Fraction actual = half.add(half);
Fraction expected = new Fraction(1, 1);
assertEquals("Could not add 1/2 to itself using same object",
expected, actual);
}
}As always with floating point arithmetic, you want to avoid direct equality comparisons.
public static void assertEquals(double expected, double actual, double tolerance)
public static void assertEquals(float expected, float actual, float tolerance)
public static void assertEquals(String message, double expected, double actual, double tolerance)
public static void assertEquals(String message, float expected, float actual, float tolerance)
Write a test for the double value of 5/3.
Implement the doubleValue() method.
public void testDoubleValue() {
Fraction a = new Fraction(5, 3);
assertEquals(5.0/3.0, a.doubleValue(), 0.000000001);
}Straight-forward because integer comparisons are straight-forward:
public static void assertEquals(boolean expected, boolean actual)
public static void assertEquals(byte expected, byte actual)
public static void assertEquals(char expected, char actual)
public static void assertEquals(short expected, short actual)
public static void assertEquals(int expected, int actual)
public static void assertEquals(long expected, long actual)
public static void assertEquals(String message, boolean expected, boolean actual)
public static void assertEquals(String message, byte expected, byte actual)
public static void assertEquals(String message, char expected, char actual)
public static void assertEquals(String message, short expected, short actual)
public static void assertEquals(String message, int expected, int actual)
public static void assertEquals(String message, long expected, long actual)
Write a test for the numerator and proper numerators of 5/3
That is, write tests for getNumerator() and getProperNumerator()
The "proper numerator" is the 2 in the 1 and 2/3 form of 5/3. The proper numerator is always positive. (Is zero a boundary condition?)
public void testGetNumerator() {
Fraction a = new Fraction(5, 3);
assertEquals(5, a.getNumerator());
}
public void testGetProperNumerator() {
Fraction a = new Fraction(5, 3);
assertEquals(2, a.getProperNumerator());
}How many edge conditions can we diagnose for getProperNumerator alone?
Each of these could be a separate unit test, but that's tedious.
Store input and expected output in matching arrays and write one test to iterate through them.
Even better: store the input and output in a text or XML file.
Working in teams of 4, develop a list of different fraction pairs likely to expose bugs and cover the problem space. Then write a parameterized test that loads this data to test addition, multiplication, subtraction, and division.
This really isn't a unit test any more, but it's still useful.
Good assertion messages are critical to identify the actual failing test
But these can slow down the tests
One failing test masks subsequent test failures.
If any bugs show up, those test cases can be pulled out into explicit individual tests for debugging.
Advanced trick: generate source code for the tests from the input data
Sometimes the integration tests find bugs the unit testing can miss. In particular, unitarity can mask reinitialization failures:
This variant now runs all tests even if one test fails:
public void testIntegrationGetProperNumerator() {
int failures = 0;
StringBuffer failureMessage = new StringBuffer();
for (int i = 0; i < data.length; i++) {
int[] row = data[i];
Fraction f = new Fraction(row[0], row[1]);
try {
assertEquals(row[2], f.getProperNumerator());
}
catch (AssertionFailedError err) {
failures++;
failureMessage.append(err.getMessage() + "\n");
}
}
if (failures > 0) fail(failureMessage.toString());
}Same test with much different data.
Pass the test data and the test method name into the test case constructor
Instantiate multiple different instances of the test case in the suite method
import junit.framework.*;
import junit.textui.TestRunner;
public class DataDrivenTest extends TestCase {
private int numerator;
private int denominator;
private int result;
public DataDrivenTest(int numerator, int denominator, int result) {
super("testIntegrationGetProperNumerator");
this.numerator = numerator;
this.denominator = denominator;
this.result = result;
}
public static TestSuite suite() {
TestSuite result = new TestSuite();
result.addTest(new DataDrivenTest(5, 3, 2));
result.addTest(new DataDrivenTest(-5, 3, 2 ));
result.addTest(new DataDrivenTest(0, 3, 0));
result.addTest(new DataDrivenTest(6, 2, 0));
return result;
}
public void testIntegrationGetProperNumerator() {
Fraction f = new Fraction(numerator, denominator);
assertEquals("Testing " +
numerator + "/" + denominator, result, f.getProperNumerator());
}
}Generate much data through random choice.
Store this data
Figure out what the answers should be (ideally with independent code).
Use this to create many more tests.
Generate much data through random choice.
Store this data
Verify that addition inversts subtraction and vice versa
Verify that multiplication inverts subtraction versa
Write a program that randomly generates a thousand different test, and uses round tripping to test multiplication, addition, subtraction, and division.
assertEquals uses the equals() method:
public void testAEqualsA() {
Fraction a = new Fraction(17, 56);
assertEquals(a, a);
}
public void testAEqualsA2() {
Fraction a = new Fraction(17, 56);
Fraction b = new Fraction(17, 56);
assertEquals(a, b);
}assertSame
and assertNotSame
use the == operator:
public void testAIsA() {
Fraction a = new Fraction(17, 56);
assertSame(a, a);
}
public void testAIsNotA2() {
Fraction a = new Fraction(17, 56);
Fraction b = new Fraction(17, 56);
assertNotSame(a, b);
}Does the last test go too far? is that really part of the class's contract?
For general boolean tests, one can assert that some condition is true
public static void assertTrue(boolean condition)
public static void assertTrue(String message, boolean condition)
Test superclass:
public void testSuperclass() {
assertTrue(fourth instanceof Number);
}Can also assert that some condition is false
public static void assertFalse(boolean condition)
public static void assertFalse(String message, boolean condition)
public static void assertNull(Object o)
public static void assertNull(String message, Object o)
e.g. testing for memory leak:
public void testMemoryLeak() throws InterruptedException {
Fraction a = new Fraction(1, 2);
WeakReference ref = new WeakReference(a);
a = null;
System.gc();
System.gc();
System.gc();
Thread.sleep(1000);
assertNull(ref.get());
}public static void assertNotNull(Object o)
public static void assertNotNull(String message, Object o)
public void testToString() {
assertNotNull(list.toString());
}public static void fail()
public static void fail(String message)
Runtime exceptions cause tests to fail
Different than a test failure
But do not shutdown the VM or stop the other tests
Unexpected checked exceptions must be declared to be thrown from the test method
Otherwise treated the same as runtime exceptions
Errors are treated the same as a runtime exception
try
fail()
catch
public void testZeroDenominator() {
try {
new Fraction(1, 0);
fail("Allowed zero denominator");
}
catch (ArithmeticException success) {
}
}
Can you add null to a Fraction? What happens if you do?
i.e.
Is an exception thrown?
If so which one?
Or is null treated as zero?
Or is null returned?
Something else?
Write a test that specifies the answer to this question, and then adjust the behavior to match it.
Is this behavior properly documented? If not, it's a bug.
public void testAddNull() {
Fraction a = new Fraction(5, 3);
try {
Fraction f = null;
a.add(null);
fail("added null");
}
catch (NullPointerException success) {
assertNotNull(success.getMessage());
}
}Test data can be stored alongside with the test cases using relative paths
Sometimes the files can be created on the fly in a temporary directory
Write sample data onto a ByteArrayOutputStream.
Convert to byte array
Read from a ByteArrayInputStream
Write onto a ByteArrayOutputStream
Test directly or convert to string.
Write a test that serializes and deserializes a Fraction object; That is:
Create a Fraction
Create a ByteArrayOutputStream
Chain an ObjectOutputStream to the ByteArrayOutputStream
Write the Fraction object onto the ObjectOutputStream using writeObject()
Close the ObjectOutputStream
Get a byte array from the ByteArrayOutputStream using toByteArray()
Make a ByteArrayInputStream from the byte array
Chain an ObjectInputStream to the ByteArrayInputStream
Read the object from the ObjectInputStream using readObject()
Verify that the object is equal to but not the same as the original object
public void testSerialization() throws IOException, ClassNotFoundException {
Fraction source = new Fraction(1, 2);
ByteArrayOutputStream out = new ByteArrayOutputStream();
ObjectOutputStream oout = new ObjectOutputStream(out);
oout.writeObject(source);
oout.close();
byte[] data = out.toByteArray();
ByteArrayInputStream in = new ByteArrayInputStream(data);
ObjectInputStream oin = new ObjectInputStream(in);
Fraction result = (Fraction) oin.readObject();
assertEquals(source, result);
assertNotSame(source, result);
}Generally more than one test class per project
Often one (or more ) test classes per tested class
Multiple test classes can be combined in a test suite class:
public class FunctionTests {
public static Test suite() {
TestSuite result = new TestSuite();
result.addTest(new TestSuite(TranslateFunctionTest.class));
result.addTest(new TestSuite(SubstringTest.class));
result.addTest(new TestSuite(SubstringBeforeTest.class));
result.addTest(new TestSuite(SubstringAfterTest.class));
result.addTest(new TestSuite(LangTest.class));
result.addTest(new TestSuite(LastTest.class));
result.addTest(new TestSuite(ConcatTest.class));
result.addTest(new TestSuite(ContainsTest.class));
result.addTest(new TestSuite(StringLengthTest.class));
result.addTest(new TestSuite(StartsWithTest.class));
result.addTest(new TestSuite(CountTest.class));
result.addTest(new TestSuite(LocalNameTest.class));
result.addTest(new TestSuite(NameTest.class));
result.addTest(new TestSuite(NamespaceURITest.class));
result.addTest(new TestSuite(SumTest.class));
result.addTest(new TestSuite(NumberTest.class));
result.addTest(new TestSuite(RoundTest.class));
result.addTest(new TestSuite(StringTest.class));
result.addTest(new TestSuite(BooleanTest.class));
result.addTest(new TestSuite(CeilingTest.class));
result.addTest(new TestSuite(FloorTest.class));
result.addTest(new TestSuite(IdTest.class));
result.addTest(new TestSuite(TrueTest.class));
result.addTest(new TestSuite(FalseTest.class));
result.addTest(new TestSuite(NotTest.class));
result.addTest(new TestSuite(NormalizeSpaceTest.class));
return result;
}
}Multiple test methods can also be combined in a test suite class:
public class JaxenTests {
public static Test suite() {
TestSuite result = new TestSuite();
result.addTest(SAXPathTests.suite());
result.addTest(FunctionTests.suite());
result.addTest(CoreTests.suite());
result.addTest(DOMTests.suite());
result.addTest(JDOMTests.suite());
result.addTest(DOM4JTests.suite());
result.addTest(XOMTests.suite());
result.addTest(JavaBeanTests.suite());
result.addTest(PatternTests.suite());
result.addTest(BaseTests.suite());
result.addTest(HelpersTests.suite());
result.addTest(ExprTests.suite());
result.addTest(UtilTests.suite());
return result;
}
}Public fields are part of the API and need to be tested too.
Often these are constant.
Test that they're equal to the expected value:
public void testOneHalfField() {
Fraction a = new Fraction(1, 2);
assertEquals(a, new Fraction(1, 1)_HALF);
}Better to test through the public API
But can put test classes in same package as tested API
Use different src directories for separation
Test through the public API if at all possible
Delete unreachable private methods
The reflection API and setAccessible()
Template methods to test implementations
Especially important if you expect multiple independent implementations
Test that what should happen, does happen
Test that what should not happen, doesn't
All public APIs
All protected APIs
Anything that is promised not to change
Private API
Package protected API?
Localized strings
Anything that is not promised
Three are included in the Framework:
Swing: junit.swingui.TestRunner
AWT: junit.awtui.TestRunner
Text: junit.textui.TestRunner
You can write your own
Use the one of the test runners to run the current class:
public static void main(String[] args) {
junit.textui.TestRunner runner = new junit.textui.TestRunner();
runner.run(FractionTest.class);
}Add a main method to your test class and use it to run the tests.
public static void main(String[] args) {
TestRunner.run(FractionTest.class);
}public interface TestListener {
void addError(Test test, Throwable t);
void addFailure(Test test, AssertionFailedError t);
void endTest(Test test);
void startTest(Test test);
}Ant
Maven
Eclipse
<property name="test.outputFormat" value="xml"/>
<target name="test" depends="compile"
description="Run JUnit tests using command line user interface">
<junit printsummary="off" fork="yes">
<classpath refid="test.class.path" />
<formatter type="${test.outputFormat}" />
<batchtest fork="yes" todir="${testoutput.dir}">
<fileset dir="${build.src}">
<include name="**/*Test.java" />
<exclude name="**/pantry/*" />
<exclude name="**/MegaTest.java" />
<exclude name="**/benchmarks/*.java" />
<exclude name="**/EBCDICTest.java" />
</fileset>
</batchtest>
</junit>
<junitreport todir="${testoutput.dir}">
<fileset dir="${testoutput.dir}">
<include name="TEST-*.xml" />
</fileset>
<report todir="${testoutput.dir}" />
</junitreport>
</target> <build>
<nagEmailAddress>dev@jaxen.codehaus.org</nagEmailAddress>
<sourceDirectory>src/java/main</sourceDirectory>
<unitTestSourceDirectory>src/java/test</unitTestSourceDirectory>
<!-- Unit test classes -->
<unitTest>
<includes>
<include>**/*Test.java</include>
</includes>
<excludes>
<!-- currently broken -->
<exclude>org/jaxen/jdom/XPathTest.java</exclude>
</excludes>
</unitTest>
</build>Test classes and methods must be public.
However they are generally not published.
Put the tests in a separate package, and don't include that package when producing JavaDoc.
Fixtures can do too much work.
Push fields into local variables.
Break up test classes into smaller classes that share the same fixtures
How much coverage is enough?
Cobertura
Local Databases
Mock Objects
URL Class
HttpUnit
Local Servers
Mocking Servers
Pragmatic Unit Testing in Java with JUnit
by Andy Hunt and Dave Thomas
JUnit Recipes by J. B. Rainsberger; Manning, 2004; ISBN 1932394230
This presentation: http://www.cafeaulait.org/slides/catt/junit/
JUnit: http://www.junit.org/
JUnit mailing list: http://tech.groups.yahoo.com/group/junit/