Test Driven Web Applications with FitNesse

Elliotte Rusty Harold

Thursday, September 14, 2006

elharo@metalab.unc.edu

http://www.cafeaulait.org/

Test Driven Development

Write a test for a feature
Code the simplest thing that can possibly work
Run the test
Test passed ? goto 1 : goto 2

Test Driven Development

Have a new feature? Write a test.
Want to change the API? Write a test.
Find a bug? Write a test.

Benefits of Test Driven Development are Well Known

Faster development; faster time to market
More robust, error free code
YAGNI: Writing tests first avoids writing unneeded code.
Easier to find and fix bugs
Easier to add features or change behavior; less worry about unintentionally introducing bugs
Makes refactoring/optimization possible: any change that doesn't break a test suite is de facto acceptable.

Framework Limitations

JUnit/NUnit/etc. are not ideal for testing web apps.
Even with extensions like HTTPUnit, XMLUnit, etc.
Too low-level
Require programmer skills many web app developers don't have (e.g. Java/C# vs. JavaScript/PHP)

Enter FitNesse

A WIKI designed for acceptance testing.
Adds WIKI markup for tests
Includes a built-in web server (nearly zero-conf)
Open source
Written in Java 1.4
Sits on top of Ward Cunningham's Fit Framework

Acceptance Testing vs. Unit Testing

In theory:
- - Unit tests are written by programmers.
  - Acceptance tests are written by customers
- - When unit tests pass, write another test that fails.
  - When acceptance tests pass, stop coding. The job is done.
In reality:
- The difference is not so clear-cut; and we can often use the same tools for either or both kinds of tests.
- Acceptance tests become future unit tests; and vice versa

Demo Installing FitNesse

Download the latest binary zip file (20060719 at the time of this writing) from http://fitnesse.org/FitNesse.DownLoad
Unzip fitnesse20060719.zip to create a folder named "fitnesse"

Run run.sh (Unix) or run.bat (Windows) in the fitnesse folder. This launches a web server on port 80:

       FitNesse (20060719) Started...
        port:              80
        root page:         fitnesse.wiki.FileSystemPage at ./FitNesseRoot
        logger:            none
        authenticator:     fitnesse.authentication.PromiscuousAuthenticator
        html page factory: fitnesse.html.HtmlPageFactory
        page version expiration set to 14 days.

Point your browser at http://localhost

Install Problems

If there's already a server on port 80, you can run it on a different port:
java -cp fitnesse.jar -p 8080 fitnesse.FitNesse
Unix requires root privileges to run on ports below 1024 so you should probably run it on 8080 or some such anyway.
On Unix you'll probably need to chmod +x or use source to run it because zip files don't preserve Unix permissions.

Live Demo!

Two-minute example
Bug workaround: In Fitnesse 2006 add this line to the page before running the tests:
```
!path fitnesse.jar
```

Possible Results

Green: Test Passed
Red: Test Failed
Yellow: test threw an exception (could be a config error; could be an exception from the software under test)
Grey background: Input data (not a test result)
Grey text: Answer provided by model software for user inspection

The FitNesse Architecture

FitNesse runs on one server where users write tests
Each test table references one fixture class that connects the table to the real application
If the application is a web application, it usually runs on a different server (or at least a different port)

The Fixture is where the Magic Is

// Copyright (C) 2003,2004,2005 by Object Mentor, Inc. All rights reserved.
// Released under the terms of the GNU General Public License version 2 or later.
package eg;

import fit.ColumnFixture;

public class Division extends ColumnFixture
{
  public double numerator;
  public double denominator;
  public double quotient() {
    return numerator/denominator;
  }
}

Compare to the Wiki markup

|eg.Division|
|numerator|denominator|quotient?|
|10       |2          |5        |
|12.6     |3          |4.2      |
|100      |4          |24       |

First line of the table references the adapter class using a fully package qualified class name.
Input data are public fields?!
Result (suffixed with a question mark) is a method.

Don't read too much into this

This is an unusual example. Normally the fixture only invokes the model code. It is not itself the model code being tested.
For web apps, the fixture is going to need to talk to the web server where the application runs.

A (Marginally) More Realistic Example

http://localhost:8080/FitNesse.FixtureCode

Types

The HTML table just has strings
Primitive data types (double, boolean, etc.) are converted automatically
Other types require toString() methods for output or parse() methods for input.

toString() is standard. parse() method in the fixture class looks like this:

  public Object parse(String s, Class type) throws Exception {
    if (type.equals(CustomClass.class)) {
      return parseCustomClass(s);
    }
    else return super.parse(s, type);
  }

  private static Object parseCustomClass(String s) {
   // code to convert
  }

You can invoke any method in the model class, the fixture class, or some other class to convert the String to the object as convenient. Here I've just made it a static method in the fixture class.

Why the Indirection?

The goal is to allow business users and non-programmer testers who can use a web browser but not a compiler to write tests.
Programmers are still necessary to write the application code and the fixture that connects the application code to the WIKI.

Setting the Classpath

Java code needs to be added to FitNesse's classpath.
Use a !path directive
Classpaths are relative to the fitnesse directory

!path *.jar
!path lib/*.jar
!path classes
!path lib

Importing spreadsheets

The business world runs on Excel
Business users love Excel
Write tables in Excel and upload them to FitNesse

Table Fixture Styles

ColumnFixture: Rows contains input and expected output. Tests are executed in order.
RowFixture: An exact set of order-independent results
ActionFixture: A script that imitates user actions
Comment: just a comment, not a test
RowEntryFixture: for entering data into a database
TableFixture: custom test
CommandLineFixture: execute shell script
SummaryFixture: List all the tests on a page

HTMLFixture

Joseph Bergin's extension for testing Web pages
Based on HTMLUnit and DOM
http://dps.csis.pace.edu:8077/FixtureDevelopment.HtmlFixture

A simple web application

Creating a new page

Load http://localhost:8080/FitNesse.TestPetition into your browser
Or http://localhost:8080/SoftwareDevelopment2006.TestPetition
Click "create this page"
It won't create any URL, only root-level URLs that begin with a "WikiWord"
Or a hierarchy of WikiWords separated by periods
"A WikiWord starts with a capital letter and has at least one more capital letter in it. Between the capitals there must be lower case letters or numbers. There cannot be two capitals in a row."

Identifying tests

FitNesse looks for tests on pages whose WIKI name begins with "Test"
If it finds any, it adds a Test button to the side bar you press to run the tests.
You can put more than one test on a page.

Test that the Page has a Title

This page is the first test of the online Petition

Let's test that the page has a title:

!path fitnesse.jar
!path htmlunit-1.5/lib/*.jar
!path htmlfixture20050422.jar

!|com.jbergin.HtmlFixture|
|http://www.cafeconleche.org/petition.phtml|
|Element Focus|t1|title|

Initial Result

You want the test to fail first:

Fix the problem

Upload the file and then...
It still fails!
This is not uncommon in TDD. Often the tests are wrong. That's what's happened here.
The test is looking in the wrong directory. Fix that and then...

It still fails

Results
This is a code bug; no id for the title
Fix that and then...

Now the test passes!

Results

Test that the Page has a Title and that the title is "Petition"

This page is the first test of the online Petition

Let's test that the page has a title:

!path fitnesse.jar
!path htmlunit-1.5/lib/*.jar
!path htmlfixture20050422.jar

!|com.jbergin.HtmlFixture|
|http://www.cafeconleche.org/petition.phtml|
|Element Focus|title|title|
|Text|Petition|

The general structure of an HTMLFixture test table


com.jbergin.HtmlFixture

URL

Instruction Name
Instruction arguments
Instruction Name
Instruction arguments
Instruction Name
Instruction arguments
...

HTMLFixture Instructions

Element Focus: test or retrieve a given element and make this element the focus of following instructions
Element: test or retrieve a given element
Elements: test or retrieve the number of child elements
Types: assert or retrieve the number of descendants with the specified element name
Type: like Element but depth first instead of breadth first
Type Focus: like Element Focus but depth first instead of breadth first
Attribute: test or retrieve a given attribute
Focus: set the context for the following commands
Focus Relative: set the context for the following commands relative to the current focus
Text: assert the exact equality of the text of the current element
Has Text: assert the substring equality of the text of the current element
Matches Text: assert the regular expression equality of the text of the current element
Execute: a script
Set Value: "type" text into a form element
Submit: follow a link or submit a form
Click: synonym for Submit
Save: assign the current focus to a named variable for later reference
Preserve: do not clear variables when submitting
Clear: unbind all variables
Nodes: assert or retrieve the number of child nodes (as opposed to elements) of the focused element
Node: assert or retrieve the type of the specified node
Fail: boolean not; i.e. reverse the sense of a test

Addressing

Elements in the document you're aiming at need id attributes with unique, XML-legal ID-values
Each ID should be an NCName: all letters, digits, and ideographs and must begine with a letter/ideograph (hyphen and underscore also allowed)
No XPath addressing

Test that the character set is UTF-8

i.e. test that this element is present:

<meta http-equiv="Content-Type" content="text/html; charset=utf-8" id="charset">

The Table

!|com.jbergin.HtmlFixture|
|http://www.cafeconleche.org/demo/petition.phtml|
|Element Focus|charset|meta|
|Attribute|content|text/html; charset=utf-8|
|Attribute|http-equiv|content-type|

Breaking this Down

Set the classpath
Load the HTMLFixture
Load the URL for the page we're testing.
Find the element whose ID is charset and assert that its name is meta.
Assert that this element has an attribute with the name content whose value is "text/html; charset=utf-8"
Assert that this element also has an attribute with the name http-equiv whose value is "content-type"

Can combine multiple tests in one table

!|com.jbergin.HtmlFixture|
|http://www.cafeconleche.org/demo/petition.phtml|
|Element Focus|title|title|
|Text|Petition|
|Element Focus|charset|meta|
|Attribute|content|text/html; charset=utf-8|
|Attribute|http-equiv|content-type|

Use whitespace to line columns up if you like

!|com.jbergin.HtmlFixture|
|http://www.cafeconleche.org/demo/petition.phtml|
|Element Focus|title     |title|
|Text         |Petition  |
|Element Focus|charset   |meta|
|Attribute    |content   |text/html; charset=utf-8|
|Attribute    |http-equiv|content-type|

Annoyingly you can't just put in a blank line or a blank row to separate tests

Blank line creates a new test.

Blank row causes:

com.jbergin.HtmlFixture

http://www.cafeconleche.org/demo/petition.phtml

Element Focus title title

Text Petition

Invalid token. Valid tokens are:
Attribute, Blank Token, Clear, Click, Element, Element Focus, Elements, Execute, Fail,
Focus, Focus Relative, Has Text, Javascript, Last ELement Token, List,
Matches Text, Node, Nodes, Preserve, Save, Set Value, Submit,
Symbol, Symbol Token, Text, Type, Type Focus, Types

Element Focus charset meta

Attribute content text/html; charset=utf-8

Attribute http-equiv content-type

Test that the petition data is included.

!|com.jbergin.HtmlFixture|
|http://www.cafeconleche.org/demo/petition.phtml|
|Element Focus|body     |body|
|Has Text     |Mickey Mouse|

Text Tests

Has Text for substring
Text for exact match
Matches Text for regular expressions

!|com.jbergin.HtmlFixture|
|http://www.cafeconleche.org/demo/petition.phtml|
|Element Focus|body     |body|
|Has Text     |Mickey Mouse|
|Text     |Mickey Mouse|
|Matches Text     |M.* Mouse|

Text Tests Do cross tag boundaries

|Has Text |Mickey Mouse|

will find

<p>Mickey <em>Mouse</em></p>

Regular expressions

Java

Negative tests

Assert that "Tom Delay" has not signed the petition.
Use Fail to reverse sense of test (essentially a boolean not)

!|com.jbergin.HtmlFixture|
|http://www.cafeconleche.org/demo/petition.phtml|
|Element Focus|body     |body|
|Fail|Has Text|Tom Delay|

Operators

<
<=
>
>=
==
!=

Test signing the petition

!|com.jbergin.HtmlFixture|
|http://www.cafeconleche.org/demo/petition.phtml|
|Element Focus|name     |input|
|Set Value|John Doe|
|Element Focus|email     |input|
|Set Value|jdoe@example.com|
|Element Focus|s1|
|Submit|
|Element Focus|body     |body|
|Has Text     |John Doe|

Verify you're at the Top-10 in Google

Can use a straight query URL; don't actually need to submit.
http://www.google.com/search?hl=en&q=political+petition
GET vs. POST

Watch out for cross-site scripting

JavaScript Tests

Focus on a script element
Then Execute onClick/onLoad/onChange/onClick/onFocus/onLoad/onMouseOver/onSelect/onSubmit/onUnload/etc.

!|com.jbergin.HtmlFixture|
|http://www.cafeconleche.org/demo/petition.phtml|
|Element Focus|loadhandler|script|
|Execute |onLoad|

Setting Page Properties

Add ?properties query string after page whose properties you want to set
Or press the Properties button

SubWikis

Organize tests hierarchically
Separate levels with periods, not slashes
Classpaths inherit through the hierarchy

Standard Headers and Footers

The HTML on SubWiki.PageHeader is added to the top of every page.
The HTML on SubWiki.PageFooter is added to the bottom of every page.

Suites

FitNesse treats pages whose name begins with "Suite" as collections of tests
Or set the suite property on a page to give it a Suite button that runs all tests in SubWiki

SetUp and TearDown

SubWiki.SetUp runs before each test in SubWiki
SubWiki.TearDown runs after each test in SubWiki
SubWiki.SuiteSetUp runs before all (not each) test in SubWiki
SubWiki.SuiteTearDown runs after all (not each) test in SubWiki
Useful for setting up a test database, server, etc. before running tests

More WIKI markup

Italic: via double single quotes ''foo''
Bold: three single quotes '''foo'''
Strike: double hyphen --foo--
Cross Reference: !see .PageName
Headers: !1 !2 !3
Centering: !c foo
Notes: !note
Horizontal Rules: four+ dashes ---- (more dashes makes the line thicker)
Images: !img http://www.example.com/foo.png
External links: just the URL http://www.example.com/foo/bar.html
Internal links: .#foo
Wiki links: "A WikiWord starts with a capital letter and has at least one more capital letter in it. Between the capitals there must be lower case letters or numbers. There cannot be two capitals in a row."
Alias links: click [[here][.WikiPage]]
Alias links: click [[here][http://www.example.com]]
Lists: Use indentation, asterisk and digits
Literal Text: !- foo -!
Preformatted text: triple curly braces {{{ foo }}}
Comments: # foo
Variables: !define foo { replacement text }
Variable dereference: ${foo}
Table of contents: !contents -R
Includes: !include PageName
Last Modification Time: !lastmodified
Fixture Specifiers:
Collapsable Sections:
```
!* title
foo foo foo
*!
```

WIKI Features

Versions
Properties
Refactor
Where Used
Recent Changes
Files
Search

Other Languages

FitNesse can run fixtures written in other languages: .NET, Ruby, etc.

.NET:

!define COMMAND_PATTERN {%m %p}
!define TEST_RUNNER {dotnet\FitServer.exe}
!define PATH_SEPARATOR {;}

Ruby:

!define COMMAND_PATTERN {ruby -I %p ruby/bin/FitServer.rb -v}

Need to install Ruby Fit server

RowFixture

ActionFixture

Used to perform actions and make test in a certain order
Four actions:
- start the fixture
- check the result of invoking a method
- enter a string into a method
- press a GUI item
Last three actions have a name that maps to a method in the fixture class

A Final Thought

Some tests are better than none
Don't let the perfect be the enemy of the good.

To Learn More

This presentation: http://www.cafeaulait.org/slides/sdbestpractices2006/fitnesse/
FitNesse: http://fitnesse.org/
Fitnesse Testing for Fast-Paced Agile Web Development: http://today.java.net/pub/a/today/2005/11/22/fitnesse-testing-for-agile-web-development.html
Test-first development with FitNesse: http://www.javaworld.com/javaworld/jw-02-2006/jw-0220-fitnesse.html
HTMLFixture docs: http://dps.csis.pace.edu:8077/FixtureDevelopment.HtmlFixture
Fit for Developing Software by Rick Mugridge and Ward Cunningham:
- Paperback: 384 pages
- Publisher: Prentice Hall PTR (June 29, 2005)
- ISBN: 0321269349
- Price: $44.99
The Fit Framework: http://fit.c2.com/

Index | Cafe con Leche

com.jbergin.HtmlFixture
URL
Instruction Name	Instruction arguments
Instruction Name	Instruction arguments
Instruction Name	Instruction arguments
...

com.jbergin.HtmlFixture
http://www.cafeconleche.org/demo/petition.phtml
Element Focus	title	title
Text	Petition
Invalid token. Valid tokens are: Attribute, Blank Token, Clear, Click, Element, Element Focus, Elements, Execute, Fail, Focus, Focus Relative, Has Text, Javascript, Last ELement Token, List, Matches Text, Node, Nodes, Preserve, Save, Set Value, Submit, Symbol, Symbol Token, Text, Type, Type Focus, Types
Element Focus	charset	meta
Attribute	content	text/html; charset=utf-8
Attribute	http-equiv	content-type