Testing GUIs with Abbot and Costello Elliotte Rusty Harold Thursday, March 16, 2006 elharo@metalab.unc.edu http://www.cafeaulait.org/

Benefits of Test Driven Development are Well Known

• Faster development; faster time to market

• More robust, error free code

• YAGNI: Writing tests first avoids writing unneeded code.

• Easier to find and fix bugs

• Easier to add features or change behavior; less worry about unintentionally introducing bugs

• Makes refactoring/optimization possible: any change that doesn't break a test suite is de facto acceptable.

Why test?

Often a developer will want to make some optimizations to a piece of code on which a lot of user-interface behavior depends. Sure, the new table sorter will be ten times faster and more memory efficient, but are you sure the changes won't affect the report generator? Run the tests and find out. A developer is much more likely to run a test that has encapsulated how to set up and test a report than he is to launch the application by hand and try to track down who knows how to do that particular test.

For Unit Testing to Work

• Run automatically without any human participation beyond launching the test suite

• And maybe not even that: Cruise Control, Continuum, etc.

• Results should be pass/fail

• Test failures need to be blindingly obvious

• Test must be reproducible and consistent

But Testing GUIs is Hard

• GUIs are designed to be driven by user; not a program.

• The public interface of a GUI app is the GUI.

• Test code often outruns the GUI it's supposed to be testing.

Different Styles of Testing

• Glass box: testing the nonpublic API

• White box: testing the public API

• Grey box: testing internal components (Abbot; java.awt.Robot)

• Black box: testing the GUI itself (Costello)

Some pieces can be tested using public APIs

public class PlayerFrameTest extends TestCase {
    
    private JFrame frame;
    private JMenuBar menubar;

    protected void setUp() throws QTException {
        frame = new PlayerFrame();
        menubar = frame.getJMenuBar();
    }

    public void testTitleOfUntitledFrameIsAmateurPlayer()
      throws QTException {
        assertEquals("Amateur Player", frame.getTitle());
    }
    
    public void testInitialFrameIsNotTooSmall() throws QTException { 
        Dimension d = frame.getSize();
        assertTrue(d.width > 300);
        assertTrue(d.height > 300);
    }
    
}

But this gets old fast

    public void testMinimizeCommandKeyEquivalent() {
       
        Component[] fileitems = menubar.getMenu(3).getMenuComponents();
        for (int i = 0; i < fileitems.length; i++) {
            if (fileitems[i] instanceof JMenuItem) {
                JMenuItem item = (JMenuItem) fileitems[i];
                if (item.getText().equals("Minimize")) {
                    assertEquals('M', item.getAccelerator().getKeyCode());
                }
            }
        }
        
    }

We need some generic methods

• List windows on screen

• List dialogs onscreen

• Find frontmost window

• Find component by name

• Find component by name and type (menu item, button, etc.)

• Find component by label (including internationalized label).

• Find menu item by label

• Find menu item by name

• Delay until window/menu/dialog is shown

Is there the seed of a library here?

Enumerating all windows

• Frame[] allFrames = Frame.getFrames();

• For open windows test each one with isVisible()

• For front window test with isFocused()

Enumerating all dialogs

1. Get all frames

2. Get each frame's owned windows with getOwnedWindows()

    public void testPresentMovieDialog() {
        
        JMenuItem presentMovie = findJMenuItem(menubar, "Present Movie");
        presentMovie.doClick();
        
        Frame[] allFrames = Frame.getFrames();
        for (int i = 0; i < allFrames.length; i++) {
            Window[] owned = allFrames[i].getOwnedWindows();
            for (int j = 0; j < allFrames.length; j++) {
                Dialog dialog = (Dialog) owned[j];
                if ("Present Movie".equals(dialog.getTitle())) {
                    return;
                }
            }
        }
        fail("No present dialog");
        
    }

Enumerating all components in a container

You can use (potentially recursively) the getComponents() method to find the component you want to test:

    private static void listComponents(Container container) {
        Component[] components = container.getComponents();
        for (int i = 0; i < components.length; i++) {
            System.out.println(components[i]);
            if (components[i] instanceof Container) {
                listComponents((Container) components[i]);
            }
        }
    }

Finding a menu item by label

• Use getMenu(), getMenuComponents(), and getMenuCount() to walk the menu bar:

    private JMenuItem findJMenuItem(String name) {
      
      for (int menu = 0; menu < menubar.getMenuCount(); menu++) {
        Component[] menuitems = menubar.getMenu(menu).getMenuComponents();
        for (int i = 0; i < menuitems.length; i++) {
          if (menuitems[i] instanceof JMenuItem) {
            JMenuItem item = (JMenuItem) menuitems[i];
            if (item.getText().equals(name)) {
              return item;
            }
          }
        }
      }
      return null;
    }

• Menu items can be activated with the doClick() method:

    public void testNewMenuItem() {
      int oldNumberOfWindows = Frame.getFrames().length;
      JMenuItem newMenuItem = findJMenuItem("New Player");
      newMenuItem.doClick();
      int newNumberOfWindows = Frame.getFrames().length;
      assertEquals(oldNumberOfWindows+1, newNumberOfWindows);
    }
    

Driving the application through the event queue

• Other pieces can be tested by posting events into the event queue using EventQueue.postEvent():

  public void postEvent(AWTEvent theEvent)

• Or by dispatching straight to the component using Component.dispatchEvent()

  public final void dispatchEvent(AWTEvent e)

• Still can be hard to read the results

java.awt.Robot

• Control mouse and keyboard programmatically

• Generates native system input events

• Will actually move the mouse cursor instead of just generating mouse move events.

• May need special privileges or extensions. If disallowed, the Robot constructor will throw an AWTException.

Robot API

package java.awt;

public class Robot {

  public Robot() throws AWTException
  public Robot(GraphicsDevice screen) throws AWTException

  public void mouseMove(int x, int y);
  public void mousePress(int buttons);
  public void mouseRelease(int buttons);
  public void mouseWheel(int scrollAmount)

  public void keyPress(int keycode);
  public void keyRelease(int keycode);

  public Color getPixelColor(int x, int y);
  public BufferedImage createScreenCapture(Rectangle screenRect);

  public boolean isAutoWaitForIdle();
  public void    setAutoWaitForIdle(boolean isOn);
  public int     getAutoDelay();
  public void    setAutoDelay(int ms);
  public void    delay(int ms);
  public void    waitForIdle();

  public  String toString();
}

Robot Example

• Test that clicking the minimize button iconifies the window.

• Automatic in Swing but not necessarily in the AWT.

    public void testMinimizeButton() throws AWTException {
       
        frame.setVisible(true);
        
        Point p = frame.getLocationOnScreen();
        Robot r = new Robot();
        r.setAutoWaitForIdle(true);
        r.mouseMove(p.x+35, p.y+10);
        r.mousePress(InputEvent.BUTTON1_MASK);
        r.mouseRelease(InputEvent.BUTTON1_MASK);
        r.delay(1000); // needs a couple of seconds for the state to update
        int state = frame.getExtendedState();
        assertEquals(Frame.ICONIFIED, state);
        
    }

Testing Pixel Colors

• Test individual pixels

• This opens a file handcrafted in Photoshop to contain nothing but pure blue, and tests that this is indeed what is seen:

      public void testOpenPNGFile() throws QTException, AWTException {
          
         QTFile file = new QTFile("data/pureblue.png");
         FileOpener.openFile(file);
         Robot r = new Robot();
         r.delay(5000);
         Color expected = new Color(0, 0, 255);
         Color actual = r.getPixelColor(50, 100);
         assertEquals(expected, actual);
         
      }

• Improve robustness by getting coordinates to grab from the component you're testing with getLocationOnScreen():

  public Point getLocationOnScreen()

Testing With Screen Captures

• Sometimes you can't really test pixel by pixel; or carefully contrive the pixels.

• Less robust. Requires pixel perfection.

• Platform, locale, and look and feel differences mean this can only really test pictures; not GUI components

This is still pretty low-level stuff

• Very dependent on exact position of components

• Changes to GUI can break the test code

• Different platforms can move things around enough to break the tests

• Different localizations can move things around enough to break the tests

• Mix and match: find the component; get its position, then move the mouse there

Abbot

The events the Robot provides are like assembly language for GUI testing. To facilitate and encourage testing, you need a higher-level representation.

--Timothy Wall, Abbot Developer

Key components

• A better Robot class to drive GUIs

• A ComponentTester to drive components (at a higher, more semantic level than the Robot) and make assertions about them

• Testers for individual components: JTable, Window, JComboBox, JMenuItem, JSpinner, etc. optimized to drive each such component

• A finder package to locate components including a matcher interface (like FileFilter)

Example: IntegerTextField

• A component that allows the user to type in a non-negative integer (and only a non-negative integer)

• A subclass of JTextField

• Extra methods: getNumericValue() and setNumericValue()

Positive tests:

• 0

• 1

• 2

• 3

• 4

• 5

• 6

• 7

• 8

• 9

• 10

• 100

• 1234

• 123456789

• 2^32-1

• 2^64-1

• 2^64+1

Exception Tests

• getNumberValue() before anything is typed

• -1

• -2

• -10

• -123456789

• -2^32-1

• -2^64-1

• -2^64+1

Test we can enter "1" into the textfield and get the value 1 out

1. Extend ComponentFixture

2. Get a tester for the component we're testing.

3. Show the component in a frame.

4. "Type" text in the component with actionKeyString

package com.elharo.swing.test;

import com.elharo.swing.WholeNumberField;

import abbot.tester.ComponentTester;
import junit.extensions.abbot.ComponentTestFixture;

public class WholeNumberFieldTestCase extends ComponentTestFixture {

    private ComponentTester tester;
    private WholeNumberField field;
    
    protected void setUp() {
        tester = ComponentTester.getTester(WholeNumberField.class);
        field = new WholeNumberField();
        showFrame(field);
    }
    
    public void testOne() {
        tester.actionKeyString(field, "1");
        int result = field.getNumberValue();
        assertEquals(1, result);
    }
    
}

Now test that non-numbers are ignored

    public void testText() {
        tester.actionKeyString(field, "12ab34");
        int result = field.getNumberValue();
        assertEquals(1234, result);
    }

Other things ComponentTester can make a component do

public void actionClick(Component c, ComponentLocation loc, int buttons, int count)
public void actionClick(Component c, ComponentLocation loc, int buttons)
public void actionClick(Component c, ComponentLocation loc)
public void actionClick(Component comp, int x, int y, int buttons, int count)
public void actionClick(Component comp, int x, int y, int buttons)
public void actionClick(Component comp, int x, int y)
public void actionClick(Component comp)

public void actionDrag(Component dragSource, ComponentLocation loc, int modifiers)
public void actionDrag(Component dragSource, ComponentLocation loc, String modifiers)
public void actionDrag(Component dragSource, ComponentLocation loc)
public void actionDrag(Component dragSource, int sx, int sy, String modifiers)
public void actionDrag(Component dragSource, int sx, int sy)
public void actionDrag(Component dragSource)

public void actionDragOver(Component target, ComponentLocation where)

public void actionDrop(Component dropTarget, ComponentLocation loc)
public void actionDrop(Component dropTarget, int x, int y)
public void actionDrop(Component dropTarget)

public void actionFocus(Component comp)

public void actionKeyPress(Component comp, int keyCode)
public void actionKeyPress(int keyCode)

public void actionKeyRelease(Component comp, int keyCode)
public void actionKeyRelease(int keyCode)

public void actionKeyString(Component c, String string)
public void actionKeyString(String string)

public void actionKeyStroke(Component c, int keyCode, int modifiers)
public void actionKeyStroke(Component c, int keyCode)
public void actionKeyStroke(int keyCode, int modifiers)
public void actionKeyStroke(int keyCode)

public void actionSelectAWTMenuItem(Frame menuFrame, String path)
public void actionSelectAWTMenuItemByLabel(Frame menuFrame, String path)
public void actionSelectAWTPopupMenuItem(Component invoker, String path)
public void actionSelectAWTPopupMenuItemByLabel(Component invoker, String path)

public void actionSelectMenuItem(Component item)

public void actionSelectPopupMenuItem(Component invoker, ComponentLocation loc, String path)
public void actionSelectPopupMenuItem(Component invoker, int x, int y, String path)
public void actionSelectPopupMenuItem(Component invoker, String path)

public void actionShowPopupMenu(Component invoker, ComponentLocation loc)
public void actionShowPopupMenu(Component invoker, int x, int y)
public void actionShowPopupMenu(Component invoker)

ComponentLocation

• A location relative to a component, to the center of the component unless specified otherwise

• Subclasses provide locations for specific component types; for instance a JListLocation can convert a row index into a location

• public Point getPoint(Component c) throws LocationUnavailableException

• public Rectangle getBounds(Component c) throws LocationUnavailableException

Many Components don't need anything more than ComponentTester and their public APIs to test them

• JLabel

• JButton

• JMenuItem

• JPopupMenu

There are subclasses for each component (JLabelTester, JButtonTester, etc.) but you can ignore these.

Testing Windows

package abbot.tester;

public class WindowTester extends ContainerTester {

    public String deriveTag(Component comp);

    public void actionClose(Component c);
    public void actionMove(Component w, int screenx, int screeny);
    public void actionMoveBy(Component w, int dx, int dy);
    public void actionResize(Component w, int width, int height);
    public void actionResizeBy(Component w, int dx, int dy);
    public void actionActivate(Window w);
    
}

Testing Window Resize

import java.awt.Dimension;
import quicktime.QTException;
import abbot.tester.WindowTester;
import com.elharo.quicktime.*;

public class RatioTest extends junit.extensions.abbot.ComponentTestFixture {
    
    private PlayerFrame frame;
    private WindowTester tester = new WindowTester();
    
    static {
        try {
            QuicktimeInit.setup();
        }
        catch (QTException ex) {
            throw new RuntimeException(ex);
        }   
    }
    
    protected void setUp() throws QTException {
        frame = new PlayerFrame();
        frame.show();
    }

    protected void tearDown() {
        frame.hide();
        frame.dispose();
    }

    
    public void testRatioIsConstant() { 
        Dimension d = frame.getSize();
        double ratio = d.getHeight() / (double) d.getWidth();
        tester.actionResize(frame, 2*d.width, d.height);
        d = frame.getSize();
        assertEquals(ratio, d.getHeight() / (double) d.getWidth(), 0.01);   
    }
    
}

Testing Buttons

package abbot.tester;

public class ButtonTester extends ComponentTester {

  public void click(Component c, int x, int y,
                    int mask, int count)

}

Testing Checkboxes

package abbot.tester;

public class CheckboxTester extends ComponentTester {

  public void click(Component c, int x, int y,
                    int mask, int count)

}

Testing Choices

package abbot.tester;

public class ChoiceTester extends ComponentTester {

  public void actionSelectIndex(Component c, int index)
  public void actionSelectItem(Component c, String item)

}

Testing JLists

package abbot.tester

public class JListTester extends JComponentTester {

  public void   actionSelectIndex(Component c, int index)
  public void   actionSelectItem(Component c, String item)
  public void   actionSelectValue(Component c, String item)
  public void   actionSelectRow(Component c, JListLocation location)
  public ComponentLocation getLocation(Component c, Point index)
  public int    getSize(JList list)
  public String getContents(JList list)
  public Object getElementAt(JList list, int index)

}

Testing JFileChooser

package abbot.tester;

public class JFileChooserTester extends JComponentTester {

  public void actionSetDirectory(Component c, String path)
  public void actionCancel(Component c)
  public void actionSetFilename(Component c, String filename)
  public void actionApprove(Component c)

}

Testing ComboBoxes

package abbot.tester;

public class JComboBoxTester extends JComponentTester {

  public void   actionSelectIndex(Component c, int index)
  public void   actionSelectItem(Component c, String item)
  public JList  findComboList(JComboBox X)
  public String getValueAsString(JComboBox c, JList list, Object item, int index)
  public String getContents(JComboBox c)

}

Testing Sliders

package abbot.tester;

public class JSliderTester extends JComponentTester {

  public void actionIncrement(Component c)
  public void actionDecrement(Component c)
  public void actionSlide(Component c, int value)
  public void actionSlideMaximum(Component c)
  public void actionSlideMinimum(Component c)

}

Testing Spinners

package abbot.tester;

public class JSpinnerTester extends JComponentTester {

  public void actionIncrement(Component c)
  public void actionDecrement(Component c)
  public void actionSetValue(Component c, String value)

}

Testing Split Panes

package abbot.tester;

public class JSplitPaneTester extends JComponentTester {

  public void actionMoveDivider(Component c, double X)
  public void actionMoveDividerAbsolute(Component c, int index)

}

Testing Tables

package abbot.tester;

public class JTableTester extends JComponentTester {

  public void actionSelectCell(Component c, JTableLocation X)
  public void actionSelectCell(Component c, int index, int index)
  public ComponentLocation getLocation(Component c, Point index)

}

Testing TextComponents

package abbot.tester;

public class JTextComponentTester extends JComponentTester {

  public void actionClick(Component c, int index)
  public void actionEnterText(Component c, String text)
  public void actionSetCaretPosition(Component c, int index)
  public void actionStartSelection(Component c, int index)
  public void actionEndSelection(Component c, int index)
  public void actionSelect(Component c, int start, int end)
  public void actionSelectText(Component c, int start, int end)

}

Testing Trees

package abbot.tester;

public class JTreeTester extends JComponentTester {

  public void actionClick(Component c, ComponentLocation location)
  public ComponentLocation parseLocation(String location)
  public void actionSelectRow(Component c, ComponentLocation location)
  public void actionSelectRow(Component c, int index)
  public TreePath pathToStringPath(JTree tree, TreePath path)
  public void actionClickRow(Component c, int index, String modifiers)
  public void actionClickRow(Component c, int index, String modifiers, int index)
  public void actionClickRow(Component c, int index)
  public void actionMakeVisible(Component c, TreePath path)
  public void actionSelectPath(Component c, TreePath path)
  public void actionToggleRow(Component c, ComponentLocation location)
  public void actionToggleRow(Component c, int index)
  public boolean assertPathExists(Component c, TreePath path)
  public ComponentLocation getLocation(Component c, Point index)
  public boolean isLocationInExpandControl(JTree tree, int index, int index)

}

Configuring Abbot's Robot:

abbot.robot.auto_delay

number of milliseconds between generated events. Default zero. About 100-200 for a typical user

abbot.robot.mode

Set this to either "robot" or "awt" to designate the desired mode of event generation. "robot" uses java.awt.Robot to generate events, while "awt" stuffs events directly into the AWT event queue.

abbot.robot.event_post_delay

maximum number of milliseconds for the system to post the AWT event corresponding to a Robot-generated event

abbot.robot.default_delay

default delay setting

abbot.robot.popup_delay

the maximum number of milliseconds to wait for a menu to appear

abbot.robot.component_delay

the maximum time in milliseconds to wait for a component to become available

Finding Components To Test

package abbot.finder;

public class BasicFinder implements ComponentFinder {

  public BasicFinder();
  public BasicFinder(Hierarchy h);

  public static ComponentFinder getDefault();
  public Component find(Container root, Matcher m);
   throws ComponentNotFoundException, MultipleComponentsFoundException
  public Component find(Matcher m)
   throws ComponentNotFoundException, MultipleComponentsFoundException;
                             
}

Matcher

package abbot.finder;

public interface Matcher {

  public boolean matches(Component c);

}

Match the New Player menu item

public NewPlayerMatcher implements Matcher {

  public boolean matches(Component c) {
  	return c instanceof JMenuItem
      && "New Player".equals(((JMenuItem) c).getText());
  }

}

Find the New Player menu item

Matcher matcher = new NewPlayerMatcher();
BasicFinder finder = new BasicFinder();
JMenuItem item = (JMenuItem) finder.find(container, matcher);

ComponentTestFixture

• TestCase subclass your own test cases should extend

• Sets up and cleans up the GUI.

• Methods to place GUI component within a frame

• Methods to show/hide windows and dialogs

• Converts exceptions thrown from the event dispatch thread into test failures.

Costello

• Script Recorder

• Example: java -cp lib/abbot.jar junit.extensions.abbot.ScriptFixture doc/editor-tutorial-1.xml

Costello Recording

1. Launch the script editor with java -jar abbot.jar

2. File/New Script

3. Test/Launch the application you're testing

4. Select Capture/All Actions (or shift+F2)

5. Use the GUI

6. Press Shift-F2 when the GUI reaches a state you want to test.

Running a Saved Script

Integrating Costello Script into JUnit Test

• Subclass ScriptFixture

• Write a suite() method that returns a ScriptTestSuite object specifying the saved scrripts to run

• Scripts are loaded from the file system and filtered from directories.

To Learn More

• Abbot Home Page: http://abbot.sourceforge.net/

• This presentation: http://www.cafeaulait.org/slides/sd2006west/guitesting/

• Robot class: http://java.sun.com/j2se/1.4.2/docs/api/java/awt/Robot.html

• JFCUnit: http://jfcunit.sourceforge.net/

• Jemmy: http://jemmy.netbeans.org

• SwingUnit: https://swingunit.dev.java.net/

• Abbott and Costello radio shows: http://www.radiolovers.com/pages/abbottcostello.html