www.BrettDaniel.com

This semester I am overseeing two undergraduate senior theses. The two students are working on a project involving JUnit Theories. Theories are a very useful feature of JUnit, but they have not been widely adopted since they are still experimental and not documented very extensively. The project's short-term goal is to address this problem by writing a suite of Theories for use as benchmarks in testing research. In the longer term, we hope to apply knowledge gained by writing Theories to other research projects and find areas in which Theories can be improved.

This post describes what Theories are and what they do. In future posts, I hope to write about why I find them interesting and how they enable more complex testing tasks.

"Theories in practice: Easy-to-write specifications that catch bugs" defines Theories in the following way¹:

[Theories] are partial specifications of program behavior. Theories are written much like like test methods, but are universally quantified: all of the theory’s assertions must hold for all arguments that satisfy the assumptions...A theory can be viewed in several ways. It is a universally quantified ("for-all") assertion, as contrasted to an assertion about a specific datum. It is a generalization of a set of example-based tests. It is a (possibly partial) specification of the method under test.

To understand what this definition means, it is easiest to explain a simple example. Say we have a simple Counter class that allows one to increment an integer value every time the increment method is called.

public class Counter {
    private int value;

    public Counter(int init) {
        this.value = init;
    }

    public void increment() {
        value = value + 1;
    }
		
    public int getValue() {
        return value;
    }
}

We wish to test that incrementing always increases a counter's value by one. The standard way to test this functionality is to write an example-based unit test that creates a Counter, increments it a few times, and asserts that the incremented values are correct.

@Test 
public void testIncrement()	{
    Counter c = new Counter(3);
    c.increment();
    assertEquals(4, c.getValue());
    c.increment();
    assertEquals(5, c.getValue());
}

This is a useful test, but it only verifies that a single counter initialized to three is incremented correctly. It would be good to test additional counters initialized to many different values. Doing so using example-based testing requires multiple test methods or testing objects in a loop.

Theories provide an elegant alternative that complements example-based tests. From the test writer's point of view, Theories are just like normal unit tests but with one or more parameters.² To test incremement, one can write a Theory that accepts a Counter object, increments it, then asserts that the value has increased by one. This is more general than an example-based test because it verifies a property common to all counters, regardless of how they were initialized. In the following code, the incrementTheory method implements such a Theory.

@RunWith(Theories.class)
public class CounterTheories {
   
    @DataPoints 
    public static Counter[] data() {
        return new Counter[] {
            new Counter(0),
            new Counter(1),
            new Counter(-1),
            new Counter(Integer.MIN_VALUE),
            new Counter(Integer.MAX_VALUE), // overflows when incremented
        };
    }

    @Theory
    public void incrementTheory(Counter toIncrement) {
        int oldValue = toIncrement.getValue();
        assumeTrue(Integer.MAX_VALUE != oldValue);
        toIncrement.increment();
        int newValue = toIncrement.getValue();
        assertEquals(oldValue + 1, newValue);
    }

    //... more theories 
}

The @RunWith(Theories.class) annotation tells JUnit that it should run all methods in the class that are annotated with @Theory. The @DataPoints annotation marks methods that return values that JUnit should supply to applicable Theories. At runtime, JUnit matches the values returned from data point methods or fields to appropriate Theory parameters.³ In the example above, it sees that Counter objects are produced by data and consumed by incrementTheory, so it executes incrementTheory once for each element in the array returned from data.

Theories decouple test inputs from test implementation. Data points are automatically reused across multiple theories (even in subclasses), making it easier to write new tests. Adding a data point often provides a value that a test writer may not have originally considered, thus improving all Theories that use the data point.

But certain data points may not be applicable to a particular Theory. The test writer describes what data points apply by using methods provided by the org.junit.Assume class. Assumptions are similar to normal assertions except they cause a Theory to skip certain data points rather than fail. In our example, incrementTheory should not increment a counter whose value is equal to Integer.MAX_VALUE since the value would overflow. Therefore, incrementTheory uses assumeTrue to check for this special case.

This summary and example briefly describes the basics of Theories but glosses over how Theories work internally and lacks practical advice like how to write "good" theories. I hope that the undergrads and future weblog posts can explore these topics and deeper research issues further.

Notes

1. See also an early paper on Theories and the initial announcement of their inclusion as part of an experimental project called Popper. ⬏
2. .NET offers a similar feature but calls it—perhaps more descriptively—parameterized unit tests. JUnit uses this term to mean test classes that are instantiated with input data. ⬏
3. Internally, JUnit finds data points whose declared types derive from the declared types of Theory parameters. It does not currently box and unbox primitive data points. I submitted a simple patch that fixes the problem, but it has not yet been accepted. ⬏