www.BrettDaniel.com

JUnit Theories

This semester I am overseeing two undergraduate senior theses. The two students are working on a project involving JUnit Theories. Theories are a very useful feature of JUnit, but they have not been widely adopted since they are still experimental and not documented very extensively. The project's short-term goal is to address this problem by writing a suite of Theories for use as benchmarks in testing research. In the longer term, we hope to apply knowledge gained by writing Theories to other research projects and find areas in which Theories can be improved.

This post describes what Theories are and what they do. In future posts, I hope to write about why I find them interesting and how they enable more complex testing tasks.

"Theories in practice: Easy-to-write specifications that catch bugs" defines Theories in the following way1:

[Theories] are partial specifications of program behavior. Theories are written much like like test methods, but are universally quantified: all of the theory’s assertions must hold for all arguments that satisfy the assumptions...A theory can be viewed in several ways. It is a universally quantified ("for-all") assertion, as contrasted to an assertion about a specific datum. It is a generalization of a set of example-based tests. It is a (possibly partial) specification of the method under test.

To understand what this definition means, it is easiest to explain a simple example. Say we have a simple Counter class that allows one to increment an integer value every time the increment method is called.

public class Counter {
    private int value;

    public Counter(int init) {
        this.value = init;
    }

    public void increment() {
        value = value + 1;
    }
		
    public int getValue() {
        return value;
    }
}

We wish to test that incrementing always increases a counter's value by one. The standard way to test this functionality is to write an example-based unit test that creates a Counter, increments it a few times, and asserts that the incremented values are correct.

@Test 
public void testIncrement()	{
    Counter c = new Counter(3);
    c.increment();
    assertEquals(4, c.getValue());
    c.increment();
    assertEquals(5, c.getValue());
}

This is a useful test, but it only verifies that a single counter initialized to three is incremented correctly. It would be good to test additional counters initialized to many different values. Doing so using example-based testing requires multiple test methods or testing objects in a loop.

Theories provide an elegant alternative that complements example-based tests. From the test writer's point of view, Theories are just like normal unit tests but with one or more parameters.2 To test incremement, one can write a Theory that accepts a Counter object, increments it, then asserts that the value has increased by one. This is more general than an example-based test because it verifies a property common to all counters, regardless of how they were initialized. In the following code, the incrementTheory method implements such a Theory.

@RunWith(Theories.class)
public class CounterTheories {
   
    @DataPoints 
    public static Counter[] data() {
        return new Counter[] {
            new Counter(0),
            new Counter(1),
            new Counter(-1),
            new Counter(Integer.MIN_VALUE),
            new Counter(Integer.MAX_VALUE), // overflows when incremented
        };
    }

    @Theory
    public void incrementTheory(Counter toIncrement) {
        int oldValue = toIncrement.getValue();
        assumeTrue(Integer.MAX_VALUE != oldValue);
        toIncrement.increment();
        int newValue = toIncrement.getValue();
        assertEquals(oldValue + 1, newValue);
    }

    //... more theories 
}

The @RunWith(Theories.class) annotation tells JUnit that it should run all methods in the class that are annotated with @Theory. The @DataPoints annotation marks methods that return values that JUnit should supply to applicable Theories. At runtime, JUnit matches the values returned from data point methods or fields to appropriate Theory parameters.3 In the example above, it sees that Counter objects are produced by data and consumed by incrementTheory, so it executes incrementTheory once for each element in the array returned from data.

Theories decouple test inputs from test implementation. Data points are automatically reused across multiple theories (even in subclasses), making it easier to write new tests. Adding a data point often provides a value that a test writer may not have originally considered, thus improving all Theories that use the data point.

But certain data points may not be applicable to a particular Theory. The test writer describes what data points apply by using methods provided by the org.junit.Assume class. Assumptions are similar to normal assertions except they cause a Theory to skip certain data points rather than fail. In our example, incrementTheory should not increment a counter whose value is equal to Integer.MAX_VALUE since the value would overflow. Therefore, incrementTheory uses assumeTrue to check for this special case.

This summary and example briefly describes the basics of Theories but glosses over how Theories work internally and lacks practical advice like how to write "good" theories. I hope that the undergrads and future weblog posts can explore these topics and deeper research issues further.

Notes

  1. See also an early paper on Theories and the initial announcement of their inclusion as part of an experimental project called Popper.
  2. .NET offers a similar feature but calls it—perhaps more descriptively—parameterized unit tests. JUnit uses this term to mean test classes that are instantiated with input data.
  3. Internally, JUnit finds data points whose declared types derive from the declared types of Theory parameters. It does not currently box and unbox primitive data points. I submitted a simple patch that fixes the problem, but it has not yet been accepted.

3 Comments

jayagopi jagadeesan Says:

Short and illustrative example of the use of Theories. Nice.
I just have one question: How is this different from a Parametrized Test in JUnit ?
I mean, I could do the same thing using the Parametrized.class runner and using
the @Parameters annotation for generating the array of Parameters and running
a normal JUnit test annotated with @Test using the generated parameters.
Any insight will be much appreciated.

cheers,
jay

Brett Says:

JUnit’s parameterized unit tests (PUTs) offer many of the same benefits as Theories in that both separate test data from test implementation. However, PUTs act at the class rather than method level, are much more verbose, and offer no equivalent to “datapoint discovery”. They also suffer from the problems described in my mutable datapoints post, but there is no “alternative three” workaround with PUTs.

To illustrate the difference between PUTs and Theories, the following code shows one way of writing the Counter example using PUTs:

@RunWith(Parameterized.class)
public class ParamCounterTest {

    @Parameters
    public static List getParams() {
        return Arrays.asList(new Object[][] {
            { new Counter(0) },
            { new Counter(1) },
            { new Counter(-1) },
            { new Counter(Integer.MAX_VALUE) },
            { new Counter(Integer.MIN_VALUE) },
        });
    }

    private Counter underTest;

    public ParamCounterTest(Counter c) {
        this.underTest = c;
    }

    @Test
    public void testIncrement() {
        int oldValue = underTest.getValue();
        assumeTrue(Integer.MAX_VALUE != oldValue);
        underTest.increment();
        int newValue = underTest.getValue();
        assertEquals(oldValue + 1, newValue);
    }

    //...more tests
}

Generally, the choice of whether to use PUTs or Theories is a matter of design and taste. I have used both and think Theories are clearer because it is obvious that a particular @Theory is acting on its arguments rather than a field as in the case of PUTs. Also, say I needed to add a second type of data point. In the example above, I would have to modify every subarray in getParams, add a new argument to the class’ constructor, and create a new field to hold the value. With Theories I could simply create some new @DataPoints and a new @Theory.

jayagopi jagadeesan Says:

Thanks Brett for your prompt reply. Much appreciated.
I can see what you mean.
cheers,
jay

Leave a Comment

Allowed Tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>