ReAsserting ReAssert

Tests break when the system under test evolves in ways that invalidate the assumptions encoded in the tests. ReAssert addresses this problem by making it easier to update tests to reflect the changed behavior. Like any other complex piece of software, ReAssert itself has evolved, making it susceptible to the same problem that it attempts to solve. There have been several times in which a change to ReAssert broke its unit tests. It is natural to ask whether ReAssert could repair them.

Recall from the first post in this series that ReAssert's unit tests have two parts: a failing test method marked with the @Test annotation and its expected repair marked with the @Fix annotation. When such a test breaks, it means that the @Fix method must change to reflect ReAssert's actual output.

Here is a real example of one time that ReAssert's evolution caused tests to break. An early version of ReAssert lacked the ability to trace an expected value back to its declaration. Instead, ReAssert would simply overwrite the expected side of a failing assertion. The following code (similar to the example used in the first post) shows the @Test and @Fix methods that verified this early behavior.

@Test
public void testString() {
  String expected = "expected";
  String actual = "actual";
  assertEquals(expected, actual);
}
@Fix("testString")
public void fixString() {
  String expected = "expected";
  String actual = "actual";
  assertEquals("actual", actual);
}

This is probably not what the developer would expect. Overwriting the expected side removes the use of the expected variable. This makes the test harder to understand and might cause a compiler warning since the variable is not used anywhere else. Such a repair could also cause other tests to fail if the assertion was located in a utility method called from multiple places. Indeed, this behavior confused several participants in the user study that Vilas and I performed to evaluate ReAssert.

I changed ReAssert such that it would instead replace the initial value of a variable used on the expected side of a failing assertion. Even though I wrote tests to verify this behavior prior to making the change, it still caused many tests to break. The example above broke, and it was necessary to update the @Fix method in the following way:

@Fix("testString")
public void fixString() {
  String expected = "actual";
  String actual = "actual";
  assertEquals(expected, actual);
}

Since I had the ReAssert plugin installed as per the second post in this series, I wanted it to automate repairs like the one above. Doing so proved challenging because the process was so self-referential: ReAssert used ReAssert's result to repair a test that (as per part one) triggered ReAssert. Don't worry if that sounds confusing because it is. The following diagram illustrates the process more clearly:

To avoid confusion, I will refer to the "upper" and "lower" instances of ReAssert. The upper instance is triggered when I tell the plugin to repair a failing @Test-and-@Fix method pair. The upper instance executes the test under JUnit, which—via FixChecker, my custom test runner—invokes the lower instance of ReAssert. The lower instance "repairs" the body of the @Test method and saves the result in memory. Finally the upper instance copies this result into the body of the @Fix method and outputs the repaired source code.

But what prevents the lower instance from introducing an infinite recursive loop? After all, the lower instance of ReAssert invokes JUnit, which runs the test with FixChecker, which repairs the test with ReAssert, which invokes JUnit, and so on. FixChecker breaks this loop by ensuring that only one instance of itself is active. This allows the lowermost instance of JUnit to execute @Test normally.

This experience with ReAssert reinforced my belief that meta-execution is an ideal way to test and improve software development tools. Not only does the developer discover bugs that would otherwise impact users, but executing a program on itself can indicate how easily one can extend the tool's behavior. In ReAssert's case, meta-execution not only uncovered many bugs but also led to several improvements in the internal design of the tool.

I think ReAssert's meta-repair capability is one of the most interesting aspects of the tool. Unfortunately, I didn't have room to describe it in the paper, which is why I wanted to write this series of weblog posts.

Build and Deploy Local Eclipse Plugin

When developing software tools, it is good practice to eat one's own dog food by using the tool oneself. It is one of the easiest ways to uncover bugs and improve usability. I ate ReAssert's dog food by using it to repair tests in other research projects and (as I'll describe in the next post) ReAssert itself.

ReAssert is implemented as an Eclipse plugin. The easiest way to deploy such a plugin is to include it in Eclipse's plugins directory¹. When Eclipse starts, it automatically loads all plugins in the directory. If there is more than one version of a particular plugin, Eclipse loads the most recent.

To keep from having to install ReAssert's plugin manually, I made it such that the normal build process automatically updates version numbers and copies the plugin to the appropriate place. A single Ant build script handles the entire process.

The main challenge lies in accessing Eclipse's build metadata in the script. Each Eclipse plugin holds the metadata in a file called MANIFEST.MF. It contains things like the plugin's name, it's version number, and the other plugins it depends on. For example, here is part of ReAssert's:

Manifest-Version: 1.0
Bundle-ManifestVersion: 2
Bundle-Name: edu.illinois.reassert.plugin
Bundle-SymbolicName: edu.illinois.reassert.plugin;singleton:=true
Bundle-Version: 0.3.0.201002231921
Bundle-Activator: edu.illinois.reassert.plugin.ReAssertPlugin
Require-Bundle: org.eclipse.ui,
 org.eclipse.core.runtime,
 ...
Bundle-Vendor: University of Illinois at Urbana-Champaign

ReAssert's build script first updates this file with the build number (based on the current date and time). Then, it reads the version and build number by converting MANIFEST.MF into a Ant property file. Finally, it bundles the plugin using the values.

Here are the relevant pieces of the script that implements this process:

1. Set the date and time with tstamp, then replace the build number in the manifest with the date and time.

2. Convert MANIFEST.MF into manifest.properties that Ant can read.²

3. Set plugin name using the properties and bundle the plugin's JAR file.

4. Copy the JAR to Eclipse's plugins directory. The eclipse.home variable is set when the script is run within Eclipse.

   
   Restart Eclipse to enable plugin
   

It is a pretty straightforward process but useful to keep my installed version of ReAssert up to date.

I have also seen a similar process scaled up to an entire enterprise. The company had an automated nightly build process that would post the plugin to an Eclipse Update Site on the local intranet. Every morning developers would update to the latest plugin. Any bugs they found while using the tool went straight into the bug tracking system.

Notes

1. One can also install plugins through Eclipse's Update Manager with a plugin update site, but I felt this was overkill for ReAssert since it was such a simple plugin.
2. Adapted from this article describing how to build a simple plugin.

This is the first of what I expect to be three posts in which I discuss these challenges.

Tests as Test Inputs

ReAssert transforms source code. Given the source code of a failing unit test, it outputs a transformed test that passes. Testing program transformation tools is difficult because writing input programs, passing them to the tool, and checking the output requires a lot of effort. Developers often automate the process by saving inputs and expected outputs to the filesystem or including them as string literals in their unit tests. Tests pass the input file contents or string literal to the tool and then verify that the tool's output exactly matches the expected output.

Both approaches are exceptionally common but have several disadvantages. First, files make it difficult to debug failures, since it can be difficult to figure out which file(s) corresponds to which test(s). Second, strings can make test code very verbose, and one has to worry about linebreaks, escape characters, and character encoding. Strings are also opaque to the IDE, so they lack helpful features like syntax highlighting and automatic formatting. Finally, both approaches require that the tool's output exactly matches the expected output byte-for-byte, whitespace and all. Such strict matching is rarely necessary when checking source code and can make tests very fragile. As soon as one changes the tool's pretty-printer, it can break every test even if program contents remain the same (which is actually one of the problems that ReAssert aims to solve).

I wanted ReAssert's unit tests to make testing as simple as possible while avoiding the problems caused by input files or string literals. The solution I built relies on the fact that ReAssert acts on unit tests. The test itself serves as the input to ReAssert, and another method in the same test class represents the expected output. To implement this idea, I extended JUnit's default behavior with a custom test runner called FixChecker and a new @Fix method annotation.

Here is an example: say I want to test that ReAssert replaces the initial value of a string used in a failing assertion. The failing test would look something like the following:

@Test
public void testString() {
  String expected = "expected";
  String actual = "actual";
  assertEquals(expected, actual);
}

ReAssert should repair the test by replacing the "expected" string with "actual". To verify this behavior, I create a second method annotated with @Fix whose body contains the expected repair.

@Fix("testString")
public void fixString() {
  String expected = "actual";
  String actual = "actual";
  assertEquals(expected, actual);
}

Then, I tell JUnit to use FixChecker by annotating the test class with JUnit's @RunWith annotation. FixChecker intercepts JUnit's normal result when a test fails. It then "repairs" the test and checks that the repair matches the body of the @Fix method. If not, or if no @Fix method exists, then the runner reports that the test fails. Otherwise, it reports that the test passes. In a sense, the @Test-and-@Fix method pair act like assertEquals with the repaired test on the actual side and the @Fix method on the expected side.

FixChecker's repairs do not change the source code directly. Instead, it holds the modified source code in memory and compares it against the parsed source code of the @Fix method. In this way, the comparison ignores differences in source code formatting, and one can use either qualified or unqualified class names. Also, since both the test and the @Fix are normal methods, they can reuse aspects of the surrounding test class, and both receive the full support of the IDE.

FixChecker also provides two other useful features. First, it is smart enough to ignore tests that pass and lack an @Fix method. Instead, it forwards them along to JUnit unchanged. This allows me to mix ReAssert tests with standard unit tests. Second, I can test when ReAssert is expected to fail by marking unrepairable tests with @Unfixable, another new annotation that FixChecker knows to look for.

@Test
@Unfixable
public void testIgnoreAssertFail() {
  fail();
}

Several aspects of this framework are applicable to other program transformation tools. It is often useful to use real application code when testing even when a tool does not act solely on unit tests. Also, custom test runners can be exceptionally powerful and allow one to tailor tests to many contexts.

In later posts, I'll describe how I automatically deployed ReAssert for my own use, and how I used the tool to repair its own unit tests.

This post describes what Theories are and what they do. In future posts, I hope to write about why I find them interesting and how they enable more complex testing tasks.

"Theories in practice: Easy-to-write specifications that catch bugs" defines Theories in the following way¹:

[Theories] are partial specifications of program behavior. Theories are written much like like test methods, but are universally quantified: all of the theory’s assertions must hold for all arguments that satisfy the assumptions...A theory can be viewed in several ways. It is a universally quantified ("for-all") assertion, as contrasted to an assertion about a specific datum. It is a generalization of a set of example-based tests. It is a (possibly partial) specification of the method under test.

To understand what this definition means, it is easiest to explain a simple example. Say we have a simple Counter class that allows one to increment an integer value every time the increment method is called.

public class Counter {
    private int value;

    public Counter(int init) {
        this.value = init;
    }

    public void increment() {
        value = value + 1;
    }
		
    public int getValue() {
        return value;
    }
}

We wish to test that incrementing always increases a counter's value by one. The standard way to test this functionality is to write an example-based unit test that creates a Counter, increments it a few times, and asserts that the incremented values are correct.

@Test 
public void testIncrement()	{
    Counter c = new Counter(3);
    c.increment();
    assertEquals(4, c.getValue());
    c.increment();
    assertEquals(5, c.getValue());
}

This is a useful test, but it only verifies that a single counter initialized to three is incremented correctly. It would be good to test additional counters initialized to many different values. Doing so using example-based testing requires multiple test methods or testing objects in a loop.

Theories provide an elegant alternative that complements example-based tests. From the test writer's point of view, Theories are just like normal unit tests but with one or more parameters.² To test incremement, one can write a Theory that accepts a Counter object, increments it, then asserts that the value has increased by one. This is more general than an example-based test because it verifies a property common to all counters, regardless of how they were initialized. In the following code, the incrementTheory method implements such a Theory.

@RunWith(Theories.class)
public class CounterTheories {
   
    @DataPoints 
    public static Counter[] data() {
        return new Counter[] {
            new Counter(0),
            new Counter(1),
            new Counter(-1),
            new Counter(Integer.MIN_VALUE),
            new Counter(Integer.MAX_VALUE), // overflows when incremented
        };
    }

    @Theory
    public void incrementTheory(Counter toIncrement) {
        int oldValue = toIncrement.getValue();
        assumeTrue(Integer.MAX_VALUE != oldValue);
        toIncrement.increment();
        int newValue = toIncrement.getValue();
        assertEquals(oldValue + 1, newValue);
    }

    //... more theories 
}

The @RunWith(Theories.class) annotation tells JUnit that it should run all methods in the class that are annotated with @Theory. The @DataPoints annotation marks methods that return values that JUnit should supply to applicable Theories. At runtime, JUnit matches the values returned from data point methods or fields to appropriate Theory parameters.³ In the example above, it sees that Counter objects are produced by data and consumed by incrementTheory, so it executes incrementTheory once for each element in the array returned from data.

Theories decouple test inputs from test implementation. Data points are automatically reused across multiple theories (even in subclasses), making it easier to write new tests. Adding a data point often provides a value that a test writer may not have originally considered, thus improving all Theories that use the data point.

But certain data points may not be applicable to a particular Theory. The test writer describes what data points apply by using methods provided by the org.junit.Assume class. Assumptions are similar to normal assertions except they cause a Theory to skip certain data points rather than fail. In our example, incrementTheory should not increment a counter whose value is equal to Integer.MAX_VALUE since the value would overflow. Therefore, incrementTheory uses assumeTrue to check for this special case.

This summary and example briefly describes the basics of Theories but glosses over how Theories work internally and lacks practical advice like how to write "good" theories. I hope that the undergrads and future weblog posts can explore these topics and deeper research issues further.

Notes

1. See also an early paper on Theories and the initial announcement of their inclusion as part of an experimental project called Popper.
2. .NET offers a similar feature but calls it—perhaps more descriptively—parameterized unit tests. JUnit uses this term to mean test classes that are instantiated with input data.
3. Internally, JUnit finds data points whose declared types derive from the declared types of Theory parameters. It does not currently box and unbox primitive data points. I submitted a simple patch that fixes the problem, but it has not yet been accepted.

This is the same paper mentioned in my crunch time analysis and typography request.

Here is the (working) abstract:

Developers often change software in ways that cause tests to fail. When this occurs, developers must determine whether failures are caused by errors in the code under test or in the test code itself. In the latter case, developers must repair failing tests or remove them from the test suite. Fixing tests is time consuming but beneficial, since removing tests reduces a test suite's ability to detect regressions. Fortunately, simple program transformations can repair many failing tests automatically.

We present ReAssert, a novel technique and tool that suggests repairs to failing tests' code which cause the tests to pass. Examples include replacing literal values in tests, changing assertion methods, or replacing one assertion with several. If the developer chooses to apply the repairs, ReAssert modifies the code automatically. Our experiments show that ReAssert can repair many common test failures and that its suggested repairs correspond to developers' expectations.

The conference will be held in Auckland, New Zealand. I am excited to travel overseas to present my work. Vilas, my coauthor, and Yun-Young, my officemate, are both from New Zealand and are eager to visit home.

Update September 8, 2009

I have posted the final version of the paper and updated the ReAssert homepage.

Update November 30, 2009

The conference presentation went very well, and I got a great deal of insightful questions and feedback from other attendees. Here are the presentation slides. I told the story of Alice the software developer like in the previous ReAssert post. The presentation starts with a picture of Alice adapted (as per Creative Commons) from xkcd #662.

I had originally planned to draw a picture in my normal cartoon style, but decided instead to use something simpler. The xkcd picture turned out to be a good choice; it made the audience laugh, and one attendee mentioned Scott McCloud's assertion from Understanding Comics that a simple face causes the audience to identify themselves in a character.

Alice is a developer a large software company. She works on the company's flagship product and spends over half of her time writing unit tests to verify her code and document her assumptions. She is not alone in this respect; the company requires that functional changes and bugfixes should have corresponding unit tests to prevent regressions. As a result, the product's unit test suite achieves exceptionally high coverage.

One day, the project manager informs Alice that a key requirement has changed. The changed requirement violates many assumptions encoded the test suite, so several dozen tests fail after Alice modifies the software. Now Alice has a choice: should she remove the failing tests since they no longer reflect the correct behavior of the software, or should she attempt to repair the tests, which would require tedious and time-consuming manual editing?

Developers often have to make a similar choice. When tests fail due to problems with test code rather than the system under test, it is undoubtedly beneficial to fix the broken tests, since removing tests reduces a test suite's ability to detect regressions. However, developers may not take the time to fix the broken tests. For example, while working on the refactoring paper, my colleagues and I found many of Eclipse's refactoring tests were either commented out, marked as ignored, or most bizarrely, bypassed using "if (true) return;".

To solve this problem, I have been exploring ways of reducing the effort required to fix broken unit tests. Doing so would make developers less fearful of "deep" changes, allow them to write more detailed tests, and most importantly, provide time for more important work.

As a first step toward this goal, I developed a tool called ReAssert that automatically suggests changes to test code that are sufficient to make tests pass. Earlier this week I released a public beta. I welcome anyone reading this to download it from the ReAssert project homepage and try it out. Please contact me if you have any comments, questions, ideas for improvement, or bug reports.

Even so, I have found myself buried in crunch time more often than I like to admit. Over the last month, for example, I spent many 10-plus-hour days writing my latest research paper. I was curious how this stretch of time compared to previous research papers'. Now that the deadline has passed, I can satisfy my curiosity by examining the papers' revision control system. It contains every edit made to each of the last five papers I was involved in.

The following chart shows how the papers grew as their deadlines approached.

Vertical jumps represent lines added or removed from a paper. Horizontal plateaus represent the length of time between changes. Crunch time is obvious in this diagram: around ten days before a deadline, a paper's growth rate increases, and the length of time between edits decreases. For example, the last ten days of the most recent paper, shown in red, contained about 75% of the paper's total edits.

This number, the fraction of edits in the final ten days, creates an interesting "crunchiness" metric. The following chart shows the metric for each of the papers.

This chart echoes how I would qualitatively rank the papers: "Automated Testing of Refactoring Engines" was easily the most crunchy and "Predicting Effectiveness of Automatic Testing Tools" was the least.

More research is needed to determine if crunchiness correlates with paper acceptance.

Automatic white-box test generation is a challenging problem. Many existing tools rely on complex code analyses and heuristics. As a result, structural features of an input program may impact tool effectiveness in ways that tool users and designers may not expect or understand.

We develop a technique that uses structural program metrics to predict the test coverage achieved by three automatic test generation tools. We use coverage and structural metrics extracted from 11 software projects to train several decision tree classifiers. Our experiments show that these classifiers can predict high or low coverage with success rates of 82% to 94%.

I posted the original tech report and other supporting information at the paper's companion site.

Unfortunately, Agitar no longer exists despite having a mature product, many talented engineers, and a strong involvement in research. As I understand it, the company had difficulty getting developers to adopt their software, and the recent economic troubles were too much for it. Without Agitar to support a trip, it is unlikely that I will be able to go to Italy to present the paper in person. I am nevertheless very pleased with the work and am glad that it will get wider exposure.

I do not think I could have had a better first conference. ICSE is one of the top three or four software engineering conferences in the world, and attracts some excellent submissions. I arrived Tuesday night and spent the next four days listening to many interesting talks (and, of course, a few boring ones) and networking with other researchers. I saw too many great ideas and met too many smart people to describe in this short weblog post; check out the list of papers and presenters if you are interested.

We spent most of our time in the hotel conference center, but I did get a chance to see some of the rest of Minneapolis. On Thursday, the conference dinner was held at Nicollet Island, a historic park in the middle of the Mississippi River. There, one could try hatchet throwing, archery, or a Segway. I hit the hatchet target after only three tries, but did not get a chance to shoot an arrow or ride a Segway because the Minnesota cold drove me back indoors. Has my short time in California made me so thin-skinned?

On Saturday night, after a long day at the Automated Software Test Workshop, I went with a small group to the Mall of America. We ate dinner at a completely ridiculous rainforest-themed restaurant whose walls were covered in jungle foliage and animatronic animals with servers dressed as safari guides. After that, we walked around, basking in the glow of American consumerism.

I am very excited about this paper. Not only is it the culmination of the work I have been doing since last August, but I was fortunate enough to be first author. I feel we have some very strong results and a great point from which to continue future work. Now we need to forget about this submission and concentrate on the next conference.

Update

The paper was accepted!