www.BrettDaniel.com

In my previous post, I discussed the first of several challenges I encountered when testing ReAssert. In this, the second of three articles in the series, I will describe how I automatically deployed the tool for my own use.

Build and Deploy Local Eclipse Plugin

When developing software tools, it is good practice to eat one's own dog food by using the tool oneself. It is one of the easiest ways to uncover bugs and improve usability. I ate ReAssert's dog food by using it to repair tests in other research projects and (as I'll describe in the next post) ReAssert itself.

ReAssert is implemented as an Eclipse plugin. The easiest way to deploy such a plugin is to include it in Eclipse's plugins directory¹. When Eclipse starts, it automatically loads all plugins in the directory. If there is more than one version of a particular plugin, Eclipse loads the most recent.

To keep from having to install ReAssert's plugin manually, I made it such that the normal build process automatically updates version numbers and copies the plugin to the appropriate place. A single Ant build script handles the entire process.

The main challenge lies in accessing Eclipse's build metadata in the script. Each Eclipse plugin holds the metadata in a file called MANIFEST.MF. It contains things like the plugin's name, it's version number, and the other plugins it depends on. For example, here is part of ReAssert's:

Manifest-Version: 1.0
Bundle-ManifestVersion: 2
Bundle-Name: edu.illinois.reassert.plugin
Bundle-SymbolicName: edu.illinois.reassert.plugin;singleton:=true
Bundle-Version: 0.3.0.201002231921
Bundle-Activator: edu.illinois.reassert.plugin.ReAssertPlugin
Require-Bundle: org.eclipse.ui,
 org.eclipse.core.runtime,
 ...
Bundle-Vendor: University of Illinois at Urbana-Champaign

ReAssert's build script first updates this file with the build number (based on the current date and time). Then, it reads the version and build number by converting MANIFEST.MF into a Ant property file. Finally, it bundles the plugin using the values.

Here are the relevant pieces of the script that implements this process:

1. Set the date and time with tstamp, then replace the build number in the manifest with the date and time.

2. Convert MANIFEST.MF into manifest.properties that Ant can read.²

3. Set plugin name using the properties and bundle the plugin's JAR file.

4. Copy the JAR to Eclipse's plugins directory. The eclipse.home variable is set when the script is run within Eclipse.

   
   Restart Eclipse to enable plugin
   

It is a pretty straightforward process but useful to keep my installed version of ReAssert up to date.

I have also seen a similar process scaled up to an entire enterprise. The company had an automated nightly build process that would post the plugin to an Eclipse Update Site on the local intranet. Every morning developers would update to the latest plugin. Any bugs they found while using the tool went straight into the bug tracking system.

Notes

1. One can also install plugins through Eclipse's Update Manager with a plugin update site, but I felt this was overkill for ReAssert since it was such a simple plugin. ⬏
2. Adapted from this article describing how to build a simple plugin. ⬏

I wrote ReAssert to make it easier to maintain unit tests. Ironically, I encountered several challenges when testing ReAssert itself. First, ReAssert acts on source code, so I created a test framework that made it easy to build input programs and check ReAssert's output. Second, I ate my own dogfood by deploying the tool on my local machine. Finally, I combined both aspects by ReAsserting ReAssert itself.

This is the first of what I expect to be three posts in which I discuss these challenges.

Tests as Test Inputs

ReAssert transforms source code. Given the source code of a failing unit test, it outputs a transformed test that passes. Testing program transformation tools is difficult because writing input programs, passing them to the tool, and checking the output requires a lot of effort. Developers often automate the process by saving inputs and expected outputs to the filesystem or including them as string literals in their unit tests. Tests pass the input file contents or string literal to the tool and then verify that the tool's output exactly matches the expected output.

Both approaches are exceptionally common but have several disadvantages. First, files make it difficult to debug failures, since it can be difficult to figure out which file(s) corresponds to which test(s). Second, strings can make test code very verbose, and one has to worry about linebreaks, escape characters, and character encoding. Strings are also opaque to the IDE, so they lack helpful features like syntax highlighting and automatic formatting. Finally, both approaches require that the tool's output exactly matches the expected output byte-for-byte, whitespace and all. Such strict matching is rarely necessary when checking source code and can make tests very fragile. As soon as one changes the tool's pretty-printer, it can break every test even if program contents remain the same (which is actually one of the problems that ReAssert aims to solve).

I wanted ReAssert's unit tests to make testing as simple as possible while avoiding the problems caused by input files or string literals. The solution I built relies on the fact that ReAssert acts on unit tests. The test itself serves as the input to ReAssert, and another method in the same test class represents the expected output. To implement this idea, I extended JUnit's default behavior with a custom test runner called FixChecker and a new @Fix method annotation.

Here is an example: say I want to test that ReAssert replaces the initial value of a string used in a failing assertion. The failing test would look something like the following:

@Test
public void testString() {
  String expected = "expected";
  String actual = "actual";
  assertEquals(expected, actual);
}

ReAssert should repair the test by replacing the "expected" string with "actual". To verify this behavior, I create a second method annotated with @Fix whose body contains the expected repair.

@Fix("testString")
public void fixString() {
  String expected = "actual";
  String actual = "actual";
  assertEquals(expected, actual);
}

Then, I tell JUnit to use FixChecker by annotating the test class with JUnit's @RunWith annotation. FixChecker intercepts JUnit's normal result when a test fails. It then "repairs" the test and checks that the repair matches the body of the @Fix method. If not, or if no @Fix method exists, then the runner reports that the test fails. Otherwise, it reports that the test passes. In a sense, the @Test-and-@Fix method pair act like assertEquals with the repaired test on the actual side and the @Fix method on the expected side.

FixChecker's repairs do not change the source code directly. Instead, it holds the modified source code in memory and compares it against the parsed source code of the @Fix method. In this way, the comparison ignores differences in source code formatting, and one can use either qualified or unqualified class names. Also, since both the test and the @Fix are normal methods, they can reuse aspects of the surrounding test class, and both receive the full support of the IDE.

FixChecker also provides two other useful features. First, it is smart enough to ignore tests that pass and lack an @Fix method. Instead, it forwards them along to JUnit unchanged. This allows me to mix ReAssert tests with standard unit tests. Second, I can test when ReAssert is expected to fail by marking unrepairable tests with @Unfixable, another new annotation that FixChecker knows to look for.

@Test
@Unfixable
public void testIgnoreAssertFail() {
  fail();
}

Several aspects of this framework are applicable to other program transformation tools. It is often useful to use real application code when testing even when a tool does not act solely on unit tests. Also, custom test runners can be exceptionally powerful and allow one to tailor tests to many contexts.

In later posts, I'll describe how I automatically deployed ReAssert for my own use, and how I used the tool to repair its own unit tests.