This is the third in a series of posts in which I discuss the challenges I encountered when testing ReAssert. I already showed how I used tests as their own input and automatically deployed ReAssert for my own use. Here, I combine both aspects by demonstrating how ReAssert can repair its own unit tests.
ReAsserting ReAssert
Tests break when the system under test evolves in ways that invalidate the assumptions encoded in the tests. ReAssert addresses this problem by making it easier to update tests to reflect the changed behavior. Like any other complex piece of software, ReAssert itself has evolved, making it susceptible to the same problem that it attempts to solve. There have been several times in which a change to ReAssert broke its unit tests. It is natural to ask whether ReAssert could repair them.
Recall from the first post in this series that ReAssert's unit tests have two parts: a failing test method marked with the @Test annotation and its expected repair marked with the @Fix annotation. When such a test breaks, it means that the @Fix method must change to reflect ReAssert's actual output.
Here is a real example of one time that ReAssert's evolution caused tests to break. An early version of ReAssert lacked the ability to trace an expected value back to its declaration. Instead, ReAssert would simply overwrite the expected side of a failing assertion. The following code (similar to the example used in the first post) shows the @Test and @Fix methods that verified this early behavior.
@Test public void testString() { String expected = "expected"; String actual = "actual"; assertEquals(expected, actual); } @Fix("testString") public void fixString() { String expected = "expected"; String actual = "actual"; assertEquals("actual", actual); }
This is probably not what the developer would expect. Overwriting the expected side removes the use of the expected variable. This makes the test harder to understand and might cause a compiler warning since the variable is not used anywhere else. Such a repair could also cause other tests to fail if the assertion was located in a utility method called from multiple places. Indeed, this behavior confused several participants in the user study that Vilas and I performed to evaluate ReAssert.
I changed ReAssert such that it would instead replace the initial value of a variable used on the expected side of a failing assertion. Even though I wrote tests to verify this behavior prior to making the change, it still caused many tests to break. The example above broke, and it was necessary to update the @Fix method in the following way:
@Fix("testString") public void fixString() { String expected = "actual"; String actual = "actual"; assertEquals(expected, actual); }
Since I had the ReAssert plugin installed as per the second post in this series, I wanted it to automate repairs like the one above. Doing so proved challenging because the process was so self-referential: ReAssert used ReAssert's result to repair a test that (as per part one) triggered ReAssert. Don't worry if that sounds confusing because it is. The following diagram illustrates the process more clearly:
To avoid confusion, I will refer to the "upper" and "lower" instances of ReAssert. The upper instance is triggered when I tell the plugin to repair a failing @Test-and-@Fix method pair. The upper instance executes the test under JUnit, which—via FixChecker, my custom test runner—invokes the lower instance of ReAssert. The lower instance "repairs" the body of the @Test method and saves the result in memory. Finally the upper instance copies this result into the body of the @Fix method and outputs the repaired source code.
But what prevents the lower instance from introducing an infinite recursive loop? After all, the lower instance of ReAssert invokes JUnit, which runs the test with FixChecker, which repairs the test with ReAssert, which invokes JUnit, and so on. FixChecker breaks this loop by ensuring that only one instance of itself is active. This allows the lowermost instance of JUnit to execute @Test normally.
This experience with ReAssert reinforced my belief that meta-execution is an ideal way to test and improve software development tools. Not only does the developer discover bugs that would otherwise impact users, but executing a program on itself can indicate how easily one can extend the tool's behavior. In ReAssert's case, meta-execution not only uncovered many bugs but also led to several improvements in the internal design of the tool.
I think ReAssert's meta-repair capability is one of the most interesting aspects of the tool. Unfortunately, I didn't have room to describe it in the paper, which is why I wanted to write this series of weblog posts.
