Edit & Continue in VS 2005

Dennis Forbes talks about the dangers of Edit & Continue (via Len Holgate), and its re-introduction in VS 2005.

Having been a long-time user of E&C in the VB.Classic IDE, I have mixed feelings. There are both benefits and dangers to this feature, and on balance, I feel that the benefits outweigh the drawbacks. The arguments against E&C tend to emphasise the dangers of this feature in the hands of inexperienced developers. Well, almost everything is dangerous in the hands of an inexperienced developer, so that's just not a convincing argument for me. And if you do understand what you're doing, E&C can be a great benefit.

Enough preaching. Like any subject, the devil is in the details. So let's look closer at some typical uses of E&C.

Bad Debugging with E&C

A developer makes several attempts during a single run to fix a problem. He repeats a cycle of altering code and then resetting the current execution point to run through the altered code. For example, a developer believes that a bug is caused by an off-by-one error in a loop. So he firsts puts a +1 in the loop and then checks the result. If that doesn’t work, he puts a –1 in the loop instead, still hoping to get the right result.

Proponents of this debugging approach say that it is more efficient because there is no need to restart the program after every code change. The problem, though, is the indisputable fact that in this case the developer doesn’t actually understand the problem. If he did understand the problem, there would be no need to make multiple attempts at a fix — he could go straight to the problem area and fix it in one attempt.

While E&C makes it very tempting to try many potential solutions until something seems to work, this is akin to poking a jellyfish with a stick to see if it moves; it doesn't actually teach you very much. A quality solution is much more likely to come from some deep thinking about a problem rather than ad-hoc attempts at a fix.

More Bad Debugging with E&C

A developer starts by finding and fixing a single bug, but then attempts to fix more defects that appear. These defects are often related to or hiding behind the first bug. The temptation when using E&C is to find and fix as many bugs as possible during a single run—this can be very satisfying and at first sight seems to be an efficient method of working.

Unfortunately, this approach also has problems. First, the developer once again probably doesn't understand the original defect at a deep level. If she understood the original bug properly, it is unlikely that her original fix would have exposed the multiple new issues. Even if she did understand the bug, it is debatable how well the new issues and subsequent fixes were understood. As before, E&C makes it far too tempting to fix bugs as they arise, rather than taking time to think about the issues away from the heat of the battle.

Another problem with this approach is that if there are multiple issues being exposed within a block of code, it is likely that the code itself is fundamentally flawed. Such code always benefits from taking a step back from the situation rather than fixing the bugs in an ad-hoc manner. Using E&C to push multiple fixes is unlikely to improve the code’s quality.

The next problem is that several programming experiments have been done that collectively show that the average developer introduces one new bug for every two bugs that he or she fixes. Even the best developers produce one new bug for every four that they fix. Making multiple concurrent fixes, some of which might potentially interfere with each other, is not conducive to producing high-quality solutions.

Perhaps the final nail in the coffin of “shotgun debugging” is that many of the E&C fixes made are small ones, often affecting only a few lines of code. These are made because they are the type of changes that look manageable without having to perform desk-checking or proper review.

Unfortunately, the literature shows that the chances of creating a bug significantly increase when making such small changes. Specifically, as the number of lines changed increases from one to five lines, the chance of making a bad change is high and increases. With more than five altered lines, the chance of making a bad change decreases, probably because the developer becomes more careful as the fix becomes larger.

Bad Unit Testing with E&C

A developer edits code several times within a single run in order to run multiple unit tests on a procedure or component. For example, a program that executes an embedded SQL script sees the developer correcting, tuning, and re-executing the SQL script many times until he is happy that everything is working properly. The typical argument for this approach is that it is much faster to run these multiple edit-and-test cycles in one hit rather than having to restart the program every time a change is made.

There are several problems with this type of ad-hoc testing. The first is that any tests done in this manner are not going to be as comprehensive as a manual or automated unit test harness that has been implemented beforehand. It’s hard to invent several good unit tests on the fly, especially when the psychological mindset is aimed at making the code work rather than proving that the code doesn't work.

The temptation is always to fix the code in-flight when each test fails rather than running a test suite, collating the results, and then thinking about the fixes as a single, well-controlled patch to the program. Simultaneous mixing of the unit testing mindset with the quite different mindset of fixing code that fails the unit tests is very dangerous.

A second problem with this approach is that it makes it very difficult to perform unit regression testing on future versions of the code. Even if the developer could manage to remember the entire unit test suite that he had performed, repeating a set of interactive regression tests every time the code changes is very boring and therefore prone to mistakes.

Another trap that many developers fall into with E&C in this context is changing code on both sides of the interface being tested. It is just too easy to make a change to the calling code that makes the interface being tested appear to work. What might actually be happening is that two errors, one on either side of the interface, could compensate for each other and result in the interface appearing to work.

Once again, the psychological need to get the code working can interfere with the unit tests being performed. In effect, this makes it easier for the developer to fool himself into believing that everything is working correctly through an artificial test case created by manipulating two sets of code.

Good Debugging with E&C

One appropriate way of debugging with E&C is when you find an obvious coding error, often while looking at something completely different, and change the code for an obvious fix. While it is a fine line between an “obviously correct” fix and a “subtly wrong” fix, most developers know when they've found a glaringly obvious problem. In this case, it can be good practice to fix the problem, test the fix immediately, and then proceed on your way with your original mission.

Another reasonable way of using E&C is when you've found an error and managed to reproduce it successfully. The next step is often to reproduce the error in several different ways in order to learn more about it. E&C can be very useful in allowing multiple code changes and tests during a single run in order to triangulate the bug and learn more about it. The proviso, of course, is that you must put the code back into its original state once you have completed your investigation.

Good Unit Testing with E&C

E&C can prove very useful when you're unit-testing a GUI. Although it’s still better to create a proper test harness if possible, this can be difficult to automate because you have to simulate a very unpredictable component (the end user), and also it’s difficult to validate the unit test results. For this reason, many developers prefer to test interactively, so I'll grant that this can be a reasonable use of E&C.

Good Prototyping with E&C

In this scenario, a developer makes multiple code changes during a single run in order to prototype, typically when investigating the behaviour of a class, a control, or some other component. With E&C, it is much faster to perform this type of testing, and because it is only prototyping, there is much less danger of bugs running rampant. This scenario is one of the genuinely useful ways of using E&C.