Bug Reports: compiler bugs

What the heck, why not stretch this topic out even more, right? ;-) One topic that's near and dear to my heart are compiler bugs. It's near because I'm the one who sees and fixes all of them, and it's dear because the compiler needs to be one of the most rock-solid, stable, reliable components to all of REALbasic. If you can't trust the compiler, then there's not much else you can trust! This is even more important since 2005, when we started using REALbasic to build the IDE. This post is going to talk about how compiler bugs are generally handled as a more specific case of the bug life cycle.

Compiler bugs are generally very few and far between -- it's easily the least-frequent category of bug areas. But when a compiler bug comes in, it goes through the usual verification process that all bugs go through. Usually it comes directly to me for verification though, since I'm in more of a position to say "expected" vs "unexpected" for the bug's behavior. Once I've verified that it truly is a bug, I scrap the user's example project if I can. You see, we have an automated testing suite for language features -- that's where all compiler bug reports end up. This testing suite makes use of RBScript to exercise all sorts of things. I use it to ensure errors are reported properly, verify expected behavior of existing language features, write test suites for new compiler features, etc. So I typically use the example project from the user as a starting point, but ultimately end up adding a regression test to the existing suite. This provides three-fold benefits. 1) I know I have a very simple example to reproduce the bug, because I wrote it. 2) I have a constant testing environment that I'm very comfortable working in. 3) I can be assured this bug will never crop up in public again, because I run the test suite against every internal build to ensure it's not broken.

Unfortunately, not every compiler bug can go into this test suite. There are a few compiler features that cannot be exercised by RBScript because of its sandbox mode. Things like exceptions and declares need to be tested in a GUI application. It's not the end of the world, but it's not ideal either. I'd say 90% of the compiler functionality is testable via the test suite (though we're not to the point where we are testing all 90% of it -- but we're getting there!).

So now that there's a reproducible example in the test suite, it's time to actually fix the bug. Unless it's a PPC issue, I use Visual Studio to track down the compiler bugs. I'm more comfortable with the toolset, and honestly, it's debugger works significantly more reliable than anything based on gdb. However, I will dive right in to XCode whenever I have to and am relatively comfortable with it. (As a side note, I really like the fact that I can interface with gdb directly from XCode's debugger command window, and wish I could make use of WinDbg commands from VS' on occasion.)

Magic hand-waving happens, and I fix the bug.

The way I can tell the bug is fixed is when all of the tests in the compiler test suite pass. At this point, the new test is checked in to version control, and I might even spend a little time writing corollary tests to commit as well. For instance, if there's a bug with class inheritance, and I notice I don't have any tests cases to verify the error cases, I'll whip up some examples to make sure errors are reported appropriately too. Finally, once I'm satisfied that everything passes the tests, I commit the actual changes to the branch, and mark the bug as resolved.

You might think that's the end of the story, but it's not. After the bug has been marked as resolved, it gets assigned to our testing staff. They make certain that the bug is resolved by testing the fix out in an actual build (instead of just an internal one). Every once in a while, they'll find the behavior is different than what's expected (usually because my internal build has other changes in it that the main branch doesn't have), and the report gets kicked back to me for further work. But eventually, the bug fix is verified by testing. Then it rolls out into a public release (like an alpha or a beta), where it gets further testing by more real world situations. (I should note that the first real world situation that gets tested before I ever reaches the public is the IDE itself -- that's a pretty large real world situation!)

The hope is that by this point in time, the change has been so thoroughly tested that it's not going to have a negative impact on the general public. But we like to be sure, so we subject our alpha/beta testers to it as well. ;-) Almost invariably, one of two things happens at this point: nothing, or a very, very large something. 99% of the time, nothing happens because everyone is happy and stuff keeps working like they expect. This happens for two reasons: the fix works, and it's usually a fix to a code path rarely exercised. But 1% of the time, it causes a ruckus because either someone found a real world test cases that demonstrates a new problem, or because someone was relying on the buggy behavior when they shouldn't have been. In the case of a real world test case, we start the entire process over again. In the case of relying on buggy behavior, there's usually a discussion as to what the correct behavior is that drags on for a long time. ;-)

Finally, when all is said and done, what comes out the other end is a better product that gets released in a final build. If some part of the process I've described breaks down, the changes don't make it in to the public. Usually, the closer we get to "late beta", the more likely we are to roll back any changes which are 100% certain to be improvements. This is especially true with something like the compiler, since it can have far-reaching implications and stability is extremely important.