Debuggers and packet filters, oh my!

A recent revival of an old thread brought up the discussion about REALbasic's debugger, and how poorly Zone Alarm (amongst other personal packet filters) handles debugging sessions. I figured that a better understanding of debuggers might make it more apparent as to why this happens.

Most developer tools are only designed to run on a single platform, which means that the tool can make use of all sorts of great OS facilities for features like debugging. Even when you see a cross platform dev tool (like Code Warrior), it's still a rewrite from one platform to the next. REALbasic, on the other hand, is written to be as cross-platform as possible. So if we add support for another OS (perhaps OS2/Warp will be next!), we can implement the framework, and most of the IDE comes "for free" along with it. Because of this, we try to keep our technologies as agnostic as possible. That's especially true of debugging.

Most debuggers are written to use the OS, and ultimately, the kernel and processor debugging features. So, on Windows, most debuggers are written to use DebugBreak, ReadProcessMemory, ContinueDebugEvent, et al. However, that would require us to rewrite our debugger for all three platforms we currently support, which is a huge amount of work for what amounts to no gain (actually, it amounts to a loss, as you'll see in a moment). Instead, the REALbasic debugger is written in a quite creative way. Since we control the framework, and every REALbasic application must come with the REALbasic framework, we're able to put debugging facilities directly into the application itself in an agnostic way. Ultimately, we rely on network protocol for all debugging functionality.

When a REALbasic application is being debugged, the first thing it does (before ever running user code) is to listen on the local loopback interface for an incoming connection -- then it waits. The IDE will connect (also via the loopback interface, of course) to the debuggee, at which point execution continues as normal. There is a complex protocol used by the two parties to allow for all of the debugging functionality you see. So, for instance, if the debugger needs to display all of the local variable values, it simply sends a message to the debuggee asking it for the various values. The debuggee can then send them back to the debugger via a response message and the debugger can display them.

By doing our debugging facilities this way, we get all sorts of awesome benefits. For starters, it's a fully cross-platform design, so adding new platforms means no extra work for debugging to continue to function. But, what's more, we can provide a more powerful debugging experience to REALbasic users than more traditional debuggers. Even better though: remote debugging is essentially free! To make the remote debugging work, we simply need to listen on a non-loopback interface. The remote debugger stub is simply a file transfer mechanism -- once the file has been transferred from the IDE to the remote machine, the rest is the identical debugging facilities you're already used to.

However, by doing our debugging via the loopback, there are some downsides as well. (Technical decisions like this are always a series of trade offs, after all). Debugging is going to be slower than the traditional method because it won't be making use of shared memory, or machine-level breakpoints. Instead, there has to be data marshaling via the network stack. Another problem is that manipulations of the program counter become much more tricky -- you have to send a message to the debuggee that says "when you return from this call, return to this other location over here", which is certainly more complex than just modifying the execution counter directly! (As an aside: this glosses over the fact that moving the execution pointer is a bad idea in a reference counted language anyways since that can lead to really bad problems with objects being destroyed or leaked incredibly easily.)

Most of those problems are very much behind-the-scenes and not terribly noticeable to the user. One problem which is noticeable to the user is: poorly written packet filters will whine about the debuggee doing a listen, even though there's no security problem. ZoneAlarm falls into this category -- I see a fair amount of bug reports from users who are wondering what is going on and why ZoneAlarm is alerting them. The reason this happens is because the debuggee application is doing a listen so that the IDE can connect to it to initiate the debug session. If you get one of these alerts from your packet filters when you go to debug, you can safely ignore it. In fact, you can usually set up a rule in your packet filter that basically says "ignore loopback activity."

Btw, I know I'm using pretty harsh language to describe this problem -- but it is quite silly behavior, IMSEO (in my semi-educated opinion). When something listens on the loopback, or attempts to connect while bound to the loopback, there is *no* way for there to be a remote exploit (barring a horribly implemented networking stck). And my personal opinion is: once you have local access to the machine, security becomes mostly a joke anyways. So all that dialog is really doing is needlessly alerting the user to a situation that's not even close to being a problem -- basically a scare tactic.

In any case, hopefully you have a better idea of how the REALbasic debugger works! Whether that's actually interesting to you or not... well, that remains to be seen. ;-)