Remote Debugger Stub Post Mortem (part three)

We left off last time talking about everything that went right with the process. But as you're aware, nothing's ever that simple.

For as well as things went, there were also things that went horribly wrong. For example, I ran into a number of bugs with the way AutoDiscovery, and EasyTCPSocket were implemented. For example, it desperately needed a polling mechanism that was tied to the event loop, otherwise it would use :: gasps :: App.DoEvents. But the good news is -- every time I'd run into one of those bugs, it'd get fixed. Some of the fixes were easy (encoding issues for example), and others were a royal pain (adding a polling mechanism and hooking it up in a way that provided the same behavior only without bugs). It was also stunningly difficult to debug the application using REALbasic's debugger. Not only because of things like timeout issues built into the application itself, but because of things like the event loop not pausing while I was in the debugger. I'd hit a breakpoint in a DataAvailable event, and more data would be arriving under the hood, so ReadAll would return incorrect information. These were the type of bugs that really slowed things down, but they too were fixed in time. It was mostly just trying to figure out why something would behave the way it did that took the time.

After about a month of working on the remote debugging process, doing it mostly in the afternoons or in-between other projects, I finally had written a file transfer application. It would auto discover the stub on command, and when you'd drag files or folders to it, they would be transferred to the remote machine. I also hooked up the ability for the "IDE" to tell the stub when to launch a file that it transferred. Basically, the rudimentary implementation of the remote debugger stub was complete.

Then I took it upon myself to start redesigning the stub's interface. The part of the interface that the user sees didn't change much -- it still needs to show the same information to the user. However, the preferences portion was completely revamped so that it would be less confusing and have more options. That was the point at which I was able to add password protection to the stub (but the "IDE" still had no way to specify the password, so it was a simple 'works' or 'doesn't work' test). Things were finally starting to take shape. I modeled the preferences design off of the standard Windows properties pane since I thought it was a slick design. I had an idea that there would be two different implementations of the preferences dialog -- one for Windows and Linux, the other for the Mac. But after seeing how the dialog appears on the Mac, I figured it looked nice over there as well (let's face it, it's a remote debugging stub, just how much time needs to be spent on the preferences pane for it?).

I was also toying with the idea of making it a service (for all platforms) so that you could set it up to run automatically in the background. Unfortunately, there are a lot of complications to doing something like that. For example, on Windows, not only can't a service have a GUI itself, but it can't launch applications that have a GUI when running as a service unless a specific option is turned on (for the service itself). Microsoft specifically talks about this option as being something that should never be turned on because there are security flaws with it that are unavoidable (it was basically an option put in there to appease a large ISV). So I was loathe to add known security holes to any product I develop. There were similar problems for Mac and Linux, so eventually I put the idea on the back-burner and focused on making it a quality desktop application. Users who want the stub to launch when the OS is booted can put it in the startup items for the OS if they'd like.

At this point, I felt the debugger stub's user interface was sufficient, so I put the entire project down. The IDE wasn't even able to locally debug yet, so there was no point to trying to hook the feature up -- it's not like it would get more testing!

Fast forward to January of 2005. Geoff and Matt wander into my office and ask "hey Aaron, remote debugging is done, right?" That's never a good sign... They were preparing for the DEMO conference and were heavily relying on the remote debugger working for their demo. I basically had about a month to finish hooking up the remote debugger and make sure it works well enough to be demo'ed to strangers.

Hooking the protocol's helper classes into the IDE project was a breeze -- took maybe 10 minutes to stick them in there and make sure I could still compile. So that first hurdle of getting the classes into the project was overcome. But now came the hard part, making them actually do something.

I started out by working on the preferences pane for locating remote debugger stubs and adding them. Since I was the one who designed the preferences dialog, it was quite easy to hook that up. The AutoDiscovery class made everything go very smoothly -- it only took about two days to get the entire hook-up done. So now I was able to locate publicly available stubs, as well as add stubs manually. They would save to the preferences file and load back up when the IDE would launch.

Then came the true test -- hooking it up to the debugger interface. That went surprisingly well -- it only took about three days to get fully functional. I had a special keyboard shortcut (just holding down Shift while selecting Run) that would start the process. Before rendering the project out, the IDE would attempt to establish a TCP connection with the stub, and communicate with the stub to find out what options to build (like platform, etc). Once it had the full information, the project would be rendered much like the previous IDE would do, and transfer the compiled application to the stub. Once the file was transferred, the IDE would tell the stub to launch the file and voila! Everything worked.

Of course, nothing goes that easy. There was a bug, but I didn't catch it until after DEMO (because it didn't affect the demo at all) -- remote debugging a Mac application from Windows wouldn't work because the compiler would change the name of the build application from MyApplication to MyApplication.bin (for example). Oops! But I digress.

Once I had it actually remotely debugging the applications, I had to hook up the Run Remotely menu. This proved to be the most difficult part of the entire process. Getting those menus items to be in the right order, do the right thing, have the right shortcuts, etc took me a lot longer than I was expecting. But I'd say the entire process of hooking the remote debugger up to the new IDE took about 1.5 weeks. It went a lot smoother than expected -- and it apparently works well enough for us to win a DEMOgod award from DEMO. :-)

At this point, I thought I was free and clear. It worked for DEMO, so it must be working, right? Wrong. I realized that I wanted the public to have a chance to comment about the protocol. I figured that it would either get no response because no one really cares so long as it works, or I would get a lot of good feedback and everyone would love everything. I got a response somewhere in the middle. ;-) About a dozen people responded to my posting on the betas list about opening the protocol up for comments. Out of those dozen people, about 4 or 5 actually responded with any comments. Again, I was very glad I took this step.

The first thing that someone pointed out to me was that the security scheme which was so elegant and simple, was also completely broken if the IDE was on the other side of a NAT device from the stub. It never even occurred to me that this may be the case. People also pointed out limitations like the fact that the challenge/response system was broken unless we added ever-changing data to it. I also got a lot of nice feature requests that were easy to add on, like the ability to send command line parameters for the launch packet, send permission information along with the files, etc.

Most everything that was commented on was easy to fix. Which just tells me that the initial design, although slightly short-sighted, was not flawed and would work well. I was very excited to know that this was a solid, forward-thinking protocol that I could stand behind and say "this is good" because I just never got that feeling from the first remote debugging implementation. I knew the first one was OK, but I never felt that it was great. I implemented a lot of the fixes people alerted me to, updated the documentation, and checked everything in.

Which brings us up to the present day. Since remote debugging is a pro-only feature, it's not been something any of our beta users have been able to test out since we're also testing our registration code at the same time. But I expect that within the next while, the remote debugging functionality will be available to use (not just look at) for beta users. I don't expect to run into any bugs with it because it's been pretty heavily tested internally, but who knows. My hope is that it's solid and I don't have to monkey with a lot of bug fixes with it. And I'm happy to say that a lot of REALbasic bugs were fixed because of my work on this project. But only time will tell.

If you are interested in seeing the remote debugging protocol, feel free to contact me. I'd be happy to furnish the protocol specs on request. And if you enjoyed this post-mortem, let me know -- maybe I'll write other ones.