How to use CreateWindow to make a dialog

So one of the challenges that I've faced with working on the RB framework is how to mesh the Win32 APIs with the REALbasic way of doing things to accomplish a goal. One of those goals, recently, was how to make a modeless dialog frame type that fits with the REALbasic model.

In traditional Win32 programming, modeless dialogs are created by using a dialog template and a call to CreateDialog (or one of its siblings). You can make this template either by doing some drag and drop operations, which then writes a resource file script out, or you can make it entirely in code by manually positioning things where they belong.

Unfortunately, this doesn't mesh with the REALbasic way of doing things at all. REALbasic doesn't store your window layouts as resource scripts (think of this like the old resource-based way of designing windows on Mac Classic), and what's more, it'd be a royal PITA to try to construct the dialog on the fly as the information isn't stored in a fashion that's conducive to that idea.

However, with a bit of ingenuity and black magic skills, it's still possible to accomplish this. Everything boils down to being a window in Windows. And all windows are eventually created with a call to CreateWindow. So you know, deep down inside, that the dialog box creation is just a wrapper around CreateWindow at some level. So there's our jumping off point.

The first sleuth-work was finding out what all the various style flags should be, and this includes figuring out the WNDCLASS to be using. By using Spy++ to check out the window styles, we can see that modeless dialogs made by VC had these for styles:

dwStyle |= WS_POPUP | WS_CAPTION | WS_VISIBLE | WS_CLIPSIBLINGS | DS_3DLOOK | DS_SETFONT | DS_MODALFRAME;
dwStyleEx = WS_EX_LEFT | WS_EX_LTRREADING | WS_EX_DLGMODALFRAME | WS_EX_WINDOWEDGE | WS_EX_CONTROLPARENT;

What's more: after looking in Spy++, the window class' name is actually an ordinal: "#32770" I did a bit of checking online, and Microsoft does document this as being a user dialog box class style.

Great, so how do we mimic this? Well, the naive approach is to simply go nuts:

hWnd = CreateWindowEx( dwStyleEx, "#32770", "The dialog title", dwStyle, x, y, cx, cy,
		parentHandle, NULL, gInstance, NULL );

However, this poses a problem. Since you're using the system's dialog WNDCLASS to create your window, you have absolutely no control over the WndProc. So how do you get your window's messages so that you can respond to things you've created on the window?

Again, sticking with the naive approach, you can subclass the window, like this:

::SetWindowLong( hWnd, GWL_WNDPROC, (long)YourWndProc );

This will switch out the old WndProc with your new one. But as I said before, this is naive (and not just because an x64 version of your application won't work), and there's a much better way to solve the problem.

The reason this approach isn't ideal is because you're still going to miss out on a bunch of messages. When you call CreateWindow, the window manager starts firing off messages left and right, before the call has even completed. For instance, you'll get a WM_CREATE message. By using the system's WndProc until CreateWindow returns, you're going to miss out on messages you may not want to miss out on.

So what is the correct solution then? We know we have to make our own WNDCLASS, because that's the only way we can setup the WndProc to be called immediately. However, we want to still have all the other class settings of the system's dialog box.

Never fear, there's still a good solution for you. GetClassInfo will let you get all of the class information out of an existing registered class. So we'll use that to register a new WNDCLASS.

WNDCLASSEX myModelessDialogClass = { 0 };
if (::GetClassInfoEx( NULL, "#32770", &myModelessDialogClass )) {
  myModelessDialogClass.lpfnWndProc = YourWndProc;
  myModelessDialogClass.lpszClassName = "MyModelessDialog";

  ::RegisterClassEx( &myModelessDialogClass );

}

Ta da! What we've done is made a copy of the system's dialog class, and then modified it to point to our own WndProc. Then we registered this new class under a new name. So now we have to change our call to CreateWindow to use the new class name.

hWnd = CreateWindowEx( dwStyleEx, "MyModelessDialog", "The dialog title", dwStyle, x, y, cx, cy,
		parentHandle, NULL, gInstance, NULL );

Huttah! We now have a less naive implementation which yields us the same visual style as the system's modeless dialog class. What this gets us is the ability to swap out frame types without regard to style. So we can have a Document window and a Modeless Dialog window all using the same underlying code, without requiring the use of dialog templates (either in the resource fork or in memory).