Using a custom WindowProc from Ruby - Wide Awake Developers

This is off the beaten path today, maybe even off the whole reservation. Still, I searched for some code to do this, and couldn't find it. Maybe this will help somebody else trying to do the same thing.

I'm currently prototyping a desktop utility using Ruby and wxRuby. The combination actually makes Windows desktop programming palatable, which is a very pleasant surprise.

Part of what I'm doing involves showing messages with Snarl. I want my Ruby program to generate messages that can be clicked. Snarl is happy to tell you that your message has been clicked. It does it by sending your window a message, using whatever message code you want.

So, for example, if I want to get a WM_USER message back, then I create a new notification like this:

@msg = Snarl.new('Clickable message', {:message => 'Click me, please!', :timeout => Snarl::NO_TIMEOUT, :reply_window => @win_handle, :reply_window_message => Windows::WM_USER})

If the user clicks on my message, I'll get a WM_USER event delivered to my window (identified by @win_handle). Since I'm using wxRuby, which wraps wxWidgets, that presents a bit of a problem. Although wxWidgets allows you to subclass its default window proc, wxRuby does not. A couple of forum posts suggested using the Windows API to hook the window proc, which is what I did.

Here's the code:

begin
  require 'rubygems'
rescue LoadError
end

I installed wxRuby as a gem, so that's boilerplate.

require 'lib/snarl'
require 'wx'
require 'windows/api'

module WindProc
  include Windows
  
  GWL_WNDPROC = -4

  WM_USER = 0x04FF

  API.auto_namespace = 'WindProc'
  API.auto_constant = true

  API.new('SetWindowLong', 'LIK', 'L', 'user32')
  API.new('CallWindowProc', 'PIIIL', 'L', 'user32')
end

This module just gets me access to the Windows API functions SetWindowLong and CallWindowProc. SetWindowLong is deprecated in favor of SetWindowLongPtr, but I couldn't get that to load properly through the windows/api module. At some point, when you're prototyping something, you just have to decide not to solve every puzzle, especially if you can find a workable alternative.

API.new() constructs a Ruby object implemented by some C native code. It uses the prototype string in the second argument to translate Ruby parameters into C values when you eventually call the API function. The conversion is done in glue code that knows how to map some Ruby primitives to C values, but it's not all that bright. In particular, there's no way to introspect on the Win32 API itself to see if you're lying to the glue code. In fact, I'm lying a little bit here. The prototype I used---'LIK'---tells the API module that I'm looking for a function that takes a long, an integer, and a callback. Strictly speaking, this should have been 'LIL', but I needed the glue code to convert a Ruby procedure into a C pointer.

The next section defines a subclass of Wx::Frame, the base type for all standalone windows.

class HookedFrame < Wx::Frame
  def initialize(parent, id, title)
    super(parent, -1, title)

    evt_window_create() { |event| on_window_create(event) }
  end

I register a handler for the window create event. At this point, I'm still within the bounds of wxWidget's own event handling framework. The interesting bits happen inside the on_window_create method.

  def on_window_create(event)
    @old_window_proc = 0
    @my_window_proc = Win32::API::Callback.new('LIIL', 'I') { |hwnd, umsg, wparam, lparam|
      if not self.hooked_window_proc(hwnd, umsg, wparam, lparam) then
        WindProc::CallWindowProc.call(@old_window_proc, hwnd, umsg, wparam, lparam)
      end
    }
    @old_window_proc = WindProc::SetWindowLong.call(self.handle, WindProc::GWL_WNDPROC, @my_window_proc)
  end

There are several juicy bits here. First, I'm using Win32::API::Callback.new() to create a callback object. How does this get used? It's a little roundabout. When I call WindProc::SetWindowLong(), I pass the callback object. (This is why I used 'LIK' as the prototype string earlier.) Now, WindProc::SetWindowLong() isn't just a pointer to the native Windows library function. It's actually a Ruby object that wraps the library function. The API object is implemented by C code. Like the API object, the callback object is a Ruby object implemented by C code. In particular, it has an ivar that points to a Ruby procedure. Because I passed a block to Callback.new(), the block itself will be the procedure. Inside API.call(), any argument of type "K" gets set as the "active callback" and then substituted with a C function called CallbackFunction. CallbackFunction looks up the active callback, translates parameters according to the callback's prototype, then tells Ruby to invoke the proc associated with the callback.

Whew.

So, I call SetWindowLong.call(), passing it the Callback I created with a block. SetWindowLong.call() ultimately callls the Windows DLL function SetWindowsLong, passing it the address of CallbackFunction. When Windows calls CallbackFunction, it looks up the Ruby Callback object and invokes it's procedure.

Another oddity. For some reason, although the callback object has an instance variable called @function, there seems to be no way to set it after construction. If you pass a block, @function will point to the block. If you don't, @function will be nil, with no way to set it to anything else. In other words, the API will happily let you create useless Callback objects.

The rest is easy. Inside my block, I just call out to a method that can be overridden by descendants of HookedFrame. My test implementation just blurts out some stuff to let me know the plumbing is working.

  def hooked_window_proc(hwnd, uMsg, wParam, lParam)
    puts "In the hook: 0x#{uMsg.to_s(16)}\t#{wParam}\t#{lParam}\n"
    if uMsg == NotifierApp::WM_USER then
      puts "That's what I've been waiting to hear:\t#{wParam}\t#{lParam}\n"
      true
    end
    false
  end

As I reviewed this post, I realized a something else. ActiveCallback is static in the C glue code. That means there can only be one callback set at a time. If I called some other Windows API function with its own callback, that would overwrite the reference to my Ruby code. But, Windows would still keep calling to the same pointer as before. In other words, calling any other Windows API function that takes a callback would cause that callback to become my window proc! Yikes!

Overall, this works, but seems like a kludge. Ironically, even as I got this working, I started getting dissatisfied with Snarl itself. I think I need more flexibility to display persistent information, rather than just alerts.