Using code from Sign.CPP (a sample file in the SDK), I was able to reproduce the problem. However, the return code was 1 (true) -- I was able to select a cert, but the hook did not get called.
There is a possble clue:
Note that the docs refer you to "See Hooks" which is at
http://msdn.microsoft.com/en-us/library/ms632589(VS.85).aspx
and is all about Windows Hooks... which seems odd to me. This is really a callback, not a hook, per se.
However, one item in there might apply. Note that in SetWindowsHookEx http://msdn.microsoft.com/en-us/library/ms644990(VS.85).aspx the docs now say that you need 64-bit hook code to inject into a 64-bit process. Like I say, it does not seem like it should apply, but it might.
Are you having this problem on a Win7 or Vista 64-bit machine?
====================
If you can't fix this, then a possible workaround would be to spin-off a thread that would watch for the dialog to appear then move it. See Simple Multithreading in Visual C++