Yes, you are correct. It took me a while to track down my error, but once I did it started working as expected.
My error: I had created both maps as CV_16SC2 type, but the second map needs to be CV_16UC1.
So this is great news for me - I think I can do exactly what I want. Sorry for the confusion, and thanks for the help.
-Steve