Right, I see. I guess I will try to find the mistake in the opencv source.
I also found this post which seems to be related Odd result on cv::fisheye::undistortPoints and also the same thing on stackoverflow. Too bad noone has figured it out yet (or at least not posted it).