Help with augmented reality reprojection

Python
,
1

I have a stereo calibrated camera-projector setup where the cameras are different resolutions where cam1 res < cam2 res. My plan is to get an image from cam1, do some CV processing on it to get a mask of the object I am interested in, and then re-project on it using the projector.

Let’s say I am using a planar calibration board. I can get the pose from estimatePoseCharucoBoard. Through some CV I can then get the mask of just the board and at this point know the world coordinate (which is wrt cam1) of every pixel of this mask.

I can then use the known extrinsics along with cv2.projectPoints with cam2’s intrinsics to project this back to cam2’s image plane.

My question is that now I simply have a list of pixel coordinates. I can create an empty image and set every pixel to the color copied from cam1 image, but the image will be full of holes. I can probably use morphological operators to fill holes, but I feel there’s a better way to do this that I’m missing.

Is there an efficient way to remap an image in cam1 view to an image in cam2 under stereo calibration that takes care of the interpolation between points given cam2s resolution is larger than cam1? Or is there a better way to do this which is what I’m missing?

2

If I understand what you are trying to do, I think you could use initUndistortRectifyMap() along with remap(). The remap function will take care of interpolating, so you shouldn’t have holes in the output image.

You might also consider using OpenGL or D3D with a texture-mapped mesh if you want high performance.