Problems using FindHomography to transfer from webcam to 3D scene

[Andrew_Chatham]

I’m using opencvsharp with unity, though this seems platform-agnostic, and I’m struggling to get the transforms right.

I want to point a webcam at a screen, detect contours, then project those contours back into the scene as unity objects. The goal is for virtual objects in unity to interact with the contours as if they’re in the same space. I’m doing this in 2D (z=0), so seems like a homography problem (webcam of screen → scene → screen), but I keep getting unexpected results. I wonder if I’m thinking about the problem incorrectly or just messed up some math.

I’m ignoring camera distortion for now and just using FindChessboardCorners → FindHomography → PerspectiveTransform, but the projected results are warped in a way I haven’t been able to debug.

Here’s an example of the contour being mis-rendered:

My calibration approach:

• Render chessboards with known scene coordinates. Use FindChessboardCorners to build up a correspondence between webcam (src points) and 2D scene coordinates (dest points).
• Run FindHomography, and confirm projecting webcam points to scene coordinates has low projection error.

Runtime:

• Use contours = FindContours(webcam_image), followed by scene_coordinates_2d = PerspectiveTransform(contours, H). Then add balls at the scene coordinates.

[Andrew_Chatham]

Adding an image of the projected chessboard corners, which looks pretty good to me (ignoring distortion for now):

[Andrew_Chatham]

Er, actually, this all seems to be working, and I was just failing to put the objects directly on the screen so they were all in the same Z plane as the calibration points. Will attempt to delete this post.