I’m using opencvsharp with unity, though this seems platform-agnostic, and I’m struggling to get the transforms right.
I want to point a webcam at a screen, detect contours, then project those contours back into the scene as unity objects. The goal is for virtual objects in unity to interact with the contours as if they’re in the same space. I’m doing this in 2D (z=0), so seems like a homography problem (webcam of screen → scene → screen), but I keep getting unexpected results. I wonder if I’m thinking about the problem incorrectly or just messed up some math.
I’m ignoring camera distortion for now and just using FindChessboardCorners → FindHomography → PerspectiveTransform, but the projected results are warped in a way I haven’t been able to debug.
Here’s an example of the contour being mis-rendered:
My calibration approach:
- Render chessboards with known scene coordinates. Use
FindChessboardCornersto build up a correspondence between webcam (src points) and 2D scene coordinates (dest points).
FindHomography, and confirm projecting webcam points to scene coordinates has low projection error.
contours = FindContours(webcam_image), followed by
scene_coordinates_2d = PerspectiveTransform(contours, H). Then add balls at the scene coordinates.