I guess you could go through both lists of points and only keep what was seen in both cameras. I haven’t done much with stereo calibration, so I’m not really sure how Charuco plays with that.
However, I believe that you can calibrate the intrinsics separately from the relative pose. That is what I would do, but I’m not certain it’s the better approach, it just seems better to me.
Here is what I would do:
- Calibrate camera 1 intrinsics with Charuco board.
- Calibrate camera 2 intrinsics with Charuco board.
Once you are satisfied with the intrinsic calibration results, save the camera matrix and distortion coefficients and use them in the future. I’m suggesting doing this as a separate one-time step so you can focus on getting good results - you may not use the same calibration target / input images for your stereo calibration.
- Calibrate them as a stereo pair, making sure to pass in the intrinsics you got from steps 1 and 2, and also being sure to pass in the CALIB_FIX_INTRINSIC flag.
I’m a little bit out of my depth, though. You might be able to get great results by calibrating the intrinsics and stereo pose at the same time. In any case if the stereoCalibrate function wants the same points for both cameras, you will have to take care of that somehow.
Good luck.