Again, I’m not sure I follow what you are doing exactly (for example why you want to crop the image instead of using the full image)
TL;DR: Make sure you are using calibrated values for cx,cy in every step. I suspect this is the root of your problem.
A few comments:
“Distortion center is irrelevant…” Note that cx,cy is also used in the camera matrix for the 3D->2D projection, so if cx,cy is wrong, then your projections (and everything that relies on them) will be wrong. So, for example, if you used the cx,cy values from the big image when doing the stereo calibration for the cropped image, things will be wrong.
I noticed that your fx,fy values are different. Most modern cameras have square pixels, so it is typically best to use the CALIB_FIX_ASPECT_RATIO when claibrating the intrinsics. Otherwise you allow the optimizer to vary the fx and fy independently, which might result in a better score but a less accurate model (since physically fx and fy are actually the same). This is a minor point and is not what is causing your issues.
I also noticed that the intrinsics for both cameras has cx,cy as exactly the numerical center of the images. This suggests that you passed in a guess for the cx,cy and used the parameter CALIB_FIX_PRINCIPAL_POINT (or just overrode the calibrated values with the numerical center of the image). I didn’t see code for how you called calibrateCamera, so I’m just guessing, but I would suggest you calibrate the intrinsics fully (including cx,cy) because having accurate cx,cy values is very important. As an example, for cameras that I use regularly, an error of just 5 pixels for cx,cy is enough to make the camera unusable until it is re-calibrated.
My best advice at this point is:
- Abandon the idea of cropping images until you can get things working with the full size images - it just complicates things and obscures the source of error.
- Do everything you can reasonably do to make sure the optics stay fixed throughout the whole process. If either camera has autofocus, disable it / use a fixed absolute focus value always, and consider mechanically fixing the lens in position (epoxy) or changing to a manual / lockable lens if possible. If either camera has manual focus or zoom, lock them down mechanically.
- Fully calibrate both cameras, including cx,cy - thes are critically important parameters.
- Use CALIB_FIX_ASPECT_RATIO when calibrating intrinsics.
- Make sure you are being consistent with distorted / undistorted image and parameters. Prefer passing in distorted images with the distortion coefficients. When calling functions and passing undistorted images, make sure you aren’t passing in distortion coefficients.
- For debugging, check each step by using the calibration results to project known 3D points to the images and visually check to make sure they are accurate.
I hope this is helpful.