Stereo rectify to the left image plane

Python
, ,
1

I’m having trouble with stereo rectification, where I want to keep the left camera center stationary and only warp the right image. I’m not sure if there is something I’m missing with the math but I feel like the y and z translation is not being included with my current implementation.

Currently the pipeline I have goes like this:

  • Take images of the checkerboard with left and right camera
  • Perform monocular calibration on both left and right camera with the CALIB_FIX_PRINCIPAL_POINT setting
    • left camera reprojection error is 0.46 pixels
    • right camera reprojection error is 0.41 pixels
  • Perform stereo calibration with the setting CALIB_FIX_INTRINSIC
    • reprojection error is 0.52 pixels
  • Stereo Rectify with some of my own code
    • reprojection error is about 6.5 pixels

The issue is that if I directly use stereoRectify, the reprojection error is about 0.5 pixels which is great. I call it as normal and here is an example image as output:

    r1, r2, p1, p2, _, _, _ = cv2.stereoRectify(
        left_camera_matrix,
        left_dist_coeffs,
        right_camera_matrix,
        right_dist_coeffs,
        dimensions,
        stereo_rotation,
        stereo_translation,
        flags=cv2.CALIB_ZERO_DISPARITY,
        alpha=-1,
    )

EDIT: It seems like as a new user I can only embed one image per post. This image would have shown that if I allow stereoRectify to warp both images as usual, there is low reprojection error but a huge shift in the center of the left image.

The green and red crosshair that don’t correspond to a checkerboard point shows the shift in the center of the image.

The issue is that I need to have the left camera rectified to its own image plane. I can only modify the right image. Here is another code snippet and image output:

    _, _, _, p2, _, _, _ = cv2.stereoRectify(
        left_camera_matrix,
        left_dist_coeffs,
        right_camera_matrix,
        right_dist_coeffs,
        dimensions,
        stereo_rotation,
        stereo_translation,
        flags=cv2.CALIB_ZERO_DISPARITY,
        alpha=-1,
    )

    # Left camera stays completely unchanged
    r1_identity = np.eye(3)
    p1 = left_camera_matrix.copy()
    p1 = np.column_stack((p1, np.zeros((3, 1))))

    # Apply rotation to right camera
    r2_custom = stereo_rotation.T

    # Create P2 with same intrinsics as right camera, but add translation for baseline
    P2_custom = right_camera_matrix.copy()
    P2_custom = np.column_stack((P2_custom, np.zeros((3, 1))))
    baseline_pixels = abs(p2[0, 3])
    P2_custom[0, 3] = -baseline_pixels

stereo_reprojection_pair_0
stereo_reprojection_pair_01920×714 244 KB

Is this something that is possible with the method that I’m using or is there something wrong with how I’m using these functions?