# Stereo rectification done wrong :( **URL:** \ **Category:** Python\ **Tags:** calib3d\ **Created:** [April 24, 2024, 11:56pm UTC](https://forum.opencv.org/t/stereo-rectification-done-wrong/17410 "2024-04-24T23:56:12Z")\ **Posts on this page:** 1\ **Page:** 1
**Author:** ![mdupr](https://avatars.discourse-cdn.com/v4/letter/m/c2a13f/32.png) [@mdupr](https://forum.opencv.org/u/mdupr)\ **Post date:** [April 24, 2024, 11:56pm UTC](https://forum.opencv.org/t/stereo-rectification-done-wrong/17410/1 "2024-04-24T23:56:13Z")
As others before me I am getting frustrated trying to calibrate a pair of stereo cameras. I am using opencv 4.9.0 and a solid Charuco board (18x12) where each squares is about 6cm each. I took 30 images from different distances and position. The calibration of each camera works fine and leads to a errors of 0.5. but the stereo calibration error is about 20. left images: ![left0](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/d/d7ca2aab065e05c988bce86d8e5513c2a0ea411d.png) ![left1](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/f/f916c698640cc80f1146f515fb50b221eb4644d3.png) ![left2](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/a/aaccea91bde5ede06b81871191cad1162f999b40.png) ![left3](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/7/7be49fdec75bf7e0194ba7beccd9c580b1f722de.png) ![left4](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/4/4205491896b8c3bb912029fa5c9a62db05b6d018.png) ![left5](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/f/f9fbb82744e44bcd66e1fa31953e400c69d856ec.png) ![left6](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/0/0271c040dd81ba4d6aa7120db2a6106d77b1d37d.png) ![left7](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/d/df2f12d62e711b2934358e6458e9f8404d6511ed.png) ![left8](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/6/6244ac4f49d419b7eb35bf0da63a2ae52e0dc7df.png) ![left9](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/1/1cc710629b67f69f20b98b04aaca774a747c4591.png) ![left10](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/7/778e19ebdc70d562358fef27d969ff38b9ee8456.png) ![left11](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/4/4d0bebfa7249cecdc95d6ef4114f5cd85805e31b.png) ![left12](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/c/ce80ec5c43861b69cf62e3a21f87cff6d48b1af4.png) ![left13](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/a/a8a6aa187bba98f55b3d7c3ad44b37c35884aef7.png) ![left14](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/8/8ccbb8cab4c44cbd78f77ed779c42614011887b3.png) ![left15](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/e/ea278776c243eb2d87d54965b163b7180d797a16.png) ![left16](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/d/d34200b8779c3e84ccf872d8b33443880bdea3a8.png) ![left17](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/2/2dba493ab9785fe95c4864b58c8f12aef2a247a0.png) ![left18](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/9/9f78ee9b40ff73ae170fead6e5a18eb771e82a58.png) ![left19](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/d/d5232c13d8965328a70ef43d38e623c26fd881e7.png) ![left20](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/a/a0a136802ca6cb8e982abb46b5de89dccf1b2fe4.png) ![left21](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/c/c858d52dc44e39e601b893ba90c21384c2df62ac.png) ![left22](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/b/b1b6d7b8148324d93566453a03371614f8220a72.png) ![left23](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/5/51014c9ba01a0f4f925315af4009b56c4eae3a08.png) ![left24](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/c/c21e8a9e132ee1043d8a5906d1334aa8c7ec13c3.png) ![left25](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/7/77906212fcb788d16b278928e72bbeb404719566.png) ![left26](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/c/c9ac01cacba70dcb7793488e6902b66f12844ddc.png) ![left27](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/e/e73b69c417785c82fa772ff6f5b5d492c841fd0a.png) ![left28](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/e/e784790965b5726869c61df57089b6c49baffd07.png) ![left29](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/f/f7e5d287554361d3ac09249b56d3cfd3388428c3.png) ![left30](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/9/9a2434dd7e702e1e094cb343876740ad03ac8859.png) right images: ![right0](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/b/bd2c26bfb181a48369e6526f04b58221accb8327.png) ![right1](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/c/c477858d7181b1f5914435fe33c14905de85b176.png) ![right2](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/6/679f248718e22e562dff5c95110bccd2f4145bd0.png) ![right3](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/a/adf78e8d9a73280fc3c4f0f67aeda4f84bc4adb5.png) ![right5](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/0/0267fddb60cfe212460808ee2be47c17c5672553.png) ![right6](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/2/2a1591f10f4b1101a5af72453a94325606aaf07e.png) ![right7](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/d/daf624be5b996729f01cf724bd9426ed7f7436d6.png) ![right8](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/3/39b3fc43c792c94a505584937daa9a5de1f57269.png) ![right9](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/0/05b1ee62c886e9c68b47b226afe9f0e31871efd5.png) ![right10](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/a/a7b367c73c539315b73a8c7d162a65c7bfaf6337.png) ![right11](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/9/970aef88c18233579714a5abaf01e7de8bad2179.png) ![right12](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/8/8dc92444976a930efd161e556492b5f0b70fcdef.png) ![right13](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/9/932a6a0170466e2155ca6d28bbcabca5d7b9215e.png) ![right14](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/1/15a9106bfd33ff2c30d8b33ffcc34c16d1f4fd28.png) ![right15](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/d/d57987dab56dccbfbeab1e26859ee830d096c8a4.png) ![right16](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/5/52e05287afd82a7fc7bd8e3121a4d63c6b37e118.png) ![right17](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/1/17ed09fecfcc6bb63f2f219baf293b073a28546c.png) ![right18](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/c/c87f023c845c25888bf9e8fc674661a34f00cef5.png) ![right19](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/1/1a7cd5353ac128854594091f58a32016fb03a31a.png) ![right20](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/f/fc3a33b2ad8930420d909037b57cd5c45af8e3a4.png) ![right21](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/c/c18a458b4a7e90158f9c19e665f43de72c1840a0.png) ![right22](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/8/8716a5565d9243c643f2b22c6421004e9902e89a.png) ![right23](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/4/4d6dba486488b6d9adb10165336e49778b4fe030.png) ![right24](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/a/ab78acba60e02430aad25680cb9b6af94b88383c.png) ![right25](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/1/16655222bd789fc0c9d12ca5668cfe9573b016c2.png) ![right26](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/e/e1790041ec2708ed895c4ba8382e67d46e50d1d8.png) ![right27](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/d/d02f3b30ae3c0f33c5f40017dc8de4c4846daa06.png) ![right28](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/6/6caaa2dfac685bc36e17a4cbd0e53ea7e1f670cd.png) ![right29](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/c/c955d75da6f71ef0cacff7a61a1f279e804401df.png) ![right30](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/f/fa9dbee3a942ee6cfc2a2de556a0964e9b36c915.png) my images look fine to me and i could check that the corners found between each frame correspond to the same ones and to the good Charuco IDs: ![image](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/6/6d8a03e55eb8590e079f9c7d4142cbc032b81d37.jpeg) ![image](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/d/d8f28550c2a92c0616d7048310fc2fc2efe495b4.jpeg) Obviously with such stereo calibration error the rectified maps are useless and end up zooming in a tiny portion of the images. I show here a stack of the original/ “rectified” images: ![image](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/5/5c0090afb0b7d8f3da018d02b1ca4b53c88f1dc2.png) Last and not least, here is the code I am using: ```auto import cv2 #version 4.9.0 import numpy as np from glob import glob import matplotlib.pyplot as plt def read_chessboards(images,board): found_corners = [] allIDs = [] # Ssub pixel corner detection criteria criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 100, 0.00001) count=0 img_ind=[] for gray in images: if len(gray.shape)==3: gray = cv2.cvtColor(gray, cv2.COLOR_BGR2GRAY) corners, ids, rejectedImgPoints = cv2.aruco.detectMarkers(gray, aruco_dictionary) if len(corners)>0:# if some corners are found we go through subpixel detection # Ssub pixel resolution of the corners for corner in corners: cv2.cornerSubPix(gray, corner, winSize = (3,3), zeroZone = (-1,-1), criteria = criteria) res2 = cv2.aruco.interpolateCornersCharuco(corners,ids,gray,board) if res2[1] is not None and res2[2] is not None and len(res2[1])>3 and count%1==0: found_corners.append(res2[1]) allIDs.append(res2[2]) img_ind.append(count) else: print('Not enough corners found in image ' +str(count) +'.') count+=1 imsize = gray.shape return found_corners,allIDs,imsize #allCorners,allIds,imsize=read_chessboards(images) def calibrate_camera(allCorners,allIds,imsize,board): cameraMatrixInit = np.array([[1000., 0., imsize[0]/2.], [0., 1000., imsize[1]/2.], [0., 0., 1.]]) distCoeffsInit = np.zeros((5,1)) flags = (cv2.CALIB_USE_INTRINSIC_GUESS + cv2.CALIB_RATIONAL_MODEL + cv2.CALIB_FIX_ASPECT_RATIO) #flags = (cv2.CALIB_RATIONAL_MODEL) (ret, camera_matrix, distortion_coefficients0, rotation_vectors, translation_vectors, stdDeviationsIntrinsics, stdDeviationsExtrinsics, perViewErrors) = cv2.aruco.calibrateCameraCharucoExtended( charucoCorners=allCorners, charucoIds=allIds, board=board, imageSize=imsize, cameraMatrix=cameraMatrixInit, distCoeffs=distCoeffsInit, flags=flags, criteria=(cv2.TERM_CRITERIA_EPS & cv2.TERM_CRITERIA_COUNT, 10000, 1e-9)) print('Camera calibration error: ' ,ret ) return ret, camera_matrix, distortion_coefficients0, rotation_vectors, translation_vectors #%% def calibrate_stereo(board,left_Corners,left_IDs,left_imsize,left_camera_matrix, left_distortion_coefficients,right_Corners,right_IDs,right_imsize,right_camera_matrix, right_distortion_coefficients,left_img=None,right_img=None): ## calibrate stereo paright allCorners=board.getChessboardCorners() #array of realworld point 3D #sort and keep corners onl if they appear in the 2 set of images right and left left_corners_sampled = [] right_corners_sampled = [] world_corners_sampled = [] left_IDs_sampled=[] right_IDs_sampled=[] # loop over all left ID/corners: to sort the corners found in each frames for i in range(len(left_Corners)):# loop over pair of images frame_left_corners=[] frame_right_corners=[] frame_world_corners=[] frame_left_IDs=[] frame_right_IDs=[] for ii in range(len(left_IDs[i])):# loop over left corners of frame i for jj in range (len(right_IDs[i])): # loop over right corners of frame j if left_IDs[i][ii]==right_IDs[i][jj]: # if IDs are the same for right and left we add the corners to our list of points for extrinsics calib. frame_left_IDs.append(left_IDs[i][ii]) frame_right_IDs.append(right_IDs[i][jj]) frame_world_corners.append(allCorners[left_IDs[i][ii]]) #real world corners frame_left_corners.append(left_Corners[i][ii]) # left corners frame_right_corners.append(right_Corners[i][jj]) # right corners continue left_IDs_sampled.append(np.array(frame_left_IDs)) right_IDs_sampled.append(np.array(frame_right_IDs)) left_corners_sampled.append(np.array(frame_left_corners, dtype=np.float32)) right_corners_sampled.append(np.array(frame_right_corners, dtype=np.float32)) world_corners_sampled.append(np.array( frame_world_corners,dtype=np.float32).squeeze()) if left_img and right_img:# to plot images with corners and their IDs left=cv2.aruco.drawDetectedCornersCharuco(left_img[i].copy(),left_corners_sampled[i], left_IDs_sampled[i]) right=cv2.aruco.drawDetectedCornersCharuco(right_img[i].copy(),right_corners_sampled[i], right_IDs_sampled[i]) stack=np.hstack((left,right)) cv2.imshow('left and right frame #' + str(i)+ 'IDs',stack) cv2.waitKey(0) cv2.destroyAllWindows() cv2.waitKey(1) stereo_criteria = (cv2.TERM_CRITERIA_COUNT +cv2.TERM_CRITERIA_EPS, 1000, 1e-9) #perform the real calibration ret, M1, d1, M2, d2, R, T, E, F = cv2.stereoCalibrate(objectPoints=world_corners_sampled, imagePoints1=left_corners_sampled, imagePoints2=right_corners_sampled, cameraMatrix1=left_camera_matrix, distCoeffs1=left_distortion_coefficients, cameraMatrix2=right_camera_matrix, distCoeffs2=right_distortion_coefficients, imageSize= left_imsize, criteria=stereo_criteria, flags=cv2.CALIB_FIX_INTRINSIC) print('stereo calibration error: ' ,ret) return ret,R,T,E,F # define chAruco board nx=18 ny=12 aruco_dictionary = cv2.aruco.getPredefinedDictionary(cv2.aruco.DICT_5X5_250)# this is the dictionary used to generate the boards board = cv2.aruco.CharucoBoard((nx, ny),1, 0.75, aruco_dictionary) # this is the board used for calibration nx, by, squarelength, markerlength,dictionary) # load left and right images leftnames=sorted(glob('left*.png')) rightnames=sorted(glob('right*.png')) left_im=[] right_im=[] for im1, im2 in zip(leftnames, rightnames): left_im.append(cv2.imread(im1, 1)) right_im.append(cv2.imread(im2, 1)) # perforrm lens distortion - left camera left_Corners,left_IDs,left_imsize=read_chessboards(left_im,board) left_ret, left_camera_matrix, left_distortion_coefficients, left_rotation_vectors, left_translation_vectors=calibrate_camera(left_Corners,left_IDs,left_imsize[::-1],board) # perforrm lens distortion - rightcamera right_Corners,right_IDs,right_imsize=read_chessboards(right_im,board) right_ret, right_camera_matrix, right_distortion_coefficients, right_rotation_vectors, right_translation_vectors=calibrate_camera(right_Corners,right_IDs,right_imsize[::-1],board) #perform stereo calibration stereo_err,R,T,E,F=calibrate_stereo(board,left_Corners,left_IDs,left_imsize[::-1],left_camera_matrix, left_distortion_coefficients,right_Corners,right_IDs,right_imsize[::-1],right_camera_matrix, right_distortion_coefficients,left_im,right_im) R_left,R_right,P_left,P_right,Q,left_ROI,right_ROI=cv2.stereoRectify(left_camera_matrix, left_distortion_coefficients, right_camera_matrix, right_distortion_coefficients, left_imsize[::-1], R, T) leftmap = cv2.initUndistortRectifyMap(left_camera_matrix,left_distortion_coefficients, R_left,P_left,left_imsize[::-1],cv2.CV_32FC1) rightmap = cv2.initUndistortRectifyMap(right_camera_matrix,right_distortion_coefficients, R_right,P_right,right_imsize[::-1],cv2.CV_32FC1) count=0 #plot the raw images and the rectified ones: for left, right in zip(left_im, right_im): left_rect= cv2.remap(left, leftmap[0], leftmap[1], cv2.INTER_LANCZOS4, cv2.BORDER_CONSTANT,0) right_rect= cv2.remap(right, rightmap[0], rightmap[1], cv2.INTER_LANCZOS4, cv2.BORDER_CONSTANT,0) plt.figure() plt.subplot(2,2,1) plt.imshow(left),plt.title(str(count) +' raw left') plt.subplot(2,2,2) plt.imshow(right),plt.title(str(count) +' raw right') plt.subplot(2,2,3) plt.imshow(left_rect),plt.title('rectified left') plt.subplot(2,2,4) plt.imshow(right_rect),plt.title('rectified right') plt.show() count+=1 ``` Is there anything obvious that I am doing wrong ? Thanks for any help !