Stereo cameras calibrate well indivdually, but stereo calibration is distorted

Python
,
1

I am trying to use stereo cameras for 3D position detection. I took 150 clean images at different angles using a 7x12 chessboard. Calculated projection error is 0.094 and 0.097 for the left and right cameras respectively. As you can see, both the left and right cameras can be rectified quite nicely, but the image warps and rotates when I do stereo calibration+rectification. I have included relevant parts of my code below. Any thoughts on improving stereo rectification results?

calibration.PNG
calibration.PNG882×718 35.5 KB

.
# start code

ret, roiL, old_mtxL, mtxL, distL, _, _ = calibrate_cam('L')
ret, roiR, old_mtxR, mtxR, distR, _, _ = calibrate_cam('R')
img_ptsL,obj_pts,w,h = load_calib_points('L')
img_ptsR,obj_ptsR,w,h = load_calib_points('R')
flags = 0
flags |= cv2.CALIB_FIX_INTRINSIC 
criteria_stereo= (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001)
rectify_scale= -1
rect_l, rect_r, proj_mat_l, proj_mat_r, Q, roiL, roiR= cv2.stereoRectify(mtxL, distL, mtxR, distR, (w,h), Rot, Trns, rectify_scale,(0,0))

Left_Stereo_Map= cv2.initUndistortRectifyMap(mtxL, distL, rect_l, proj_mat_l, (w,h), cv2.CV_16SC2)
Right_Stereo_Map= cv2.initUndistortRectifyMap(mtxR, distR, rect_r, proj_mat_r, (w,h), cv2.CV_16SC2)

xL,yL,wL,hL = roiL
xR,yR,wR,hR = roiR
ret, imgL = capL.read()
ret, imgR = capR.read()
if ret:
    Left_nice= cv2.remap(imgL,Left_Stereo_Map[0],Left_Stereo_Map[1], cv2.INTER_LANCZOS4, cv2.BORDER_CONSTANT, 0)
    Right_nice= cv2.remap(imgR,Right_Stereo_Map[0],Right_Stereo_Map[1], cv2.INTER_LANCZOS4, cv2.BORDER_CONSTANT, 0)

   # Crop the image
   Left_nice = Left_nice[yL:yL+hL, xL:xL+wL]
   Right_nice = Right_nice[yR:yR+hR, xR:xR+wR]

   # Show both Stereo Images
   cv2.imshow('Right Stereo',cv2.resize(Right_nice,(640,480),interpolation = cv2.INTER_AREA))
   cv2.imshow('Left Stereo',cv2.resize(Left_nice,(640,480),interpolation = cv2.INTER_AREA))
   cv2.waitKey(1)
2

that’s a bad calibration.

low reprojection error is necessary but not sufficient.

could you show a few calibration images as examples, the complete sets of points you use for calibration (and what size the images are), and all the results (camera matrices, distcoeffs, …)?

I’d like to see original size images, not thumbnails. the forum should accept large images. if it doesn’t, I’d recommend imgur.com or some other image pastebin.

3

Thanks for the response! OpenCV forum will only let me post one picture, so here is a Google Drive link to a few characteristic images: cvpost - Google Drive

The entire set is composed of 150 images that were tested for finding corners before saving and taken from many angles and distances. I also have a similar set of 200 calibration images using a 14x20 board, but with as bad or worse stereo calibration results. All pictures are 1920x1080. As you can see, my calibration board is just a paper print-out glued onto a cardboard backing, so perhaps that could be what is throwing off my calibration(?)

The object points and image points arrays are very long, so I have included them in the Google Drive folder as ‘L_textout.txt’ and ‘R_textout.txt,’ but here are the camera matrices, etc.:

# Left cam matrix:
array([[1.04058862e+03, 0.00000000e+00, 9.85176023e+02],
       [0.00000000e+00, 1.03877429e+03, 5.20636544e+02],
       [0.00000000e+00, 0.00000000e+00, 1.00000000e+00]])
# Left dist matrix:
array([[-3.58425127e-01,  1.49647023e-01,  1.11197103e-04,
        -1.09510953e-04, -3.52071106e-02]])
# Left Projection matrix:
array([[1.04091467e+03, 0.00000000e+00, 9.07840721e+02, 0.00000000e+00],
       [0.00000000e+00, 1.04091467e+03, 5.22196270e+02, 0.00000000e+00],
       [0.00000000e+00, 0.00000000e+00, 1.00000000e+00, 0.00000000e+00]])

# Right camera matrix:
array([[1.04364417e+03, 0.00000000e+00, 9.40679608e+02],
       [0.00000000e+00, 1.04305505e+03, 5.31817126e+02],
       [0.00000000e+00, 0.00000000e+00, 1.00000000e+00]])
# Right dist matrix:
array([[-3.63986021e-01,  1.68977770e-01, -1.09848259e-03, 2.30533686e-04, -5.30913837e-02]])
# Right projection matrix:
array([[ 1.04091467e+03,  0.00000000e+00,  9.07840721e+02, -1.09587625e+05],
       [ 0.00000000e+00,  1.04091467e+03,  5.22196270e+02, 0.00000000e+00],
       [ 0.00000000e+00,  0.00000000e+00,  1.00000000e+00, 0.00000000e+00]])
4

I still have not found a solution, but I started comparing the retval results for multiple calibration attempts. Interestingly, my best visual result yet returns a retval of 5x10^23 while the retval=4.9 calibration is almost unusable.

Below is the pair RMS error vector from my most recent attempt using 119 images, obtained using cv2.stereoCalibrateExtended(). Almost all pairs have RMS values between 1 and 10. I’m not sure what value range would be expected from a good calibration.

[6.06834313 5.71365156] error at index 0
[3.38290363 4.3055713 ] error at index 1
[1.86214948 3.187891 ] error at index 2
[2.93933095 3.06527322] error at index 3
[4.55148241 4.98136996] error at index 4
[3.0055067 3.4888883] error at index 5
[2.05358923 1.34138594] error at index 6
[1.22332479 1.49938267] error at index 7
[2.99293359 4.47459382] error at index 8
[2.00910693 1.93871743] error at index 9
[2.12868743 2.14433815] error at index 10
[1.07670399 0.91021339] error at index 11
[1.40604952 1.16973695] error at index 12
[1.9086339 1.96844671] error at index 13
[1.29671214 1.4114028 ] error at index 14
[2.95353393 3.19499268] error at index 15
[4.71529754 5.03102868] error at index 16
[4.87214476 4.46010892] error at index 17
[1.50514391 1.81589116] error at index 18
[1.84465933 1.44722133] error at index 19
[4.63248286 4.45415031] error at index 20
[4.42014886 5.53129147] error at index 21
[3.74570958 4.52927189] error at index 22
[3.41816872 3.85283641] error at index 23
[4.46340798 5.08749132] error at index 24
[4.18749792 3.87556527] error at index 25
[4.39352921 4.54828518] error at index 26
[6.18410162 6.26505138] error at index 27
[4.99545868 5.21264914] error at index 28
[3.53152821 3.36703312] error at index 29
[5.31512735 5.65458673] error at index 30
[6.23464699 6.47222069] error at index 31
[6.25151893 7.53999652] error at index 32
[6.39202128 7.25018456] error at index 33
[6.15281924 5.98613213] error at index 34
[5.96315106 5.09192918] error at index 35
[6.91649515 5.65377678] error at index 36
[7.65495418 7.58914908] error at index 37
[7.73281275 7.71239709] error at index 38
[4.09516101 5.3595089 ] error at index 39
[2.4943271 4.19407254] error at index 40
[1.17557383 1.16531351] error at index 41
[0.7459478 0.62375532] error at index 42
[3.07231614 3.38270109] error at index 43
[1.18641144 0.64634279] error at index 44
[0.90669061 0.76192631] error at index 45
[4.76875905 5.63546076] error at index 46
[6.8115398 7.1748164] error at index 47
[5.57972563 5.63864471] error at index 48
[6.02972555 5.33257742] error at index 49
[6.73381126 6.39980093] error at index 50
[8.15303011 7.74632001] error at index 51
[7.2338602 6.39393614] error at index 52
[6.24077521 6.89205777] error at index 53
[6.43957105 6.32552842] error at index 54
[7.29407759 7.28236288] error at index 55
[6.54991586 7.37220607] error at index 56
[5.46225376 6.01477696] error at index 57
[6.17493678 7.38650973] error at index 58
[5.97313804 6.01527279] error at index 59
[4.62997796 4.2039031 ] error at index 60
[6.04956828 5.45400357] error at index 61
[5.36799654 5.32522449] error at index 62
[5.91199384 6.19098547] error at index 63
[4.44583563 4.00946519] error at index 64
[3.2346184 5.04525938] error at index 65
[3.55454846 4.37110598] error at index 66
[5.15939226 5.54205606] error at index 67
[2.06663609 2.27570228] error at index 68
[1.24761109 3.03669366] error at index 69
[3.65961914 3.71547498] error at index 70
[4.23821703 4.16965256] error at index 71
[4.13654487 4.31890582] error at index 72
[5.00151879 5.07375217] error at index 73
[1.76672282 2.27076323] error at index 74
[7.1340586 4.89168949] error at index 75
[5.87658634 5.63419696] error at index 76
[6.48078597 6.09565032] error at index 77
[5.00370446 4.17116761] error at index 78
[2.06482728 1.94896405] error at index 79
[3.81640319 4.44232567] error at index 80
[1.81253695 2.14706334] error at index 81
[3.76411585 3.51682633] error at index 82
[3.4334559 3.07675965] error at index 83
[1.93642284 3.79444632] error at index 84
[7.58995521 7.28501501] error at index 85
[5.85086533 5.85697103] error at index 86
[3.84065796 4.20329424] error at index 87
[3.45614596 2.57025937] error at index 88
[7.22643448 6.98302897] error at index 89
[6.90292167 7.28724454] error at index 90
[7.35345876 7.56259991] error at index 91
[7.67994987 6.61908 ] error at index 92
[4.32266547 3.94982992] error at index 93
[4.81540347 5.46313007] error at index 94
[5.12333516 5.91237465] error at index 95
[7.5030196 7.36087006] error at index 96
[2.37501482 2.62295752] error at index 97
[2.7706025 3.03739147] error at index 98
[2.02289697 3.34395744] error at index 99
[4.25581715 4.46702903] error at index 100
[5.40946559 4.9084641 ] error at index 101
[7.00138173 6.4172637 ] error at index 102
[5.56556734 5.53506035] error at index 103
[7.53817229 7.55725946] error at index 104
[5.61056823 6.00198979] error at index 105
[6.57009805 5.87800926] error at index 106
[6.83737617 6.95561876] error at index 107
[3.91419343 5.12437282] error at index 108
[3.89886821 3.34218091] error at index 109
[1.98621099 2.33821895] error at index 110
[4.24937317 4.36777042] error at index 111
[5.94252135 5.13255889] error at index 112
[1.37991916 0.81024682] error at index 113
[3.19312424 3.55872984] error at index 114
[0.55533167 0.6254616 ] error at index 115
[3.38982196 1.68348722] error at index 116
[2.13695398 1.92168184] error at index 117
[5.59342753 4.6910574 ] error at index 118
[6.29403472 6.02235958] error at index 119

5

@Charles_R : I am buidling a stereo cameras and i am currently facing the same problem. I am having retval for stereo pair ranging between 1 - 5 and i am not sure how to improvise it. Did you find out the reason for your error. Please share some light on the topic.

Many thanks