Disparity Map Nonsense - Why?

Python
, ,
1

I’m working with a stereo camera that has a baseline of 30cm, each lens has a 160° FOV.

I’m trying to create a depthmap, but the created disparity map is nonsense. I have never worked with stereo vision before, so I was wondering what the possible reasons could be, so I can debug it.

I have put quite a lot of effort into calibration, but achieved only a stereo calibration RMS of around 1.3, maybe that is not good enough?
Also, I did the calibration rather close to the cameras (around 1m), and now I’m using it on object a lot further away.

Could the overexposure in the image be a problem?

Any thoughts are welcome!

This is an example image:

test_2
test_21920×721 170 KB

and it’s rectified results:

imgL_rectified
imgL_rectified1640×1232 163 KB

imgR_rectified
imgR_rectified1640×1232 154 KB

And the resulting disparity map:

disparity_map

The code I use:

import numpy as np
import cv2 as cv2
from matplotlib import pyplot as plt

img_width = 3280
img_height = 1232

# load calibration data
npzfile = np.load('./calibration_data/{}p/stereo_camera_calibration.npz'.format(img_height))
leftMapX = npzfile['leftMapX']
leftMapY = npzfile['leftMapY']
rightMapX = npzfile['rightMapX']
rightMapY = npzfile['rightMapY']

imageToDisp = "/test_2.jpg"
pair_img = cv2.imread(imageToDisp,0)
pair_img = cv2.resize(pair_img, (int(pair_img.shape[1]/2), int(pair_img.shape[0]/2)), interpolation = cv2.INTER_AREA)
imgLeft = pair_img [0:img_height,0:int(img_width/2)] #Y+H and X+W
imgRight = pair_img [0:img_height,int(img_width/2):img_width] #Y+H and X+W

imgL = cv2.remap(imgLeft, leftMapX, leftMapY, interpolation=cv2.INTER_LINEAR, borderMode=cv2.BORDER_CONSTANT)
imgR = cv2.remap(imgRight, rightMapX, rightMapY, interpolation=cv2.INTER_LINEAR, borderMode=cv2.BORDER_CONSTANT)

#resize images
image_size = (320, 240)
imgL = cv2.resize(imgL, image_size, interpolation = cv2.INTER_AREA)
imgR = cv2.resize(imgR, image_size, interpolation = cv2.INTER_AREA)

#create disparity map
stereo = cv2.StereoSGBM.create(numDisparities=16*4, blockSize=30)
disparity = stereo.compute(imgL,imgR)

plt.imshow(disparity, cmap='magma')
plt.show()
2

the undistorted-rectified images don’t even point in the same direction. something upto that point must have gone wrong.

1 Like
3

what do you mean with that they don’t point in the right direction?

4

open them both, make sure they overlap perfectly. then switch between them. points at infinity are supposed to not move, be the same. if that isn’t the case, you’ll have a headache getting pixel disparity turned into useful triangulation.

you need to reconsider the entire calibration, intrinsics and extrinsics. browse Knowledge Base – calib.io because calibration does not consist of arbitrarily waving a pattern in front of the cameras.

5

Thanks for the hint.

I’ve been following the steps on this website meticulously, except for one major thing: I wasn’t able to calibrate the camera at working distance, as this would need a huge calibration board. (I’m doing this for my master thesis, so I have limited equipment).

Do you have any ideas for workarounds?

6

if you didn’t actually take a class that explains (and made you practice!) all of this stuff in sufficient depth for you to work with it, then good luck. prepare for all of this to be the thing that will ruin your thesis. it’s not trivial stuff. you should quickly find someone local (and competent in this subject) who can take a look at everything you do (because I don’t have the time to ask for every bit of information you have) and correct every issue. forget your advisor. their job isn’t to help you succeed, just to watch the formalities. if you don’t have the subject matter prerequisites, and your advisor let you begin this thesis anyway, they set you up to fail. if they thought this was trivial, make them prove it.

you could produce synthetic images, then distort them with known coefficients (inversion of remap maps is possible numerically). the extrinsics matrix is just supposed to contain translation and rotation. inspect it. consider the values you get. play with it. come up with completely synthetic data (3d graphics) for which you know all these parameters and can vary them under controlled conditions.

implementing lens distortion is at least an extra day of research and an extra day of trying to implement it. maybe start off assuming there is no distortion.

random google results:

if I had to do this, I’d spend a day or two whipping up a trivial scene (trivial! no fancy poses, beginning with looking dead ahead) with OpenGL or whatever else is sufficiently controllable, where you have two cameras defined by opencv camera matrices (and distortion coefficients if need-be), turned into whatever OpenGL needs, and for the scene just a checkerboard that you can whirl around. make sure the synthetics gives you enough information such that you know the corner points of the board in 3D and 2D, and they make complete sense if you follow the calculations manually. and all of that has to be scriptable, so you can produce calibration datasets with a single command within seconds to minutes.