Problem with rectifying pinhole and fisheye image

Python
, ,
1

I have a stereo camera setup which has one pinhole camera and one fisheye camera. The pinhole camera has 40 degrees FoV. The fisheye camera has 120 degrees FoV. In the end, I want to get a depth map from this camera setup. However, I got a bad rectification result before I implemented stereo matching.

I use python OpenCV to implement the project. There are 3 steps before stereo rectification:

  • Step 1: Use padding method to make the pinhole image size equal to fisheye image size
  • Step 2: Calibrate and undistort pinhole and fisheye image individually by using normal camera module
  • Step 3: Pass early calibration parameters to do stereo calibration

Figure 2 - undistortedImg_wide
Figure 2 - undistortedImg_wide548×968 464 KB
Figure 1 - undistortedImg_narrow
Figure 1 - undistortedImg_narrow548×968 401 KB

After doing stereo calibration, I also checked that both images were successfully undistorted by intrinsic and extrinsic parameters. You can see the above figures. The figure on the left is the undistorted fisheye image, and the other one is the undistorted pinhole image. Next, I did stereo rectification by following the steps below :

  • Step 1: Get the rotation and projection matrices by calling cv2.stereoRectify(…)
  • Step 2: Get maps by calling cv2.initUndistortRectifyMap(…)
  • Step 3: Using cv2.remap(…) to generate rectified images

After implementing the above steps, the remapped images show black or highly distorted. You can see the figures below :

Figure 3 - All_black_rectify_result
Figure 3 - All_black_rectify_result671×605 2.74 KB
Figure 4 - highly_distorted_rectify_image
Figure 4 - highly_distorted_rectify_image724×643 408 KB

All in all, the result is bad.

I’m not sure what reason caused the bad rectification. Did anyone implement similar projects before ? or any suggest resources I can refer to ?

2

The undistorted fisheye image looks incorrect to me. Which distortion model did you use?

3

I directly used cv::fisheye::undistortImage. I supposed it was using the rectilinear mapping function.

4

incorrect or not (in principle), the second “Rectified” picture looks like it suffers from badly estimated parameters. looks like a homography that is a sideways view of the image plane. the second “plane” (“sandwich” view) is a numerical consequence.

look at all the matrices and whatnot. show them to us. give us something debuggable. words won’t help.