Problem with rectifying pinhole and fisheye image

I have a stereo camera setup which has one pinhole camera and one fisheye camera. The pinhole camera has 40 degrees FoV. The fisheye camera has 120 degrees FoV. In the end, I want to get a depth map from this camera setup. However, I got a bad rectification result before I implemented stereo matching.

I use python OpenCV to implement the project. There are 3 steps before stereo rectification:

  • Step 1: Use padding method to make the pinhole image size equal to fisheye image size
  • Step 2: Calibrate and undistort pinhole and fisheye image individually by using normal camera module
  • Step 3: Pass early calibration parameters to do stereo calibration

After doing stereo calibration, I also checked that both images were successfully undistorted by intrinsic and extrinsic parameters. You can see the above figures. The figure on the left is the undistorted fisheye image, and the other one is the undistorted pinhole image. Next, I did stereo rectification by following the steps below :

  • Step 1: Get the rotation and projection matrices by calling cv2.stereoRectify(…)
  • Step 2: Get maps by calling cv2.initUndistortRectifyMap(…)
  • Step 3: Using cv2.remap(…) to generate rectified images

After implementing the above steps, the remapped images show black or highly distorted. You can see the figures below :

All in all, the result is bad.

I’m not sure what reason caused the bad rectification. Did anyone implement similar projects before ? or any suggest resources I can refer to ?

The undistorted fisheye image looks incorrect to me. Which distortion model did you use?

I directly used cv::fisheye::undistortImage. I supposed it was using the rectilinear mapping function.

incorrect or not (in principle), the second “Rectified” picture looks like it suffers from badly estimated parameters. looks like a homography that is a sideways view of the image plane. the second “plane” (“sandwich” view) is a numerical consequence.

look at all the matrices and whatnot. show them to us. give us something debuggable. words won’t help.