Problem with rectifying pinhole and fisheye image

Alfie
Fri, 10 Feb 2023 09:22:35 +0000
Python

I have a stereo camera setup which has one pinhole camera and one fisheye camera. The pinhole camera has 40 degrees FoV. The fisheye camera has 120 degrees FoV. In the end, I want to get a depth map from this camera setup. However, I got a bad rectification result before I implemented stereo matching.

I use python OpenCV to implement the project. There are 3 steps before stereo rectification:

Step 1: Use padding method to make the pinhole image size equal to fisheye image size

Step 2: Calibrate and undistort pinhole and fisheye image individually by using normal camera module

Step 3: Pass early calibration parameters to do stereo calibration

After doing stereo calibration, I also checked that both images were successfully undistorted by intrinsic and extrinsic parameters. You can see the above figures. The figure on the left is the undistorted fisheye image, and the other one is the undistorted pinhole image. Next, I did stereo rectification by following the steps below :

Step 1: Get the rotation and projection matrices by calling cv2.stereoRectify(…)

Step 2: Get maps by calling cv2.initUndistortRectifyMap(…)

Step 3: Using cv2.remap(…) to generate rectified images

After implementing the above steps, the remapped images show black or highly distorted. You can see the figures below :

All in all, the result is bad.

I’m not sure what reason caused the bad rectification. Did anyone implement similar projects before ? or any suggest resources I can refer to ?

4 posts - 3 participants