I have a stereo camera setup which has one pinhole camera and one fisheye camera. The pinhole camera has 40 degrees FoV. The fisheye camera has 120 degrees FoV. In the end, I want to get a depth map from this camera setup. However, I got a bad rectification result before I implemented stereo matching.
I use python OpenCV to implement the project. There are 3 steps before stereo rectification:
- Step 1: Use padding method to make the pinhole image size equal to fisheye image size
- Step 2: Calibrate and undistort pinhole and fisheye image individually by using normal camera module
- Step 3: Pass early calibration parameters to do stereo calibration
After doing stereo calibration, I also checked that both images were successfully undistorted by intrinsic and extrinsic parameters. You can see the above figures. The figure on the left is the undistorted fisheye image, and the other one is the undistorted pinhole image. Next, I did stereo rectification by following the steps below :
- Step 1: Get the rotation and projection matrices by calling
- Step 2: Get maps by calling
- Step 3: Using
cv2.remap(…)to generate rectified images
After implementing the above steps, the remapped images show black or highly distorted. You can see the figures below :
All in all, the result is bad.
I’m not sure what reason caused the bad rectification. Did anyone implement similar projects before ? or any suggest resources I can refer to ?