A project I’ve been working on for the past few months is generating a 3D point cloud using my custom stereo camera.
workflow of my project:
- Calibration of the left and right cameras using a checkerboard pattern, as well as stereo calibration.
- Rectification of the left and right images.
- Generation of a disparity map using the rectified left and right images.
- Creation of a 3D point cloud using the generated disparity map and the OpenCV
reprojectImageTo3Dfunction.
Some problem on the project:
I think my disparity map is pretty good.
The point cloud generated by the reprojectimageto3d function was visualized using the open3d library, and the result is Figure 1, where the point cloud is mostly projected onto the same plane.
However, in reality, the points in the point cloud have varying distances from the camera (refer to the original image in Figure 1).
How can I ensure that the 3D point cloud has correct x, y, z values?
What I tried before:
I understand that the Q matrix for the reprojectimageto3D function should be set as shown below. You made two attempts to set the Q matrix:
> Q matrix = [[1,0,0,-Cx],
> [0,1,0,-Cy],
> [0,0,0,f],
> [0,0,-1/Tx, (Cx-Cx’)/Tx]]
Attempt 1: Using my actual camera matrix and baseline value, I set the Q matrix as follows:
Q = np.float32([[1, 0, 0, -968.8848659],
[0, 1, 0, -545.83119],
[0, 0, 0, 1428.08808],
[0, 0, 1/60, (968.8848659-938.823154)/60]])
With this configuration, I obtained the result shown in Figure 1, where the point cloud is mostly located on the same plane.
Attempt 2: Following the instructions in the provided link, I set the Q matrix as follows:
Q = np.float32([[1, 0, 0, 0],
[0, -1, 0, 0],
[0, 0, focal_length * 0.05, 0],
[0, 0, 0, 1]])
As a result, I obtained the result shown in Figure 2. Although the shape of the point cloud appears correct, there seems to be an issue with the coordinates of the point cloud when viewed in the coordinate system with the origin at (0, 0, 0).
What am I doing wrong?
I would appreciate any guidance or suggestions you can provide.
(my stereo depth estimation code link: https://github.com/dohyeonYoon/pyStereo/blob/main/img2disp.py)
