Get 3D coordinates from 2D pixel

Python
1

Hi,

i am stuck with my 3D reconstruction project. I have bought two webcams, set them next to each other and created some images of a chessboard.

A pair of such images looks like this:

image
image730×309 39.1 KB

I implemented the following for both cameras:

  1. Calculating corners of the chessboards using cv2.findChessboardCorners
  2. Calculating the camera matrices, distortion coefficients, intrinsic/extrinsic matrix using cv2.stereoCalibrate

And now I want to take an arbitrary pixel correspondence of a feature in both images, for example a corner of a chessboard square, and want to calculate the X/Y/Z coordinates in object space. But here I am stuck bc. I am lacking the math.

I suppose I have to calculate two rays that are starting in the camera origins and are penetrating the pixel correspondences. The intersection should be the 3D coordinate I am looking for. Is that right or is there a simpler solution?

2

you can use block matching to get a disparity image (inverse depth) from a calibrated stereo pair.

from there on, you can either build a simple ortho 3d map like (x,y,disparity(x,y))
or try to retrieve the original frustum using Q from the calibration.

there’s a (much simplified) sample here, :

3

Thanks for the answer, but, as far as i know, block matching is used if I don’t know correspondences. But I know them so I don’t want to do any block matching, rectification, RANSAC or similar. Just simply calculating 3D coordinates of a known pixel correspondence.

So I think I need r1 and r2 in the following image. If I have them then a simple ray intersection should be sufficient.

image

4

yes, except the rays won’t intersect, they’ll just pass very (?) closely, and you probably want the middle of the shortest line connecting them.

that is a little linear algebra. I don’t know enough of that, so there’s probably a smarter solution. a dumb solution…

assume your illustration, and r_1 passes over/under r_2. the shortest line between both will be (roughly) vertical and going between X_1 and X_2, and it’ll be parallel to the normal of both rays, and it’ll be perpendicular to C_1 C_2.

  • calculate the rays r_1, r_2
  • cross product → the normal n to both rays
  • consider the plane spanned by n and r_1. it’ll be vertical, going through C_1 and X_1 (as yet unknown)…
  • calculate the intersection of r_2 with that plane. that’ll be the point X_2
  • same for X_1
  • take the center of both

I’m sure this information can be turned into a system of equations.

5

Thank you, I will try that out. I feel free to come back to you if I’m getting stuck.