Get 3D coordinates from 2D pixel


i am stuck with my 3D reconstruction project. I have bought two webcams, set them next to each other and created some images of a chessboard.

A pair of such images looks like this:

I implemented the following for both cameras:

  1. Calculating corners of the chessboards using cv2.findChessboardCorners
  2. Calculating the camera matrices, distortion coefficients, intrinsic/extrinsic matrix using cv2.stereoCalibrate

And now I want to take an arbitrary pixel correspondence of a feature in both images, for example a corner of a chessboard square, and want to calculate the X/Y/Z coordinates in object space. But here I am stuck bc. I am lacking the math.

I suppose I have to calculate two rays that are starting in the camera origins and are penetrating the pixel correspondences. The intersection should be the 3D coordinate I am looking for. Is that right or is there a simpler solution?

you can use block matching to get a disparity image (inverse depth) from a calibrated stereo pair.

from there on, you can either build a simple ortho 3d map like (x,y,disparity(x,y))
or try to retrieve the original frustum using Q from the calibration.

there’s a (much simplified) sample here, :

Thanks for the answer, but, as far as i know, block matching is used if I don’t know correspondences. But I know them so I don’t want to do any block matching, rectification, RANSAC or similar. Just simply calculating 3D coordinates of a known pixel correspondence.

So I think I need r1 and r2 in the following image. If I have them then a simple ray intersection should be sufficient.


yes, except the rays won’t intersect, they’ll just pass very (?) closely, and you probably want the middle of the shortest line connecting them.

that is a little linear algebra. I don’t know enough of that, so there’s probably a smarter solution. a dumb solution…

assume your illustration, and r_1 passes over/under r_2. the shortest line between both will be (roughly) vertical and going between X_1 and X_2, and it’ll be parallel to the normal of both rays, and it’ll be perpendicular to C_1 C_2.

  • calculate the rays r_1, r_2
  • cross product → the normal n to both rays
  • consider the plane spanned by n and r_1. it’ll be vertical, going through C_1 and X_1 (as yet unknown)…
  • calculate the intersection of r_2 with that plane. that’ll be the point X_2
  • same for X_1
  • take the center of both

I’m sure this information can be turned into a system of equations.

Thank you, I will try that out. I feel free to come back to you if I’m getting stuck.