How far is the plane of the 2D image from the camera in its axis Z?
Should I do the calculation with fx or fy? And average of the two?
I have try using both and with both of my sensors sizes, yet the result is never as good, down I am attaching a gif comparing an original image with the same one using the focal length obtained with openCV.
You can see that that little difference if using fy to place the image at distance z = fy*sy makes the ray be quite of from the target.
%YAML:1.0 --- K: !!opencv-matrix rows: 3 cols: 3 dt: d data: [ 2.8320285290592147e+02, 0., 3.4733853321829196e+02, 0., 2.8925007159044571e+02, 2.5631437411273401e+02, 0., 0., 1. ] D: !!opencv-matrix rows: 1 cols: 5 dt: d data: [ -6.7901378743141963e-02, 2.4764610649496546e-02, 1.7742694051229406e-03, -2.3292171328171914e-02, -4.0294887313712157e-03 ]