Hello,
According this article Camera Calibration. I can obtain undistorted images. When setting up the camera structure, there will be some deviation in the camera’s pose. If I need to obtain images captured by the camera that are processed to be undistorted and eliminate the deviation caused by the camera’s pose, how can I achieve that?
Thank you,
Hello Rick,
could you specify what are you referring to by:
Do you mean physical movement of the camera, e.g. while capturing, or are you referring to prespective distortions?
I am also concerned about this topic you may have a look here, about calibration or here. The latter contains example pictures of how to remove perspective distortion in a plane. If this doesn’t help you I will try answering your question in more depth.
(But I am also new here.)
Thank you for your reply. This type of calibration is exactly what I’m aiming for. I understand that there will inevitably be some deviation in the camera’s posture, regardless of how it is set up.
What I need is for the image, after distortion correction, to be further adjusted so that the subject appears perfectly facing the camera. How can I achieve this?
Thank you for your support.
At first I am new to the topic, so don’t take the things I am telling you for undisputable.
As far as I know you dont need to know your camera posture. You just need to (excactly) detect the “points of interest” on the paper and map them to the “destination” points.
In my case I used the squares in the corners:
e.g. getting pixel coordinates of top right corner: (2500, 100)
And knowing this maps to a “real world” point of a DINA4 paper ~(0 cm, 210 cm, 0 cm).
Edit:
Use (0 cm, 210 cm) as point for the “real world”.
Since a homograpy is only defined from 2D to 2D you dont need the last coordiante. I guess setting it to 0 is even false, because in the definition of homogeneous coordiantes this would map to a point at infinity. And you may scale it to pixels in order to get a meaningful image. e.g. instead of 210 cm 2100 mm/pixels. Now every pixel in your final image should be one mm?
(You can arbitrary place your coordiante frame in the “real world”!)
You need at least 4 correspondences like this.
Then you can use cv2.findHomography(image_points, world_points)
If you provide more than 4 points this increases the accuray of your homography.
But you may also use arbitary points. I used this quadrilateral.
e.g. Selecting the 4 points in pixel coordinates and “knowing” they map to the coordinates I printed on the sheet (in cm).
Which language are you using. I could offer sharing my python script.
P.S.:
If you know your camera pose (rotation and translation) and your intrinsics you may calculate your projection matrix from 3d to 2d.
But the other way around isn’t possible. You cant get a 3d model out of a 2d image. (You can in some cases, but thats more complicated, see Hartley & Zisserman page 230. Chapter 8.9)
So you "loose “information” about the depth. So having a photo of a plane you need to provide additional information about the distance.I guess incooperating this distance in your projection matrix reduces it from a not invertible 3x4 to a invertable 3x3 matrix. Camera projection is a “one way ticket”. Invertability means, you are able to get your plane back without projective distortions. (So image → real world points x, y, z, with fixed z coordinate, which is therefore a plane in 3D.) (A homgography doesn’t inculde lens distortion!).
But that only holds for ONE plane, see:
wait, you’re confusing terms now.
while you can remove the lens distortion using the mapxy projections from the calibration, there is NO WAY to correct (or even find !) the pose towards the camera with given data, the latter might not be possible at all !
different, unrelated things, please don’t put those into the same sentence
are you looking for frontalization of human faces ? in that case, keep the idea for later (imo), don’t try that now, while you’re still struggling with most basic things…
again, you cannot use the R/T matrices from calibration, those are only valid for the calibration chessboard images, not for arbitrary shots from your cam !
Now I am getting also a bit confused.
I want to add the following (but I dont know if it matches the topic):
The calibrateCamera function returns the rvecs and tvecs of each board, see here.
These are only relative to each other. So to have a “meaningful” relation one need to avoid moving either the camera or the board. Don’t move both. A great example is here with the visualization of the “Extrinsic Parameter Visualization” Graphic. (There are two if you scroll down further.)
So as far as my understanding goes, but please correct me on that if I am wrong, with a fixed camera and a moving target one is able to calculate the position of the target relative to the camera.
what is that ?
nope. how so ? that’s just ‘wishful thinking’
again, you can only use the ‘intrinsic’ params (cam mat, dist coeffs) from the calibration later. once you put the chessboards away, all ‘extrinsic’ Mat’s are invalid / gone.
Whoops
*fixed I corrected it 
My intention was to point out that the camera is not moved.
I think we are talking the same, but you misunderstood me.
Totally right!
Thats what I meant with moving target. I refered to a checherboard or charuco board or any valid planar or 3d target. If this target with known geometry is visible one may calculate the extrinsics. But only in images where it is visible.
So the extrinsics are, as you also pointed it out, only valid if you neiher move your camera nor move your taget.
But knowing the extrinsics you are still not able to “rectify” it to a plane. You still need a homography.
IF if is really about human heads, please take a look at satya’s solvePnP tutorial . this will give you at least the desired homography. then have a good look at the paper mentioned above on: ‘how to move the pixels’
Thank you all for the enthusiastic discussion. Initially, the calibration based on the camera’s distortion would result in various rvecs (Rotation Vectors) and tvecs (Translation Vectors) corresponding to different camera poses. I was simply wondering if there’s a way to use these rvecs and tvecs to transform the captured images into a rectified (straight) view. If my understanding is incorrect or there are any inconsistencies, please forgive me. Thank you.
so, your recent image looks like you wanted a perspective transform ?
i still don’t think so.
the tvec points to a point in space , and the rvec is the angle from this point towards the camera.
again, it’s treated as a single point, not as an (itself) orientated surface, which might e.g. be tilted against the image plane.
from: * * to: * *
* * * *
so, while you cannot use rvec/tvec for this, you still got access to the detected chessboard corners, and it should be possible to construct the src trapezoid from those !
rvec and tvec, together with an assumed plane, represent a homography.
My intuition tells me the same.
Multiplying your camera intrinsics with the extrinsics yields a 3x4 projection matrix. But you can’t invert this to “rectify” your plane.
In one of my replies I stated, that one needs to know the z coordinate of the plane. Inserting e.g. z=5 into the 3x4 projection matrix you arrive at a invertable 3x3 homography.
But given a general “plane equation” ax+by+cz+d=0
You can see that depending on how I choose x and y the z value changes.
What z to choose? (Maybe the minimum distance, because the extrinsics already contain the camera position and move the plane into the right spot?
Or may you just set z to an arbitrary value?
Depending on whether my approach is even right: What z needs to be inserted?
Or in general: How to arrive at a homography of 3x3 given the intrinsics and extrinsics?
Some equations, could be written on a piece of paper, would be highly appreciated