How does solvePnP deal with distortion

Hey guys,

I’ve a camera with obvious distortion. I calibrated this camera to receive the camera intrinsic matrix and the distortion parameters.
When you undistort your camera you’ve two options:

  1. Undistort your image and crop the image to keep your original camera matrix.
  2. Undistort your image and keep all/part of your image information, but you’ll receive a new camera matrix.

With this I can estimate a pose of an observed & known object with:
cv2.solvePnP(objectPoints, imagePoints, camMtx, dist, None, None, False)
Where camMtx is my camera intrinsic matrix and dist is my distortion vector, and the imagePoints are points of my original distorted frame. I’m getting nice results with this.

But now, I’ve a question about the handling of distortion in the solvePnP algorithm. By giving the solvePnP the distortion and (original) camera matrix, will it use method 2 of undistorting the image, to get a position for your object? If yes, will it make a new camera matrix? I couldn’t find it in the source code. Is it better to undistort your frame yourself, and giving solvePnP an empty vector for distortion?

I mean, it is working all fine when giving the solvePnP algorithm the distortion parameters and known imagepoint out of the distorted frame. But I also would like to know the steps of how this algorithm is handling this.

I hope my question is clear. Thanks!

I’m not sure if I understand your question, but you can provide solve PNP with distorted image coordinates / world coordinates and your calibrated camera matrix and distortion coefficients and it will handle the distortion correctly. It won’t undistort the image (you don’t provide an image to solvePNP), but it will account for the distortion in the points you give it (it undistorts the points you provide it, to get image point locations you would expect for your camMat if the lens had zero distortion)

You can check it by calling projectPoints with your camMat, rvec, tvec, and world points. If you provide your distCoeffs you will get distorted image locations, if you don’t provide distCoeffs you will get undistorted image locations.

You can also call cv::undistortPoints yourself and pass those points into solvePnP (and with no distortion coefficeints). Note that if you have severe distortion, you want to set up your cv::termCriteria to have maxCount set high enough that you get a good estimate of the undistorted location. UndistortPoints is an iterative algorithm and, at least historically, the default value for number of iterations was woefully inadequate to get good results - now you can pass in a termCriteria argument, which is great.

1 Like

Ah thanks, cool, didn’t know about the undistortPoints function. Still, I’m a little confused about when to use the original cameraMatrix and when to use a optimalNewCameraMatrix. Cause here someone says you need to use the optimalNewCameraMatrix for the undistortPoints function. While, in my view, you can just use the original camera matrix for undistortPoints?
Please correct me if I’m wrong; the optimalNewCameraMatrix is only needed when you are undistorting the frame and don’t crop the image?

The purpose of optimalNewCameraMatrix is to give you control over how your undistorted image gets mapped. If your distortion is minimal, you can just undistort your image with the original camera matrix - you might lose a few pixels or have a small area of black pixels, but for many applications that’s OK. With more significant distortion the mapping/undistortion process can result in significant loss of image data (outside of the image boundaries) or large areas where no pixels map to (which get rendered as black pixels). When you need/want to control how your image gets undistorted, you have to modify the camera matrix (newOptimalNewCameraMatrix, or compute your own, etc) - if you want to project points to that new undistorted image, you have to use the new camera matrix to get the right results.

So yes, you can use the original camera matrix for undistortPoints, but that will give you results those points will only be valid in an image that was undistorted using the original camera matrix. If you undistort your image using a newOptimalCameraMatrix (or similar), you will need to use that if you want to undistort points that make sense in your undistorted image.