getOptimalNewCameraMatrix aspect ratio distortion

Python
1

Hi,

I’m trying to visualize the undistorted images after performing a basic camera calibration using the cv.undistort() and cv.getOptimalNewCameraMatrix() functions. The problem is that I’m not sure how to retain all the original pixels without distorting the image.

I first assumed that in order to do so, the only thing I needed was to use an alpha=1 when calling the function getOptimalNewCameraMatrix(). But the image looks like squeezed and the region inside the ROI looks different compared to the image provided using an alpha=0. I guess it is because it is trying to fit the new undistorted image to the original resolution of the image which has a different aspect ratio?

Is there a way to undistort the image, don’t lose any pixels in the process but also don’t stretch out the result because of a change in the aspect ratio?

Here are some of the different configurations I tried using the parameters alpha, croping and centerPrincipalPoint:

Calibration matrix:
[[10328.62 0. 719.12]
[ 0. 10131.19 903.98]
[ 0. 0. 1. ]]

Distortion coefficients:
[[ 20.99 -424.72 0.84 -0.39 9315.43]]

combined images
combined images1920×1086 91 KB

  • Undistorted image without using a new calibration matrix:
  • Undistorted image using a alpha=0
    new matrix[
    [11534.85 0. 685.07]
    [ 0. 11104.3 1002.86]
    [ 0. 0. 1. ]]
  • Undistorted image using alpha=1 (cropped and no cropped):
    new matrix:
    [[10926.47 0. 680.3 ]
    [ 0. 9342.57 924.02]
    [ 0. 0. 1. ]]

I also tried using the parameter centerPrincipalPoint=True, which gives newCalibration matrix with fx/fy that is the same for both the new and original matrix. (There I expect that there is no distortion here), but gives a strange output for alpha=0

  • Undistorted image using alpha=0 and centerPrincipalPoint=True
    new matrix:
    [[80212.26 0. 959.5 ]
    [ 0. 78679.02 539.5 ]
    [ 0. 0. 1. ]]

  • Undistorted image using alpha=1 and centerPrincipalPoint=True with and without cropping.
    new matrix:
    [[5561.07 0. 959.5 ]
    [ 0. 5454.77 539.5 ]
    [ 0. 0. 1. ]]

I know the distortion that the distortion coefficient and the original matrix that OpenCV found during the calibration are wrong, but I still would like to understand how can I get the undistorted image.

I hope I have explained myself.
Do you have any suggestions to achieve this? I would like to obtain an undistorted image that does not lose any of the original pixels and it is not squeezed because of the original resolution of the image.

Thanks,

2

A few suggestions / comments.

  1. Your calibration target appears to be paper taped to something. I notice wrinkles in the corner and I suspect it isn’t flat overall. I have done testing with simulated/synthetic data (calibration images generated with OpenGL with different types of errors added to the calibration target) and I concluded that calibration target flatness is pretty important for getting good results. I would put some effort into getting your calibration target flat - taping it to a glass or otherwise “guaranteed flat” substrate is a good start.
  2. How sure are you the pattern was printed without any scaling done to it? Is it possible that it is scaled non-uniformly somehow?
  3. I noticed that your FX and FY parameters are different in your camera matrix. For most modern cameras the pixels are square, and you should expect FX and FY to be very close to each other. In fact you usually will pass in the CALIB_FIX_ASPECT_RATIO to force them to be the same. In one case I calculated a 2% difference. It’s not huge, but it makes me wonder if something else about the calibration setup isn’t right.
  4. What is the calibration score (reprojection error) when you calibrate? You said “I know the distortion coefficients and camera matrix…are wrong” Can you tell us more about why you think they are wrong? I’m concerned that you might be working with calibration results with high reprojection error - 5, 10 pixels RMS? Ideally you want as small as you can get, of course…for your setup maybe < 2.0 is a good target.
  5. I notice that your focal lengths seem very long. 10,000 pixels in your first example. Obviously there are a lot of things I don’t know about your setup, but for the types of cameras I tend to work with (3mp, 1/3" format, S-mount lenses) your focal length would be on the order of 30mm, which is very long. I mention this because when I see parameters that are out of the expected range, it tends to mean that the calibration process wasn’t done very well. If the images you show of hte full chessboard pattern are what you put into the calibration process, you need more images and a wider variety of scale and perspective. Tilt the board toward the camera in some images - most of yours are flat, which often results in incorrect focal length estimates.

As for your question about getting an undistorted image which uses all of the pixels, but doesn’t get squeezed…I think I know what you are asking for and I have done it. I will look at my code tomorrow and see if I can find where I did it, but as I recall I wasn’t able to do it with getNewOptimalCameraMatrix and had to instead compute a new matrix myself in order to undistort the image (all the pixels) to a higher resolution image. I also wonder if you are only wanting to undistort the lens distortion, or if you might also want to account for the perspective distortion so that your calibration pattern appears square / uniform scale in the final image.

It also might help to know what your end goal is. Are you only trying to calibrate a camera and undistort the images, or is there a specific feature / process / real-world problem you are trying develop?

Good luck.

2 Likes
3

First of all, thank you so much for your detailed response.

I will answer the following questions below:

  1. Yes, I’m aware of this, in fact I just purchased new calibration targets to make sure they are as flat as possible for future calibrations. Thanks for pointing it out.

  2. The printing was done from a pdf file with the correct dimensions. And I checked the result, the square size seems correct with no scaling involved.

  3. About the calibration setup, I’m trying to calibrate an intel realsense D415 camera. The camera is placed in a fixed position and the pattern is displaced on z. I make several captures of the pattern at different distances. And for each distance, I try to cover the whole field of view by moving the orientation of the camera on both axes. Right now, my only concerns are the flatness of the pattern (which is going to be resolved soon), that none of the captures have strong patter inclination (I have read that it helps to have clearly defined epipolar lines), and that the maximum distance between the camera and the pattern capture is no more than 4m when the camera is expected to make captures at longer ranges (I also read that is important to calibrate the camera using distance similar to the expected working ranges). But otherwise, the calibration setup seems good.

  4. The reprojection error of the calibration is only 1.37. But I know that the calibration results are wrong because the returned fx is 10328.62 compared to the original factory value of 1296.69. Also, the image is clearly more distorted after the calibration than before. But anyways, the main goal of my experiments right now is not to obtain a good calibration (that is going to be my long-term goal) but to know how can I obtain the undistorted images with the different settings.

  5. Yes, focal lengths are wrongly estimated. I have captured 30 images on different ranges, covering the full depth of field. But the tilt is only done by moving the camera orientation, so it is probably not enough tilt. I will attach below the whole set of images I used.

I would be really grateful if you can tell me the procedure to obtain the undistorted images without them getting squeezed. Right now, one of the provisional solutions I found is to manually obtain the new calibration matrix by scaling the fx and fy factors but then I do not have the cropping ROI which is nice to have. Right now, I only want to undistort the lens distortion so that I can then draw the detected calibration pattern lines and see the improvement of the distortion calibration and the reprojected error over the images.

Calibration images: calib images - Google Drive

4

perspective foreshortening must be witnessed in the pictures.
that means any time the board faces the camera, that’s a useless picture.

if you can, please unpack that rar and put the individual pictures on gdrive. gdrive can offer image previews but not from inside of archives.

5

Thanks for the feedback! I will now reupload the folder without the zip file and edit the link so it is easier to view the captures.

About your claim:

Are you sure about this? Because in the intel calibration manual, they use the opencv calibration module and they advice to take some flat pictures as well (see page 23).

https://www.mouser.com/pdfdocs/Custom_Calibration.pdf

6

I don’t know what they’re doing there. I’m not seeing them mention some important things in/around that section.

I can tell you that a frontal view makes estimations of the grid’s orientation very noisy. maybe they’re using math that avoids calculating the pattern’s pose, or maybe they’re just not mentioning these issues.

perhaps they’re not even trying to estimate lens distortion, or focal lengths? or “nudge” those a little bit? who knows…

7

I’m not sure if straight on / flat pictures are useless, but I am sure that if you don’t provide some with tilt, your focal length calculation will not be accurate. When I simulated the calibration process (for my lens type, working distance, etc.) I found that I needed a minimum of 15 degrees of rotation in the calibration target to get good results.

For some intuition about why the calibration tilt / perspective is important, consider two scenarios:

  1. image is taken with the calibration target close to a camera with a wide FOV (short focal length) lens.
  2. image is taken with the calibration target far from the camera with a narrow FOV (long focal length) lens.

Without any perspective distortion / depth change in the target, the images look the same, so it’s not possible to determine the focal length. The algorithm effectively has to pick some focal length and compute the translation / distances of the targets using that bogus focal length. When the target is tilted and there is perspective distortion among your images, the algorithm is able to determine a focal length that fits all of the images.

The do show taking flat pictures, and those might be useful in estimating the distortion coefficients, but they also clearly show some off angle poses (#3 and #4 in their diagram, and based on the description #5 and #6 have tilt too).

If for some reason you have to choose one over the other (all flat or all angles) I would choose all angled views.

I’m not sure what results you can expect if you angled the board to the camera, and only moved it further/closer to the camera (retaining the same angle for all images). I do know that if you angle the calibration target, and rotate around the camera’s Z axis (approximately) you will get very good results.

8

I’m noticing a structured / grid pattern in your calibration images when I zoom in. It reminds me of the type of pattern you can get when you have a raw bayer image and either don’t apply a bayer demosaicing algorithm, or if you use the wrong pattern when you demosaic the image. That’s just a guess, of course, and it’s probably not causing you too much trouble but it is something worth addresssing.

As for how to undistort the image to a larger output image (to use all of the source pixels without downscaling), I don’t use undistortImage with getOptimalNewCameraMatrix. It’s possible you can use these together to achieve what you want, but I had to do it differently because I was also applying a perspective distortion at the same time.

What I did:

  1. Adjust my camera matrix to a larger output image. this involves changing the image center so it is in the correct place proportionally, and then scaling the focal length*

  2. Use initUndistortRectifyMap and remap to undistort your original image to the larger output image. You might be able to just call undistortImage, but I needed to do it manually in two steps because I was applying a perspective distortion, too.

*Unfortunately I don’t have my original code that did this, and what I am referencing is a little different, so I’m going based on memory / understanding here. I think you will want to adjust the focal length (for newCameraMat that you pass to initUndistortRectifyMap) to be a little bit shorter than would be implied by direct scaling. For example, if you create an image that is 50% larger in width/height, I think you will want your focal length to be scaled to something that is less than 50% longer for the newCameraMat. This will result in you using more of the pixels (hopefully all of them) from your input image when creating the (larger resolution) output image.

I hope that make sense.

Here is a comment I have in my code that relates to scaling the camera matrix for a different size image:

    // Scaling the camera matrix when using pixel coordinates requires some strange 1/2 pixel shifting.
    // See link below for more information (Hammer's answer)
    // https://dsp.stackexchange.com/questions/6055/how-does-resizing-an-image-affect-the-intrinsic-camera-matrix

I hope this helps.

10

Thanks a lot, guys! Your answers are really helpful.

About the simulation you did:

Is there any procedure where I could get the requirements of my own setup? did you use a specific software or procedure?

11

Edit: I’m updating this post because the results are likely out-of-date.
My testing / simulation was done with an older version of OpenCV, so some of my findings (particularly the tight tolerance on the target planarity) are not valid.

See Eduardo’s post addressing the question on calibration method here:
https://forum.opencv.org/t/cameracalibration-algorithms-in-calibratecamera/10345

Note that “newer versions” of OpenCV use this method https://elib.dlr.de/71888/1/strobl_2011iccv.pdf
which specifically addresses the issue of inaccurate calibration target geometry (including out-of plane).

So, if you are using a newer version of OpenCV, you can probably do just fine with lower quality calibration targets!

Yay for science, research, and the people who continue to advance the state of the art!

I wrote an OpenGL program that rendered some basic geometry texture-mapped with a calibration target image (Charuco baord). Since this was synthetic data, I had “perfect” knowledge of how it was rendered. I used a camera matrix that was close in FOV to the lens I wanted to use and generated a number of images of the calibration target and then ran the OpenCV camera calibration procedure on those images.

The purpose of this was for me to get some design guidelines for an automated calibration rig (put a camera in, press go, get calibration results.) I wanted to know how target flatness, scale, point location accuracy etc. affected my results. I also wanted to know what type of motion (rotation translation of the target) I needed, and also how many images I needed to capture.

For a baseline test I rendered 10 or so images of a “perfect” calibration target (totally flat, no scaling or other perturbations) form a variety of reasonable positions/orientations. As expected I got very good results (some reprojection error, but effectively zero).

I then tried a number of different variations to see what worked and what didn’t. This is from memory, but my take away was:

  1. Flatness matters more than I would have expected. I think I ended up with a tolerance target of +/- 0.020" (~0.5mm) across a target that is probably 24" x 12" in size. This wasn’t hard to achieve, but did cost a bit.
  2. Uniform scale matters in terms of the results I get, but for my specific application it wasn’t an important factor.
  3. Point-to-point accuracy is important for generating low scores, but (not surprisingly, I guess) if the perturbations follow a normal distribution, the actual end result accuracy is minimally affected.
  4. A single flat calibration target tilted toward the camera and rotated on an axis “parallel” to the camera axis, is sufficient for high-quality calibration. (No translation is needed)
  5. After about 7 images the calibration doesn’t change much. I think I ended up using 12-14 images, but I don’t think there is much justification to go beyond 10 for my application.
  6. A tilt angle of at least 15 degrees.

I used these findings to build a calibration rig, and use it on a range of lenses ranging from about 1.5mm focal length (and a lot of distortion) to about 6mm focal length, and for focus distances of about 5" to 12".

The flatness tolerance is pretty tight, but I really need high accuracy. Also as your calibration target size (and z-distance to the camera) grows, that tolerance grows too. As for the 15 degree tilt angle, somewhat more is probably better, and somewhat smaller angles worked, but (if memory serves) the results were less stable. In my case I was balancing between depth of field / focus issues (too much tilt / z-change caused things to be out of focus) and enough tilt to get reliably accurate results. Your needs are certainly different than mine, so you’ll have to experiment to find what works for you.

2 Likes
12

Thank you! That is really valuable information! I’m sure it will help me a lot in creating my own calibration setup.