I am wondering how I can evaluate the quality of camera calibration. The average reprojection error is SOME way, but it depends on the calibration target distance and its placement in the image, so it cannot be used indiscriminately. Suppose I take a handful of shots with the chessboard placed centrally and far away from the camera - the average reprojection error will be small, but the calibration will not be correct.
Is it, then, enough to take a picture of a chessboard filling the entire image (provided it is not too close to change the camera focus) and calculate the reprojection error on this single image?
Or perhaps I could print several small chessboards whose relative positions are precisely known, and the error in distance between these chessboards will be the measure of calibration quality? This distance could be checked at different angles, not only perpendicularly to the camera axis.
How should I do it the right way?
evaluate it by how the input data is created and if it is such that a good calibration is possible. input data can be defective in lots of ways that all but guarantee a bad calibration.
if there’s rolling shutter, the camera must be held very still. there need not be evidence of motion blur for this effect to be noticeable. it’s best detectable from a video feed but I can’t think of a “simple” way to detect it. best to ask/inform/instruct the user.
if the calibration pattern doesn’t cover the corners of the view, you can count on the numerical optimization adding distortion there.
if the picture is overexposed, count on the grid detection giving bad sub-pixel localizations. not because it’s a bad algorithm but because the data is bad.
the cornerSubPix algorithm (that’s used after grid detection) probably assumes a linear color space… and nobody ever thinks of that. and then most webcams sharpen their pictures. what I’m getting at: an edge or corner in the real world is mapped to pixels, but that doesn’t happen linearly for both of these reasons, but it is assumed to happen linearly by the algorithms… and we can’t just “simplify” reality because a linear assumption demonstrably deviates from reality.
if you want to quantify this, mount a camera and a movable pattern. move the pattern by tiny increments, so tiny that you can move an edge/corner by subpixel increments. do a series of pictures for known displacements. measure the edge/corner and compare to the known displacement. take a look at the scanline profile of a single picture.
if the pattern isn’t flat, i.e. deviates from the model (which is flat, I assume), that adds error. I think there are ways to estimate the true shape from the calibration data. it’s basically Structure from Motion. autocalibration uses an unstructured scene so a known pattern isn’t even needed.
You are absolutely right, but let’s assume we have a reasonably flat chessboard with square fields (not so obvious as many printers distort printouts), a stationary camera, good lighting without overexposure, and a stationary or very slowly moving target. And I calibrate this camera several times using a different number of images, positioning the target differently. How can I then choose which of these calibrations is the best?
I can only guess. I’ve never needed to worry too much about this.
I’d have to see at least the results (picture resolution, intrinsics matrix, distortion coefficients) to generate ideas and get an impression of how close these calibrations are numerically.
you should definitely pay attention to the spread of corners across the view.
What I need is an objective measure for novice users. I would imagine it as an average reprojection error for chessboards that fill the entire image as much as possible. So sometimes it can be a single image (for narrow lenses) and sometimes it must be a number of images for wide-angle lenses. I am just thinking aloud. Say, I took 500 images, randomly split it into 5 batches and performed 5 calibrations. Which one is the best?