Problem with Charuco Calibration

C++
,
1

Hey everyone! I’m currently trying to calibrate my camera with 7 given images (see here) of a Charuco-Board following this example from the documentation. However, the result is so bad, that I think I got something fundamentally wrong here.

result
result1284×510 54 KB

Can someone spot my mistake?

#include <iostream>
#include <vector>
#include <opencv2/calib3d.hpp>
#include <opencv2/highgui.hpp>
#include <opencv2/imgproc.hpp>
#include <opencv2/objdetect/charuco_detector.hpp>

using namespace std;
using namespace cv;

int main() {
    // Define board parameters
    int squaresX = 11;
    int squaresY = 8;
    float squareLength = 15.0f;  // in mm
    float markerLength = 11.0f;  // in mm

    // Create CharucoBoard and detector
    aruco::Dictionary dictionary = aruco::getPredefinedDictionary(aruco::DICT_4X4_50);
    aruco::CharucoBoard board(Size(squaresX, squaresY), squareLength, markerLength, dictionary);
    aruco::CharucoParameters charucoParams;
    aruco::DetectorParameters detectorParams;
    aruco::CharucoDetector detector(board, charucoParams, detectorParams);

    // Vectors to store the detected corner positions and ids
    vector<Mat> allCharucoCorners, allCharucoIds;
    vector<vector<Point2f>> allImagePoints;
    vector<vector<Point3f>> allObjectPoints;
    Size imageSize;

    int successfulImages = 0;

    // Process all calibration images
    for (int i = 1; i <= 7; i++) {
        string filename = "assets/iPhone/cal_0" + to_string(i) + ".png";
        Mat image = imread(filename);
        if (image.empty()) {
            cerr << "Could not read image: " << filename << endl;
            continue;
        }

        successfulImages++;
        imageSize = image.size();

        vector<int> markerIds;
        vector<vector<Point2f>> markerCorners;
        Mat currentCharucoCorners, currentCharucoIds;

        // Detect ChArUco board
        detector.detectBoard(image, currentCharucoCorners, currentCharucoIds);

        if (currentCharucoCorners.total() > 3) {
            vector<Point3f> currentObjectPoints;
            vector<Point2f> currentImagePoints;
            board.matchImagePoints(currentCharucoCorners, currentCharucoIds, currentObjectPoints, currentImagePoints);

            // For debugging
            Mat imageWithCorners = image.clone();
            aruco::drawDetectedCornersCharuco(imageWithCorners, currentCharucoCorners, currentCharucoIds);
            imshow("Detected Corners - Image " + to_string(i), imageWithCorners);
            waitKey(500);  // Display each image for 500ms
            // End debugging

            if (!currentImagePoints.empty() && !currentObjectPoints.empty()) {
                allCharucoCorners.push_back(currentCharucoCorners);
                allCharucoIds.push_back(currentCharucoIds);
                allImagePoints.push_back(currentImagePoints);
                allObjectPoints.push_back(currentObjectPoints);
            }
        }
    }

    cout << "Successfully processed " << successfulImages << " out of 6 images." << endl;

    if (allCharucoCorners.size() < 4) {
        cerr << "Not enough corners for calibration. Need at least 4 valid images." << endl;
        return 0;
    }

    // Calibrate camera
    Mat cameraMatrix, distCoeffs;
    vector<Mat> rvecs, tvecs;
    int calibrationFlags = 0;
    double repError = calibrateCamera(allObjectPoints, allImagePoints, imageSize,
                                      cameraMatrix, distCoeffs, rvecs, tvecs, calibrationFlags);

    // Print calibration results
    cout << "Reprojection error: " << repError << endl;
    cout << "Camera matrix: " << endl << cameraMatrix << endl;
    cout << "Distortion coefficients: " << endl << distCoeffs << endl;

    // Undistort and display an image
    string filename = "assets/iPhone/cal_01.png"; // Using the first image
    Mat image = imread(filename);
    if (image.empty()) {
        cerr << "Could not read image for undistortion: " << filename << endl;
        return -1;
    }

    Mat undistorted;
    undistort(image, undistorted, cameraMatrix, distCoeffs);

    // Display original and undistorted images
    namedWindow("Original", WINDOW_NORMAL);
    namedWindow("Undistorted", WINDOW_NORMAL);
    imshow("Original", image);
    imshow("Undistorted", undistorted);

    cout << "Press any key to close the image windows..." << endl;
    waitKey(0);

    return 0;
}
2

you might want to show at least thumbnails of your seven calibration pictures

1 Like
3

What error value did you get from the calibration? I suspect it was pretty high? (I’d say shoot for a value below 1.0 - if you are getting huge error values it tells you that the calibration didn’t model the data very well, which could lead to results like you are getting.)

The obvious issues I see with your calibration process:

  1. calibration target isn’t very flat. print it again and adhere it to a piece of glass or something else really flat. or at a minimum re-print it and avoid the wrinkles.
  2. you need to move the camera (or the target) in a way that gets more perspective distortion in at least some of the images. Most of your images look to be “straight on” shots - try some with 30 degree rotation between the calibration target and the image plane.
  3. you need more data from close to the corners of the image. Since you are using the Charuco target you don’t need to see the whole thing - position the camera / target so you get points as far into the corners as you can. This will greatly help with getting accurate distortion parameters.
4

Yes, I get crazy high reprojection errors, for the example above it was 57.7803.

I tried again with your advices - reprojection error of 271.532. By now, I’m pretty certain the script contains an error.

5

you might want to show at least thumbnails of your seven calibration pictures

6

Make sure the board you define in the script is the same as the one you are using. For example, it’s fairly common to interchange the X/Y size of the board and then get nonsensical results. a 5x7 physical board and a 7x5 board in the code.

Also I always draw the detected points and back-projected / estimated point locations to the original images. This way you can get a better visual understanding of what is going wrong. Super helpful, highly recommended.

7

Thanks for your tips! I ended up trying with the board and values described on GitHub, just to see if my implementation was correct. It was.

I tried reproducing the images shown in the repository with my own board, double checking if the parameters were correct - it still didn’t work.

I then switched over to a ChArUco board using DICT_6X6_250 instead of DICT_4X4_50. Suprisingly, this made the calibration a lot better. I’m not sure why this makes a difference, but…it works.

8

that’s what I thought. image corners aren’t covered by points of the pattern.