It's possible to do cv2.stereoCalibrate() with ChArUco-Board?

1

Hello people,

I am currently working on a stereo calibration of two cameras. I am currently using a normal checkerboard. I also get correct results, for example I measure the distance of my cameras and compare them with the translation matrix.

Now I have heard that it is more robust if I use the ChArUco board. I tried to find something on the internet but there are really almost no tutorials. I then tried to program it myself, however I get nonsense results. Is it even possible to do a stereo calibration with a ChArUco board? I have added the main party of my code, maybe I did something wrong and someone can give me a hint.

Thank you.

pair_images = zip(images_l, images_r)

for images_l, images_r in pair_images:
    # Left Image Points
    img_l = cv2.imread(images_l)
   grayL = cv2.cvtColor(img_l, cv2.COLOR_BGR2GRAY)
   corners_L, ids_L, rejected_L = cv2.aruco.detectMarkers(grayL, aruco_dict, 
                                                          parameters=arucoParams)
   resp_L, charuco_corners_L, charucos_ids_L =  
                                 cv2.aruco.interpolateCornersCharuco(corners_L, ids_L, grayL, board)

   # Right Image Points
   img_r = cv2.imread(images_r)
   grayR = cv2.cvtColor(img_r, cv2.COLOR_BGR2GRAY)
   corners_R, ids_R, rejected_R = cv2.aruco.detectMarkers(grayR, aruco_dict, 
                                                         parameters=arucoParams)
    resp_R, charuco_corners_R, charucos_ids_R = 
    cv2.aruco.interpolateCornersCharuco(corners_R, ids_R, grayR, board)


   objpoints_L, imgpoints_L = cv2.aruco.getBoardObjectAndImagePoints(board, charuco_corners_L, charucos_ids_L)
   objpoints_R, imgpoints_R = cv2.aruco.getBoardObjectAndImagePoints(board, charuco_corners_R, charucos_ids_R)

   if resp_L == resp_R and (resp_L and resp_R) > 1:
    corners_list_L.append(charuco_corners_L)
    corners_list_R.append(charuco_corners_R)

    id_list_L.append(charucos_ids_L)
    id_list_R.append(charucos_ids_R)

    objpoints.append(objpoints_L)
    imgpointsR.append(imgpoints_R)
    imgpointsL.append(imgpoints_L)
    # Draw and display the corners
    cv2.aruco.drawDetectedCornersCharuco(img_l, charuco_corners_L, charucos_ids_L, (255,0,0))
    cv2.aruco.drawDetectedCornersCharuco(img_r, charuco_corners_R, charucos_ids_R, (255,0,0))

    cv2.imshow('imgL', img_l)
    cv2.imshow('imgR', img_r)
    cv2.moveWindow("imgR", 800, 0)
    cv2.waitKey(200)
    else:
       print("Chessboard couldn't detected. Image pair: ", images_l, " and ", images_r)
       continue

 cv2.destroyAllWindows()

ret, M1, d1, M2, d2, R, T, E, F = cv2.stereoCalibrate(objpoints, imgpointsL, imgpointsR, mtxL, 
                       distL, mtxR, distR,(640, 480))
print(T)
2

Quick answer without looking at the code.

Yes, it is possible. Camera calibration needs input 2D coordinates from whatever methods. The advantage of using ChArUco board vs classical calibration board should be:

  • with classical calibration board, the board must be fully visible in the image
  • more accurate 2D corners coordinates with ChArUco board (I think, to be checked)
3

Hey Hans,

had the same problem and solved it recently. The problem is coming from the object points given by

whose y-axis are mirrored in case of ChArucos which causes the stereo calibration to fail. You can easily solve the problem by inserting:

newChessboardCorners = board.chessboardCorners
newChessboardCorners[:,1] = (CHARUCOBOARD_COLCOUNT*squareLength) - newChessboardCorners[:,1]
 
objpoints = []
for idx in charucos_ids:
    objpoints.append(newChessboardCorners[idx])
objpoints = np.asarray(objpoints)
 
 imgpoints = charuco_corners
4

Hello Hasn and mt-cv,

I am also facing the same issue in the later stages of the calibration. I have estimated the camera matrices and the distortion coefficients using the funtion:

aruco.calibrateCameraCharuco

Then I am trying to calculate the object and image points inorder to stereo calibrate, by using:

aruco.getBoardObjectAndImagePoints

I assume, after using the above function, I should be getting object points and image points, but I am getting an unexpected error. Could you please help me to resolve it?

**objpoints_L, imgpoints_L = aruco.getBoardObjectAndImagePoints(board, corners_all, ids_all)**
**cv2.error: OpenCV(4.5.4) :-1: error: (-5:Bad argument) in function 'getBoardObjectAndImagePoints'**
**> Overload resolution failed:**
**>  - detectedIds is not a numpy array, neither a scalar**
**>  - Expected Ptr<cv::UMat> for argument 'detectedIds'**
5

Hi, I am facing a similar problem but even after inserting your code, i am still getting nonsense results. Can you please share your code. How did you manage to do stereo calibration using a charuco board? Thanks!

6

I finally got this working, but there seems to be numerous bugs with several different versions of OpenCV. I’m using libopencv 4.5.4 on Ubuntu 22.04. The main issue with that version is that the functions to get the chessboard corners and even size are wrong.

I defined this function to get the ObjectPoints:

        static cv::Point3f GetObjectPoint(const cv::Ptr<cv::aruco::CharucoBoard> &board, int id)
        {
            // The getChessboardSize seems to return swapped dimension based on how the markers are actually arrange
            // There is also one less charuco corner than original width;
            auto height = board->getChessboardSize().width - 1;
            auto width = board->getChessboardSize().height;

            if (id > (height * width) - 1)
                throw std::runtime_error("Invalid marker id for defined board");

            auto column = id / height;
            auto row = id % height;

            cv::Point3f point;

            point.x = column * board->getSquareLength();
            point.y = row * board->getSquareLength();
            point.z = 0;

            return point;
        }

Then you basically need to create a vector of vectors for object points, rgb points, and depth points. The inner vector is a vector of points for a given frame, and the outer vector is for the number of frames. Use the interpolatecharucocorners to find the corners of the chessboard squares in both camera frames and the function I provided above to get the object points.

I had to get rid of the flag to fix the intrinsics. With the flag, my reprojection error was 50 and after removing it, it went down to .57.

Repo of working code (with 4.5.4)

7

There seems it mean nothing. A bug is something that any body can reproduce.
Do you have an example (full code) and data (set of images )?

if intrincis parameters are not good then yes you have to rid of intrinsics parameters flags

8

Namely this:

As I mentioned above, there are even more issues such as it giving the wrong number of chessboard corners for a charuco board in seemingly the wrong dimensions too.

As you know, this is a contrib module and it is heavily updated. There’s no shortage of reproducible bugs. I have tried using a version of the contrib after the fixes for this specific issue were put in, but there seems to be even more bugs as the pixel reprojection for my intrinsic calibrations went up from subpixel to 50 with no changes to the code. I have not had time to diagnose this yet, but I wanted to post a definitive answer to this question (yes, you can do stereo cal with charuco), but no matter which recent version of opencv you use, you are very likely to encounter bugs with the charuco board points especially.

9

yes, charuco got broken recently (v4.6/v4.7). there are patches that will make old boards work again. for now, you have to make new boards or use v4.5.5

feel free to add your vote or voice to the various issues people have opened about this breakage.