SolvePnP wrong x, y coordinates

Python
1

Hello!

I’m using the solvePnP function to get a face 3D coordinates.

The camera is properly calibrated since I plotted the x, y and z axis on the chessboard image after the calibration (with the same chessboard image) and the axis are orthonormal.

But when using the function:

(success, rotation_vector, translation_vector) = cv2.solvePnP(
self.model_points,
image_points,
camera_matrix,
dist_coeffs,
flags=cv2.cv2.SOLVEPNP_ITERATIVE,
)

The translation vector gives the face coordinates (the nose endpoint is the origin coordinate so the translation vector should give the 3d pose of the nose endpoint). The ‘z’ coordinate is correct, but the x and y don’t change when i move the face around the image. In fact, they increase / decrease as i move away/closer to the camera.

Anyone knows why might this happen?

Thanks!

PD: The face coords i’m using are:

self.model_points = np.array(
[
(0.0, 0.0, 0.0), # Nose tip
(0.0, -330.0, -65.0), # Chin
(-225.0, 170.0, -135.0), # Left eye left corner
(225.0, 170.0, -135.0), # Right eye right corner
(-150.0, -150.0, -125.0), # Left Mouth corner
(150.0, -150.0, -125.0), # Right mouth corner
]
)

2

where do you get the 2d image_points from ?

3

please present values for all arguments, including camera matrix and a set of image points.

4

Hi @crackwitz @berak Thanks for the answers.

The 2d image_points come from the library face_recognition → face_recognition.face_landmarks(face_image)

https://face-recognition.readthedocs.io/en/latest/face_recognition.html#face_recognition.api.face_landmarks

image_points = np.array(
[
face[“nose_bridge”][3], # Nose tip
face[“chin”][8], # Chin
face[“left_eye”][0], # Left eye left corner
face[“right_eye”][3], # Right eye right corner
face[“top_lip”][0], # Left Mouth corner
face[“top_lip”][6], # Right mouth corner
],
dtype=“double”,
)

Where face is the result from the previous function.

The camera matrix comes from the calibration I make with a chessboard and multiple images.

    # termination criteria
    criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001)  # Type, max_count, epsilon

    # prepare object points, like (0,0,0), (1,0,0), (2,0,0) ....,(6,5,0)
    objp = np.zeros((chess_width * chess_height, 3), np.float32)
    objp[:, :2] = np.mgrid[0:chess_width, 0:chess_height].T.reshape(-1, 2)

    objp = objp * square_size  # Square size from the settings.py file

    print("Objp: ", objp)

    # Arrays to store object points and image points from all the images.
    objpoints = []  # 3d point in real world space
    imgpoints = []  # 2d points in image plane.

    images = glob.glob(calibration_images_path)
    print(images)
    gray = cv2.cvtColor(cv2.imread(images[0]), cv2.COLOR_BGR2GRAY)

    for frame in images:
        img = cv2.imread(frame)
        gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
        # Find the chess board corners
        ret, corners = cv2.findChessboardCorners(
            gray, (chess_width, chess_height), None
        )

        # If found, add object points, image points (after refining them)
        if ret:
            objpoints.append(objp)

            corners2 = cv2.cornerSubPix(gray, corners, (11, 11), (-1, -1), criteria)
            imgpoints.append(corners2)

            # Draw and display the corners
            img = cv2.drawChessboardCorners(
                img, (chess_width, chess_height), corners2, ret
            )
            cv2.imshow("img", img)
            cv2.waitKey(500)

    ret, mtx, dist, rvecs, tvecs = cv2.calibrateCamera(
        objpoints, imgpoints, gray.shape[::-1], None, None
    )
    print("Translation vectors of chessboards: ", tvecs)

    cv2.destroyAllWindows()