How to find the rotation matrix with known points?

The corner points of a rectangle are given in 2D coordinates. I also know the real distance of the points and I have the camera matrix.

Now I want to find the rotation vector with respect to the camera, but without using the cv2.calibrateCamera() method with the chessboard corners.