Error in camera pose estimation obtained through Arucos when the camera has orientation

Hi, I’m working on a project to detect some ArUco markers with an Intel RealSense camera D435.

In this project, I’m trying to obtain the perpendicular distance, the hypotenuse, and the orientation of the camera with respect to a point in the middle of the ArUco markers. This is computed by obtaining the Tvec and Rvec of the camera to the ArUco markers, then making the transformation from the ArUco markers to the center point as this point is known from the center of arucos to it, and subsequently obtaining the transformation with matrices from this center point to the camera(I tried using matrices and Euler angles). As I have four values, I calculate the mean. With this approach, everything seems to work fine when the orientation is low; however, when I increase it, the error starts to increase significantly.

Additionally, I noticed that when I provide orientation, if I only use the two closer ArUco markers, it works pretty well, but if I use only the two furthest ArUco markers, that is what produces the error. Is this problem related to calibration, or is it associated with hardware limitations?

Thanks in advance!