From the image it doesn’t seem like you are dealing with much radial distortion, but if you haven’t accounted for that it still might be helpful (especially if you want high accuracy).
I suspect the reason for the inaccuracy is because your camera isn’t pointed perfectly orthogonal to the image plane, therefore you are experiencing perspective distortion.
The thickness of the object will also contribute to inaccurate measurements. I would suggest moving the phone around in the image, and if the dimensions change based on image position, that’s (probably) a perspective distortion effect.
Crackwitz mentioned using the Aruco marker pose estimation - if you go this route you can use the recovered pose (along with the camera matrix) to compute width/height estimations of your bounding box. This is probably the better approach because, for example, you can account for object thickness (if you know it) to get correct / more accurate width/height. You might do best by using a number of Aruco markers to estimate the pose for a more accurate and robust process.
I suggested using a homography, which is probably the easier approach, but it only works for objects that are in the same plane as your Aruco marker (so thin things), and you can’t directly extend it to account for depth differences. It might be a faster way to get going, but you might run into a wall pretty quickly if you want to measure things with thickness.
(Also I suggested using the centroids of multiple markers - this was for increased accuracy / robustness. You can compute a homography based on a single marker, but the detection accuracy / noise for the individual corner points might not give you the best / most stable results.)