SolvePnP or SolveP3P with known translation vector

Hey, I am trying to reconstruct camera pose from N pairs of 3D-2D points.
Camera is calibrated and I have intrinsic matrix + dist coefs.
This works really good with SolvePnP which returns both translation and rotation vectors.
However now I actually know the exact position of the camera in 3D world. What SolvePnP returns is usually close to this but still - I think it just doesn’t make sense to use it.
I could use useExtrinsicGuess and pass initial t_vec but still it would just use this for optimization.
Is there a way to provide translation vector so that only rotation would be calculated? I think it will be much more accurate. It is not possible in “pure” SolvePnP but maybe there are other direct methods?

Thx in advance!