Have you calibrated the intrinsics of your camera? If not you won’t be able to recover the camera pose correctly.
If your camera (lens) doesn’t have distortion and you only want to calibrate to a single plane, SolvePNP should be able to generate a solution that maps all of your points correctly, but in that case maybe just use a homography for your calibration.
If you are using points from two different planes you will need to have a calibrated camera (intrinsics) for it to work.
When you say the 4th point is “way off” what do you mean? Can you provide data? Are you certain your world coordinates are correct / in a consistent space when you use 2 planes?
Yeah, an MRE would definitely help here.