look into how aruco does it. it does a type of solvePnP too but it can assume to be seeing the quad from one specific side of the plane and it knows the order of corners, so there’s no ambiguity around that axis either.
If you have the correspondences 3D object points <--> 2D points images (+ the camera intrinsic parameters), you can indeed use cv::solvePnP() to compute the rotation matrix and the translation vector from 4 pairs of correspondences.
I mean no need to have 6 points, the problem is solvable with 4 points.