If you have the correspondences 3D object points <--> 2D points images (+ the camera intrinsic parameters), you can indeed use cv::solvePnP() to compute the rotation matrix and the translation vector from 4 pairs of correspondences.
I mean no need to have 6 points, the problem is solvable with 4 points.