To understand, are you using this function for hand-eye calibration?
It looks a little bit odd for me that SolvePnP() alone would lead to large inaccuracies. For me, it could be something like not enough variations in the robot poses, or maybe bad camera calibration?