So for pnp, the problem is trying to solve for your pose or position in your environment. You need to have the parameters for your specific camera, and then as input the 2D image coordinates as well as the 3D coordinates that go with the 2D ones. What I may be confusing myself on is I see other forum posts speaking of using pnp to get your 2D-3D correspondences, but shouldn’t it be the other way around? You’re using pnp which should mean you already have the 2D-3D correspondences as input right?
solvePnP needs indeed to be given the 2D and 3D points in the right order, so yes with the correct correspondence. The 2D points are the points on your image. The 3D points are the points in your object in the coordinate basis of your choice.
And for getting these 2D-3D correspondences, is this normally done manually with you measuring the 3D object physically, or is there some other method that I just don’t know about.
Putting the right correspondence by hand in your program is easy since you know which point on the image corresponds to which of your 3D points, but the correspondence problem is complex to automate by your computer.
I have been trying to brute force solvePnP to get my object pose without know correspondence, and I had mitigated results, I am going to make another post about this.
Another method which I did not test is SoftPosit, but I have no idea if it works well or not.
Another thing I would like to know is, what are the limitations of using PnP. I’ve read papers about a number of other pnp methods out there, but I have not seen anyone state what the limitations of it are. Whether its that your points lie on a plane, how far apart these points can be, or what distances can pnp used for(such as using it indoors/outdoors).
SolvePnP is pretty good in my opinion. The limitations are that you need at least 4 points (there is a method which uses 3 points but I did not try it yet, feedbacks are appreciated), and your points disposition should not have any symmetry otherwise it might get confused. The distance does not matter, solvePnP does not extract the points from your image itself, it only processes them to return a pose. It is up to you to provide it the right points with the right correspondence. If you think your points extraction is inaccurate, you can use SolvePnPRansac.
For more information, here is a documentation : OpenCV: Camera Calibration and 3D Reconstruction