So I made this, a piece whith well known 3D points and smaller markers :
I could not notice any improvement. It seems like you cannot brute force solvePnP because there are too much best matches.
So I guess I will go on SoftPosit or find a way to get my correspondence as I did before, but with a more robust and and generic way.