Thanks for your really complete message.
I’m trying to make something as generic as possible so I do not wish to track features on a particular object. I did make a few tests with Orb, Sift and Surf, but I feel like it can’t be reliable on a 3D object, and it is also very costly in terms of computation time.
I could indeed try various pose estimation tries and evaluate the reprojection in the first place to have an idea of the pose and thus have a better knowledge of the positions of my points on the next frame, this is an interesting idea, and it might also help me get rid of the orange dots.
I also did not try to tweak the SolvePnPRansac parameters yet.
Well this was a one day piece of work, I am not particularily familiar with image processing but I’m willing to learn. Thanks again for the post. 