Some observations, suggestions:
- 0.03 reprojection error is excellent - how many input images were used to achieve this result? If only a few, you might consider using more (say 8-12) - your score might go down, but the resulting parameters might actually be more accurate / correct. (I’m a bit skeptical of your 0.03 score)
- The foreground of your image looks to be noisy / not very structured, so it seems likely you could be getting matching features between the two frames that aren’t actually from the same scene position. What happens when you filter these matchpoints out from the calculation? (Just for testing purposes, maybe do something like eliminate all points with a Y image coord > 1000 (for example))
- How about trying a RANSAC approach for the findEssentialMatrix
- Since you are applying this to video, you might want to consider using a Kalman filter (this doesn’t really apply to your current situation, which you need to address first, but will probably become relevant once you are computing the pose continually for a sequence of frames.)