I’m using this stereo camera and I have successfully calibrated it. I have implemented a Visual Odemetery pipeline that does the following:
- Extract features using ORB and match features using BFMatcher
- Perform triangulation on matches and use the resulting 3D points as the reference origin
- Track the 2D image points that correspond to the original 3D points via Optical Flow
- Run solvePnPRansac to retrieve tvec and rvec
- Repeat from step 3
I’m trying to implement a headpose algorithm for an AR/VR headset. So the camera needs to know its position relative to the world.
Here is the full Python script that I’m using:
The tvec is relatively stable when I hold the camera completely still but the reprojection error is elevated:
[ 29.6499381 12.99169597 -84.80990967]
[ 30.53411596 12.69894207 -84.61843795]
[ 30.21410405 13.09544394 -83.86818635]
[ 30.00696091 12.32848116 -84.03174404]
[ 30.28667257 12.51779398 -84.10514305]
Reprojection Error: 9.095787
But when I physically start to rotate the camera ever so slightly then the tvec jumps to a wildly different position and the reprojection error goes way up:
[ 43.39385913 8.33319204 -157.35840469]
[ 41.78548074 7.98671144 -155.7754949 ]
[ 41.40808028 7.86098696 -156.53136813]
[ 41.16710819 6.35818726 -158.28700669]
[ 38.65985097 6.12782463 -157.06420584]
Reprojection Error: 26.872517
The expected behavior is that if I rotate my camera the tvec should report the same position, not jump to a different position.
Note that even if I use the inverse to get the world position of the camera I still see the massive jump in translation when rotating on the camera.
Is there an issue with how I’m using optical flow to track the 2D image points across frames? Do I have a general misunderstanding of how solvePnP works?
As mentioned above, the link to my VO pipeline is posted here. Please have a look and let me know if there is a blatant issue that would cause this.