Hi everyone,
I’m working on a university project in collaboration with two companies, where we are developing a gait analysis program to evaluate patients’ walking patterns during rehabilitation in a hospital.
We are using Pose2Sim as the main framework, which internally uses cv2.triangulatePoints to compute 3D coordinates from 2D keypoints.
Our setup consists of 4 cameras, and the goal is to generate a .trc file with accurate 3D keypoints to extract gait parameters.
Here is our pipeline:
- We use rtmlib with CPU as device and OpenVINO for pose estimation.
- Intrinsic calibration works well, and pose estimation results look correct.
- Synchronization also appears to be working.
- However, when performing the triangulation step with
cv2.triangulatePoints, the resulting 3D coordinates are not accurate. Keypoints appear unstable and imprecise in space.
We have tested this with:
- 2 and 4 camera setups
- Extrinsic calibration performed both with scene-based calibration and with checkerboard calibration.
Unfortunately, the same issue persists in all cases: the .trc coordinates remain inaccurate even when applying filtering.
We verified that the input 2D keypoints are correct and well synchronized, so we suspect the problem may be related to:
- Possible errors in extrinsic calibration (improper relative camera transforms?).
- Incorrect assumptions about the projection matrices passed to
cv2.triangulatePoints. - Limitations of linear triangulation when using certain camera angles or baseline distances.
At this point I would like to ask:
- Are there known pitfalls when using
cv2.triangulatePointsfor gait analysis with multi-camera setups? - Could inaccurate extrinsic calibration be the main cause in such cases?
- Are there better practices or recommendations when building projection matrices for this function?
- Should we consider more advanced triangulation methods (nonlinear optimization, bundle adjustment)?
Any suggestions or insights would be greatly appreciated!
Thank you very much in advance.