open them both, make sure they overlap perfectly. then switch between them. points at infinity are supposed to not move, be the same. if that isn’t the case, you’ll have a headache getting pixel disparity turned into useful triangulation.
you need to reconsider the entire calibration, intrinsics and extrinsics. browse Knowledge Base – calib.io because calibration does not consist of arbitrarily waving a pattern in front of the cameras.