please try to keep the angle to the “expected Z-axis” as low as possible.
the camera rays should meet at an infinite point, not at your “object surface”.
(maybe you need to reduce the baseline(distance between the cameras) and recalibrate)
if there is a known angle, you could try to edit the depth map using it