I’ve seen this “effect” before as well. I think it’s estimating some translation in Z for those cameras, when there isn’t one.
the calibration routines in OpenCV frustrate me too. awful documentation of the whole process, lots of magic, impossible to get a sense for what one did right and what needs improvement…
that is to say, if this frustrates you, it’s not necessarily your fault, and you are not alone.