When you are testing different configurations, are you calibrating the rig (calling cv::stereoCalibrate), or are you manually setting the parameters based on the physical setup? If you are calibrating it each time, do you get good scores and do the results match what you expect?
I’m surprised that a 1cm Z displacement would be so disruptive and I’m curious why the rectification breaks completely. I wouldn’t expect it to be so brittle, and I would want to understand what is going on and why.