Thanks @crackwitz for your reply. I understand that zoom causes lots of problems for calibration. Currently the approximation go-to method for video with zoom involves accounting for zoom with focal distance but keeping the position of the camera fixed.
I went through OpenCV’s clibratecamera documentation many times and cannot find HOW to fix parameters like tvecs. Could you point me in the right direction?