I poked around in the code and found this as part of the termination test in
CvLevMarq::updateAlt(…) which is called by calibrateCamera():
cvNorm(param, prevParam, CV_RELATIVE_L2) < criteria.epsilon )
So epsilon is the minimum change in the parameter vector relative to the parameter vector of the previous step.
Something like L2_Norm(Param_n - Param_n-1) / L2_Norm(Param_n-1)
So for calibrateCamera, the error is the change in the parameter vector, not reprojection error of the points (which is maybe what you were expecting.) Unfortunately the error measured in this way isn’t very intuitive. I think what you are seeing is that the optimization converges relatively quickly, and the algorithm never actually runs a large number of iterations, hence you see similar computation time (and results) for MAX_ITER = 50 or 500.
That’s my take on what is going on, but you should dig into the code yourself if you want to be sure.