There’s a lot of elements to it. It’s best to understand as much as possible about the nature of the solving process that the calibration is.
Having realized how to properly provide the grid dimensions, I easily get around 200 rms over a camera resolution 4096x2160 after each calibration session of 100 accepted views, a nice start but ChatGPT thinks a 1-2 px can be typically reached by a good enoug calibration.
You can get that reprojection rms, at least in my experiment at the ~200 value range, even if your calibration is very off, if you didn’t provide sufficiently enabling samples in terms of distance and tilt. This is easily confirmed also in practice, by calibrating with only images of a single position holding your board steady at one pose and position.
I think the difficulty of reaching good rms in some settings like a webcam setting, is why “camera calibration” is seldom done without ground truth ranges such as you have in a lens or camera production environment, where you can also throw away the lens unless you manage getting a plausible calibration for it.
I’m happy to learn differently, though I’m getting down to 130 rms without going into almost any image selection logic whatsoever (which is lazy).