Extremely high reprojection error but good undistorted image

Yeah, the “ring” is because the rational model isn’t well behaved when you rely on extrapolation. If you were to plot the distortion function I think you would find a discontinuity near the “wrinkle” in the image. I have included an image that might help make it more clear.

I’d be curious to know what the calibrated focal length was on this calibration run. Any chance it was about 865 pixels?

The problem is that your calibration target doesn’t fill the whole image. If you want accurate / sane distortion correction for the whole image, you will need to collect data over the whole image. Again, the Charuco calibration process makes this much easier since you don’t need to see the full pattern. It’s worth noting that the fisheye calibration apparently handles extrapolation much more gracefully, so you can probably get away with images that don’t have samples near the corner. My experience with the fisheye calibration functions is from quite a long time ago (6 years or more), but I had trouble with the calibration algorithm not being repeatable (sometimes didn’t converge, sometimes worked really well, sometimes gave less-than-good results) and also the supporting functions (undistort, etc) didn’t seem as fully developed / supported. Additionally it was a lot easer to find documentation and forum support for the pinhole methods. I suspect the state of things is better now, so don’t let me talk you out of using the fisheye model if that’s what works for you.