A few comments:
-
I would suggest increasing your cv2.TERM_CRITERIA_COUNT value to 200 ( or more) and your TERM_CRITERIA_EPS to 0 - just for testing / understanding how they affect things. I have definitely encountered situations where I need a lot of iterations. I currently use 150 in some cases, but that isn’t necessarily a well honed value (more likely it was chosen out of frustration and left at a high level.) I suggest setting the EPS value to 0 to make sure the iterative process isn’t stopping prematurely. Maybe more iterations would help?
-
I last worked with the fisheye model 4+ years ago, so this info might be out of date. At the time I found the calibration process didn’t converge as reliably, and that the support functions for the fisheye model weren’t as robust / comprehensive. I don’t remember the details, but I decided that the standard model was a better bet for me.
-
Have you tried the standard calibration with the rational model? I found that it does a very good job with some lenses with pretty significant distortion.
-
I have wondered (but never tried) if it would be possible to compute an inverse distortion function that you could evaluate directly (instead of the iterative refinement approach). Surely it’s possible, but I don’t know if one would need to write separate functions, or if there is a clever way to get it to work within the OpenCV framework.
In any case, what you are trying to accomplish is definitely possible, so don’t give up! 