Trying to calibrate Allsky camera using fisheye calibration (fails)

I am trying to calibrate a fisheye camera with 180° fov. I am trying to use fisheye calibration but it fails with this flag cv2.fisheye.CALIB_RECOMPUTE_EXTRINSIC.

How do I get around this? I tried without the flag and the resulting undistortion looks bad too.

.

I tried working this in camera calibration which works but it could be a lot better.

crosspost: