Thanks for the suggestions! As a check I used the homography matrix to back-calculate the calibration points, and the error was quite significant.
I made a mistake in the sketch, I am actually taking a photo of a section through the cylinder. This section is perpendicular to the camera. It seems that the best solution for me is to use the simple (analytical) raytracing to find the distorted and undistorted image points for a grid over the section. Then I will fit a polynomial to these points and use the fitted surface to populate cv::remap as you suggested.