I am adding additional comments on my post, but not marking it as resolved since I dont think it really is. As most of the posts with simmilar problems as the ones I described in my OP.
I was able to obtain “good” looking rectified images afted selection calibration pictures one by one, for some reason good looking (and good corner-detected) calibration images mess up with the end result big time. Still the calibration is not perfect but I assume I will be able to obtain better results by taking more calibration images and nit-picking them…
I will post my results and findings.
J