Are you using the same input images for these two tests, or are you collecting new images for each test? If you are collecting new images, the difference is probably just due to the inherent variation in the input data and the optimization process. A focal length change of that magnitude isn’t very significant, and I wouldn’t worry about it too much.
If you are getting that difference when using the exact same input images, that would suggest some (small) problem with the algorithm. I would expect both object point orderings to give you the same results.
Another possibility is that you aren’t using the CALIB_FIX_ASPECT_RATIO flag when calibrating. This flag forces the Fx and Fy values to be the same (0,0 and 1,1 in the camera matrix). Without it they are free to be optimized independently, and typically they end up with pretty close (but not exactly the same) values. By changing the order of the object points, you are changing the rotation of the camera, which would (I think) cause the Fx and Fy values to swap. If you aren’t passing the CALIB_FIX_ASPECT_RATIO, I would suggest doing that…