My first guess would be buffering and/or scheduling.
If you write a simple program that does nothing other than read the frames and discard them, do you end up with a long-term average of 16.66 msec per frame? In other words, are you truly getting a higher FPS, or just that some frames arrive earlier than you expect, but others arrive later than you expect?
One trick is to point the camera at a timer (I use the stop watch on my phone) and grab a sequence of images into memory, then save them to file and inspect. (Don’t save as you capture because of the computational and I/O load that could affect timings). Are the images being caputred at the 16.66 msec interval as expected, or do you see jitter in the images that correlates to the jitter in the times you are receiving them?