check out pyav.
it’s not able (or not easy to make it anyway) to use the GPU for decoding but the CPU is plenty fast (it really is!) AND it lets you jump around in the video a lot quicker than OpenCV does.
if you tell OpenCV to go to a particular timestamp (can’t jump to indices, that’s an illusion), it will go to the preceding keyframe and then walk (decode) to the target. for any seek, it does that. it can’t do anything else.
with pyav, you can tell it to jump somewhere and not walk the rest of the way. it’ll put you down on that keyframe.
your video likely has regular keyframes, probably not at 30 frame intervals. if you aren’t particular about that stride, pick the stride that your video file has. or, well, don’t actually “pick” a stride, just jump to places in whatever stride you want, and accept where it sets you down.
what I’m saying is that even with your frame skips, you notice that you don’t get a 30x speedup. that’s because you’re still decoding frames you don’t care about… with OpenCV anyway.
all this assumes that your video has keyframes and P/B-frames. if it’s fully intra-coded, then every seek directly lands on a keyframe and there’s nothing to improve.