setting CAP_PROP_POS_FRAMES is called seeking in the video file. for most video files (formats), this is either slow or only able to jump to specific frames (you can call them “keyframes”)
please learn more about it.
your video file appears to have extremely long “Groups of Pictures (GOP)” or it’s severely fragmented and stored on a HDD. analyze it. ffprobe with various command line switches lets you analyze individual packets and frames.