That’s suprising, I would expect this to only use the dedicated hardware chip, are you sure GPU usage is not too much because of Yolo with the increased frame rate?
I would try cudacodec::VideoReader with allowFrameDrop==true if you can’t process the frames fast enough and you want a quick solution. Then if that works you may want to design your own pipeline as suggested by @crackwitz so you can decide which frames to drop.