So this is just for inference? I’m double checking because if its for training its maybe better to write the frames out so you can randomize the order more efficiently.
Are you sure the decoding is the bottleneck and not the inference or the implementation of your DataLoader? I understand that you want to future proof but it may be an idea to get a benchmark of how long it takes if you read the frames from a folder vs CPU decoding in the DataLoader vs GPU decoding.
In the past I used fastai which was incredibly fast at loading images from disk, not sure if its DataLoader still has many modifications over the standard pytorch one or not but you could check that out.
On a related if you are on linux and your version of Ffmpeg is built with Nvidia HW acceleration you should be able to set the below variable
'OPENCV_FFMPEG_CAPTURE_OPTIONS=video_codec;h264_cuvid'
and cv::VideoCapture()should perform the video decoding on your GPU.