cv2.CAP_PROP_POS_FRAMES sometimes starts at a keyframe sometimes doesn't

Python
1

Hey all, I have looked but don’t see any solid answers for this. I’m using opencv to do a specific job of cutting thumbs from video streams, then processing those thru several different AI systems.

The way I currently use it, I take the total frames, divide by number of CPUs in the system and then process the video as 12 (current CPU count) threads. I set a cv2.CAP_PROP_POS_FRAMES, start_frame for each of the threads.

Threads 2-12 work as expected, regardless of the frame I specify as the start frame it starts at exactly that frame.

Thread 1 however doesn’t, sometimes it properly starts at frame 0, but often it starts 1 second in to the video but thinks that it is at frame 0.

If I set start_frame to any value between 0-30 it always takes frame 30 and marks it as 0. If however I specify 31, 37, 42, etc it starts at exactly that frame. Which is also why all 11 other threads are picking up on exactly the correct thread.

In further testing, if I just negate the cv2.CAP_PROP_POS_FRAMES option, it starts at 0 and processes properly. So the bug/glitch/thing I don’t want, seems to only happen IF I set CAP_PROP_POS_FRAMES and set start_frame=0

In theory it makes sense (sorta) that it would go to the first key frame that is at 1 second, but why doesn’t it so that when PROP_POS_FRAMES is not set? Why doesn’t it search for a keyframe when I give it a start frame of 3001 (or any number after that first key frame).

I think most importantly, how do I get to a place where I can count on it starting at 0?

Thanks in advance!
S

CODE BELOW:


def process_video(start_frame, frame_partition, cudaProc, ar):
    print("START FRAME: ", start_frame)

    cap = cv2.VideoCapture(args.s3url)
    frameRate = cap.get(5) # Get the frame rate from the video
    frameCutRate = frameRate / 12 # we set to X frames per second

cap.set(cv2.CAP_PROP_POS_FRAMES, start_frame) 
    while(cap.isOpened()):

        frameId = int(cap.get(1)) # get current frame ID
        print("FRAME ID:", frameId)

        if (frameId > start_frame + frame_partition - 1):
            #print("reached end of looop for start_frame: ", start_frame)
            break
        ret, frame = cap.read()
        if (ret != True):
            print("ret does not equal True:", ret)
            break
----- Write frame to s3/etc....
2

you probably should not use opencv for this,
but something more “low-level”, like ffmpeg, gstreamer, etc

cv2.CAP_PROP_POS_FRAMES sometimes starts at a keyframe

there is no notion of keyframes here, your findings are arbitrary (random)

3

so why does opencv start at frame 30 regardless of what I put in for a start frame?

as for other tools, I have tested the thing we need to do with ffmpeg and others and opencv is > 10X faster than all of them. Why isn’t opencv a good candidate for this?

4

it starts at 30 if I specify a start frame, if I leave that blank it seems to properly start at 0. Is that what is intented? Regardless of if I’m using this for thumbs of to feed to our AI processes, if the frame it thinks is 0 is 1 second off all of our AI hits are off as well. That doesn’t seem to be the desired effect.

5

Screen Shot 2021-09-27 at 10.37.57 AM
Screen Shot 2021-09-27 at 10.37.57 AM1920×1081 163 KB

The numbers in the black bock are the actual frame numbers burned in to the video, the numbers in white “Frame 20” is what opencv thinks the frame number is on.

Take my issue of cutting thumbs out of it, is it expected and OK that opencv doesn’t know what frame number it is on?

6

and to add some input/clarity:

setting: cv2.CAP_PROP_POS_FRAMES, 0 => STARTS AT ACTUAL FRAME 30

cv2.CAP_PROP_POS_FRAMES, 0 => commenting out, actually starts at FRAME 0

But cv2.CAP_PROP_POS_FRAMES, start_frame is what I’m using to multithread the processing of video.

I can code around it:

if start_frame==0:
    start_frame+=1
else:
   cap.set(cv2.CAP_PROP_POS_FRAMES, start_frame)

But not super comfortable that this is the right hack/workaround to CAP_PROP_POS_FRAMES not being able to grab frame 0 when it is set.

7

I’m gonna reuse what I wrote elsewhere. ignore the part that doesn’t apply to you.

8

So Berak said that OpenCV doesn’t have any concept of key frames, your post asserts that it most certainly does.

I guess the question is this: If it is aware of key frames, when I put in any frame number I want to start at, it should seek to the key frame first. But it is not doing that, it is only seeking 1 second in and then all other frames requested are 100% accurate.

9

To add data to the question: I am reading an m3u8 manifest to process the video in. I have done some testing and reading the master/main m3u8 results in this weirdness where openCV thinks it is at frame 0 when its really at frame 30

HOWEVER:

If I break out the video only m3u8 manifest frame 0 lines up perfectly.

So it seems there is an issue in the OpenCV reading from M3U8 that causes the problem of skipping ahead 1 second on the first frame.

10

I said the video file has that concept, and ffmpeg does.

OpenCV can seek, kinda, sometimes, and sometimes it does that wrong, because someone once wrote some code that uses ffmpeg for that, but it’s buggy and highly depends on the specific video file (how it’s constructed), and ffmpeg isn’t the greatest library either, and some of its maintainers refuse to see the problems because that would be an insult to them personally.

OpenCV does not expose the concept of keyframes.

tldr: you are using the wrong tool for this job. you should be using ffmpeg’s libraries directly. OpenCV is not a video file handling library. it’s for computer vision. its video I/O is a mere convenience.

1 Like
11

Thanks for the clarification.

RE: shouldn’t be using opencv for this? I have the need to cut a frame from a video every .3 seconds (0.3, 0.6, 0.9, 1.3, 1.6,etc)… To do this in ffmpeg you have to loop thru doing a seek to each of the frames I need to collect, by the end of a 4 hour UHD file I’m at 3+ hours of grabbing stills with ffmpeg.

Alternatively, opencv steps thru each frame, I do a quick math calc to determine if the frame ends in .3/.6/.9 seconds based on variable FPS and if it does, I save 3 different versions of the frame (full-res, medium-res, low-res)

OpenCV does in 8 minutes what FFMPEG takes 3 hours to do, not sure why everybody keeps telling me that this isn’t a good use for OpenCV?

Literally the ONLY issue I have is that OpenCV if told to start at frame 0 on some jobs seeks to the first second and on others it doesnt… BUT doesn’t have a problem with any other frame being start_frame.

It seems to me that OpenCV thinking that frame 30 is frame 0 is not a problem with what I’m doing with it and more a problem with it not doing its desired intent. OpenCV is the gold standard for processing frames for AI, doesn’t it matter that frame 0 isn’t really frame 0 in some cases? I know for us, time matters and if the frames are all a second off the AI hits are a second off.

12

no. revisit the link I gave. it shows how to more or less precisely extract the frames for an arbitrary desired frame rate.

you could also just arbitrarily grab 10 frames and read one, repeat (a lot simpler than my SO answer above), or whatever ratio you like.