CAP_PROP_POS_FRAMES is abnormal slow

Hi, After I had to reinstall my Anaconda version I have now an issue with jumping within a video file.

I am using vs.set(cv2.CAP_PROP_POS_FRAMES, framei) and it take really long time to perform. And it is relevant from where to where I jump.
Below you can find a test code show the problem.
First I just read in 600 frames ==> 11.78 s
Then I make trials with the jumps: from frame 0 to 600 ==> 6.6s
then from 600 to 1200 ==> 12.8s
Then I make the same but just with the retrieve command which is roughly half the time of reading.

Any idea why the vs.set(cv2.CAP_PROP_POS_FRAMES, framei) command is such slow?

Also just see the warning in the output with the GStreamer. even I don’t asked to do anything with it, I have this warning.

Setup is following:

  • Windows 11
  • Anaconda 2.3.1,
  • with an environment using Python 3.9.15
  • OpenCV 4.6.0, just installed in anaconda

Here the code:

import cv2
import time

filename = "E:/DaVinciResolveProjects/20221208_OffsetSystem/20230115_2157/20230115_2157.mkv"
vs = cv2.VideoCapture(filename)

############################################################
print('Task: read 600 frames')
framei = 0
tic = time.time()
while framei< 600:
    framei = framei + 1
    ret, frame = vs.read()
 
toc = time.time()    
print(str(round(toc-tic,3))+'s - '+str(round(600/(toc-tic),3))+'fps')

############################################################
print(' ')
print('Task: Jump from frame 0 to 600')

# rewind back to 0
framei = 0
vs.set(cv2.CAP_PROP_POS_FRAMES, framei)
tic = time.time()

# jump to 600
framei = 600
vs.set(cv2.CAP_PROP_POS_FRAMES, framei)
 
toc = time.time()
print(str(round(toc-tic,3))+'s - '+str(round(600/(toc-tic),3))+'fps')
    
############################################################
print(' ')
print('Task: Jump from frame 600 to 1200')
tic = time.time()
# now jump to 1200
framei = 1200
vs.set(cv2.CAP_PROP_POS_FRAMES, framei)
 
toc = time.time()
print(str(round(toc-tic,3))+'s - '+str(round(600/(toc-tic),3))+'fps')

############################################################
print(' ')
print('Task: Jump from frame 1200 to 1800')
tic = time.time()
# now jump to 1800
framei = 1800
vs.set(cv2.CAP_PROP_POS_FRAMES, framei)
 
toc = time.time()
print(str(round(toc-tic,3))+'s - '+str(round(600/(toc-tic),3))+'fps')

############################################################
print(' ')
print('Task: Jump back from frame 1200 to 0')
tic = time.time()
# now jump to 0
framei = 0
vs.set(cv2.CAP_PROP_POS_FRAMES, framei)
 
toc = time.time()
print(str(round(toc-tic,3))+'s - '+str(round(1200/(toc-tic),3))+'fps')

############################################################
print(' ')
print('Task: grab 600 frames')
framei = 0
tic = time.time()
while framei< 600:
    framei = framei + 1
    ret = vs.grab()
 
toc = time.time()    
print(str(round(toc-tic,3))+'s - '+str(round(600/(toc-tic),3))+'fps')

############################################################
print(' ')
print('Task: continue grab from frame 600-1200')
tic = time.time()
while framei< 1200:
    framei = framei + 1
    ret = vs.grab()
    frame = vs.retrieve()
 
toc = time.time()    
print(str(round(toc-tic,3))+'s - '+str(round(600/(toc-tic),3))+'fps')

This is the output:

[ WARN:0@7115.629] global C:\b\abs_74oeeuevib\croots\recipe\opencv-suite_1664548340488\work\modules\videoio\src\cap_gstreamer.cpp (2386) cv::handleMessage OpenCV | GStreamer warning: your GStreamer installation is missing a required plugin
[ WARN:0@7115.629] global C:\b\abs_74oeeuevib\croots\recipe\opencv-suite_1664548340488\work\modules\videoio\src\cap_gstreamer.cpp (2402) cv::handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module uridecodebin22 reported: Your GStreamer installation is missing a plug-in.
[ WARN:0@7115.629] global C:\b\abs_74oeeuevib\croots\recipe\opencv-suite_1664548340488\work\modules\videoio\src\cap_gstreamer.cpp (1356) cv::GStreamerCapture::open OpenCV | GStreamer warning: unable to start pipeline
[ WARN:0@7115.629] global C:\b\abs_74oeeuevib\croots\recipe\opencv-suite_1664548340488\work\modules\videoio\src\cap_gstreamer.cpp (862) cv::GStreamerCapture::isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created
Task: read 600 frames
11.954s - 50.192fps
 
Task: Jump from frame 0 to 600
6.765s - 88.698fps
 
Task: Jump from frame 600 to 1200
13.195s - 45.473fps
 
Task: Jump from frame 1200 to 1800
19.359s - 30.993fps
 
Task: Jump back from frame 1200 to 0
0.056s - 21424.104fps
 
Task: grab 600 frames
6.892s - 87.063fps
 
Task: continue grab from frame 600-1200
11.251s - 53.331fps

I have made some further investigations.

The code first reads 600 frames to check general read performance. Then it should make jumps of 600 frames. I see that the times for the jumps take longer and longer.

Here I will post now several different combinations of Python and OpenCV versions:

Python-Version 3.10.8 + OpenCV-Version 4.6.0:
Read 600 frames ==> 12.02s
5x600 jumps ==>67.5 s

Python-Version 3.9.15 + OpenCV-Version 4.6.0:
Read 600 frames ==> 11.7s
5x600 jumps ==>65.5 s

Python-Version 3.9.15 + OpenCV-Version 4.5.5:
Read 600 frames ==> 11.5 s
5x600 jumps ==>64.6 s

Python-Version 3.8.15 + OpenCV-Version 4.5.4:
Read 600 frames ==> 11.97s
5x600 jumps ==>6.24 s

Python-Version 3.8.15 + OpenCV-Version 4.0.1:
Read 600 frames ==> 10.391s
5x600 jumps ==>2.078 s

Python-Version 3.6.13 + OpenCV-Version 3.4.2:
Read 600 frames ==> 9.1s
5x600 jumps ==>2.019 s

I have found my sort tearm solution…
it is Python-Version 3.8.15 + OpenCV-Version 4.0.1, AND I also dont get this GSTreamer warning message.

But I am curious how everybody is doing these jumps. Especially if the footage is long??

import cv2
import time
import platform

print('Python-Version '+platform.sys.version)
print('OpenCV-Version '+cv2.__version__)

filename = "20230115_2157.mkv"
vs = cv2.VideoCapture(filename)

############################################################
print('Task: read 600 frames')
framei = 0
tic = time.time()
while framei< 600:
    framei = framei + 1
    ret, frame = vs.read()
    #cv2.imshow("Frame", frame)
    #key = cv2.waitKey(1) & 0xFF
 
toc = time.time()    
print(str(round(toc-tic,3))+'s - '+str(round(600/(toc-tic+0.001),3))+'fps')


############################################################
print(' ')
tic0 = time.time()
framei = 0
reps = 0
while reps < 5:
    tic = time.time()
    vs.set(cv2.CAP_PROP_POS_FRAMES, framei)
    framei = framei + 600
    reps = reps + 1
    toc = time.time()
    print('Jump to frame '+str(framei)+': '+str(round(toc-tic,3))+'s - '+str(round(600/(toc-tic+0.001),3))+'fps')

print('total time 5x600 frames jump: '+str(round(toc-tic0,3))+'s')


Python-Version 3.9.15 (main, Nov 24 2022, 14:39:17) [MSC v.1916 64 bit (AMD64)]
OpenCV-Version 4.5.5
Task: read 600 frames

[ WARN:0@1047.406] global C:\Windows\Temp\abs_0aavye6esb\croots\recipe\opencv-suite_1659973594283\work\modules\videoio\src\cap_gstreamer.cpp (2386) cv::handleMessage OpenCV | GStreamer warning: your GStreamer installation is missing a required plugin
[ WARN:0@1047.406] global C:\Windows\Temp\abs_0aavye6esb\croots\recipe\opencv-suite_1659973594283\work\modules\videoio\src\cap_gstreamer.cpp (2402) cv::handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module uridecodebin3 reported: Your GStreamer installation is missing a plug-in.
[ WARN:0@1047.406] global C:\Windows\Temp\abs_0aavye6esb\croots\recipe\opencv-suite_1659973594283\work\modules\videoio\src\cap_gstreamer.cpp (1356) cv::GStreamerCapture::open OpenCV | GStreamer warning: unable to start pipeline
[ WARN:0@1047.406] global C:\Windows\Temp\abs_0aavye6esb\croots\recipe\opencv-suite_1659973594283\work\modules\videoio\src\cap_gstreamer.cpp (862) cv::GStreamerCapture::isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created
11.512s - 52.115fps

Jump to frame 600: 0.025s - 23070.915fps
Jump to frame 1200: 6.437s - 93.196fps
Jump to frame 1800: 12.925s - 46.419fps
Jump to frame 2400: 19.554s - 30.682fps
Jump to frame 3000: 25.651s - 23.39fps
total time 5x600 frames jump: 64.594s

setting CAP_PROP_POS_FRAMES is called seeking in the video file. for most video files (formats), this is either slow or only able to jump to specific frames (you can call them “keyframes”)

please learn more about it.

your video file appears to have extremely long “Groups of Pictures (GOP)” or it’s severely fragmented and stored on a HDD. analyze it. ffprobe with various command line switches lets you analyze individual packets and frames.

I will learn about GOPs…
GOP explanation:

python code to read out GOP:

Meanwhile I have checked GOP structure: see below. 250 frames are between keyframes. Is it extreme? I create the video using OBS Recording software and I can adjust the value to keyframe intervall to 1 sec which will be lead to GOP size 60.

File is stored on SSD.
How do you explain, the big difference in the seeking times then?
I mean there is on same hardware same file a huge difference. Please check the results

Python-Version 3.9.15 + OpenCV-Version 4.5.5:
Read 600 frames ==> 11.5 s
5x600 jumps ==>64.6 s

Python-Version 3.8.15 + OpenCV-Version 4.5.4:
Read 600 frames ==> 11.97s
5x600 jumps ==>6.24 s

Python-Version 3.8.15 + OpenCV-Version 4.0.1:
Read 600 frames ==> 10.391s
5x600 jumps ==>2.078 s

Seeking time for the versions > 4.5.5 is proportional to the frame number!
I would assume, that seeking for a frame at the beginning or at the end should not have big difference. But in that versions, it I can seek only with a 90 fps. So if the frame I want to lookup is the frame 9000, then it needs 10 seconds. Absolutely unacceptable. Something must be wrong.

GOP Structure (all GOPs are same):

GOP: IBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBPBBP 250 CLOSED

potentially.

can you test with v4.7.0? anything earlier than that is only helpful in determining a regression. if there was an issue, and it was fixed, it makes no sense at all to deal with old versions.

can you provide such a video file for which this issue happens?

I can not test with 4.7 since it is not available in Anaconda. May be I can conda it somehow, but I don’t know with which command.

The original file I tested with is 15GB. So I have created a smaller file with 170MB and 36 seconds duration.

I have copied my test code and the video file to a Google Drive:
https://drive.google.com/drive/folders/16kEoFWHFoljmPWkwRLu4dsCPc-EWJjFR?usp=sharing

I have tested with this video file, with unchanged performance:

Python-Version 3.10.8 | packaged by conda-forge | (main, Nov 24 2022, 14:07:00) [MSC v.1916 64 bit (AMD64)]
OpenCV-Version 4.6.0
Task: read 600 frames

[ WARN:0@0.108] global C:\b\abs_74oeeuevib\croots\recipe\opencv-suite_1664548340488\work\modules\videoio\src\cap_gstreamer.cpp (2386) cv::handleMessage OpenCV | GStreamer warning: your GStreamer installation is missing a required plugin
[ WARN:0@0.109] global C:\b\abs_74oeeuevib\croots\recipe\opencv-suite_1664548340488\work\modules\videoio\src\cap_gstreamer.cpp (2402) cv::handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module uridecodebin0 reported: Your GStreamer installation is missing a plug-in.
[ WARN:0@0.109] global C:\b\abs_74oeeuevib\croots\recipe\opencv-suite_1664548340488\work\modules\videoio\src\cap_gstreamer.cpp (1356) cv::GStreamerCapture::open OpenCV | GStreamer warning: unable to start pipeline
[ WARN:0@0.109] global C:\b\abs_74oeeuevib\croots\recipe\opencv-suite_1664548340488\work\modules\videoio\src\cap_gstreamer.cpp (862) cv::GStreamerCapture::isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created
11.913s - 50.362fps

Jump to frame 400: 0.026s - 22217.103fps
Jump to frame 800: 4.64s - 129.281fps
Jump to frame 1200: 9.194s - 65.252fps
Jump to frame 1600: 13.565s - 44.227fps
Jump to frame 2000: 18.575s - 32.299fps
total time 5x400 frames jump: 46.001s

that video is 60 fps, 4K, with GOPs that are 250 frames long (~4 seconds).

such long GOP sizes will cost you when you jump to exact timestamps.

I ran your code on my computer from 2012. it’s going to be slow to decode 4K content.

Python-Version 3.9.13 (tags/v3.9.13:6de2ca5, May 17 2022, 16:36:42) [MSC v.1929 64 bit (AMD64)]
OpenCV-Version 4.7.0
Task: read 600 frames
21.572s - 27.812fps

Jump to frame 400: 0.031s
Jump to frame 800: 1.687s
Jump to frame 1200: 0.607s
Jump to frame 1600: 2.18s
Jump to frame 2000: 1.161s

these times are within my expectations. for every jump, it has to decode upto 250 frames before it reaches the exact desired time. with my old computer, this decoding will take a moment. it is a bounded amount of work though.

I don’t know what’s taking several seconds in your code or why the effort should grow.

try your code with VideoCapture(..., apiPreference=cv.CAP_FFMPEG) to make sure gstreamer isn’t interfering.

here’s some more info:

for framei in range(0, 1000, 20):
    t0 = time.perf_counter()
    vs.set(cv2.CAP_PROP_POS_FRAMES, framei)
    t1 = time.perf_counter()
    dt = t1 - t0
    print(f"jump to {framei:3d}: {dt:.3f} s", "*" * int(round(dt/0.1)))
jump to   0: 0.029 s
jump to  20: 0.281 s ***
jump to  40: 0.529 s *****
jump to  60: 0.807 s ********
jump to  80: 0.954 s **********
jump to 100: 1.102 s ***********
jump to 120: 1.399 s **************
jump to 140: 1.654 s *****************
jump to 160: 1.985 s ********************
jump to 180: 2.055 s *********************
jump to 200: 2.248 s **********************
jump to 220: 2.502 s *************************
jump to 240: 2.740 s ***************************
jump to 260: 2.889 s *****************************
jump to 280: 0.418 s ****
jump to 300: 0.664 s *******
jump to 320: 0.884 s *********
jump to 340: 1.088 s ***********
jump to 360: 1.263 s *************
jump to 380: 1.453 s ***************
jump to 400: 1.664 s *****************
jump to 420: 1.972 s ********************
jump to 440: 2.093 s *********************
jump to 460: 2.395 s ************************
jump to 480: 2.772 s ****************************
jump to 500: 3.023 s ******************************
jump to 520: 0.365 s ****
jump to 540: 0.569 s ******
jump to 560: 0.862 s *********
jump to 580: 1.002 s **********
jump to 600: 1.182 s ************
jump to 620: 1.386 s **************
jump to 640: 1.694 s *****************
jump to 660: 1.816 s ******************
jump to 680: 2.033 s ********************
jump to 700: 2.286 s ***********************
jump to 720: 2.678 s ***************************
jump to 740: 2.785 s ****************************
jump to 760: 2.897 s *****************************
jump to 780: 0.427 s ****
jump to 800: 0.672 s *******
jump to 820: 0.980 s **********
jump to 840: 1.509 s ***************
jump to 860: 1.947 s *******************
jump to 880: 2.043 s ********************
jump to 900: 2.141 s *********************
jump to 920: 2.723 s ***************************
jump to 940: 2.359 s ************************
jump to 960: 3.070 s *******************************
jump to 980: 3.107 s *******************************
for timestamp in np.arange(0, 15.0, 0.5):
    t0 = time.perf_counter()
    vs.set(cv2.CAP_PROP_POS_MSEC, int(round(timestamp * 1000)))
    t1 = time.perf_counter()
    dt = t1 - t0
    print(f"jump to {timestamp:.3f}s: {dt:.3f} s", "*" * int(round(dt/0.1)))
jump to 0.000s: 0.147 s *
jump to 0.500s: 0.379 s ****
jump to 1.000s: 0.740 s *******
jump to 1.500s: 1.080 s ***********
jump to 2.000s: 1.371 s **************
jump to 2.500s: 1.948 s *******************
jump to 3.000s: 2.447 s ************************
jump to 3.500s: 2.879 s *****************************
jump to 4.000s: 3.389 s **********************************
jump to 4.500s: 0.365 s ****
jump to 5.000s: 0.681 s *******
jump to 5.500s: 1.513 s ***************
jump to 6.000s: 1.432 s **************
jump to 6.500s: 1.952 s ********************
jump to 7.000s: 1.916 s *******************
jump to 7.500s: 2.178 s **********************
jump to 8.000s: 2.559 s **************************
jump to 8.500s: 2.969 s ******************************
jump to 9.000s: 0.665 s *******
jump to 9.500s: 0.957 s **********
jump to 10.000s: 1.402 s **************
jump to 10.500s: 1.783 s ******************
jump to 11.000s: 2.090 s *********************
jump to 11.500s: 2.283 s ***********************
jump to 12.000s: 2.856 s *****************************
jump to 12.500s: 3.103 s *******************************
jump to 13.000s: 0.414 s ****
jump to 13.500s: 0.827 s ********
jump to 14.000s: 1.256 s *************
jump to 14.500s: 1.691 s *****************
1 Like

that surely changed the situation because it’s a completely different file.

test your code with the video you uploaded. what does it do?

Thanks for benchmarking.
I see your system can read the 600 frames in 21.5 s, while mine is remarkably faster with only 12s.
I already tested with the smaller file, with same behaviour of all versions.

Anaconda supports now OpenCV4.7.0 with Python 3.11.0. So I have tried it and it runs really fast.

Python-Version 3.11.0 | packaged by conda-forge | (main, Jan 16 2023, 14:12:30) [MSC v.1916 64 bit (AMD64)]
OpenCV-Version 4.7.0
Task: read 600 frames
6.933s - 86.525fps

Jump to frame 400: 0.02s - 28565.941fps
Jump to frame 800: 0.636s - 941.706fps
Jump to frame 1200: 0.284s - 2108.727fps
Jump to frame 1600: 0.772s - 776.024fps
Jump to frame 2000: 0.457s - 1309.75fps
total time 5x400 frames jump: 2.17s

But still earlier versions still are slow in seeking.

I have tried the option: apiPreference=cv.CAP_FFMPEG and it turns out that it doesn’t read the video, and no time is consumed:

Python-Version 3.10.8 | packaged by conda-forge | (main, Nov 24 2022, 14:07:00) [MSC v.1916 64 bit (AMD64)]
OpenCV-Version 4.6.0
Task: read 600 frames
0.0s - 600000.0fps

Jump to frame 400: 0.0s - 600000.0fps
Jump to frame 800: 0.0s - 600000.0fps
Jump to frame 1200: 0.0s - 600000.0fps
Jump to frame 1600: 0.0s - 600000.0fps
Jump to frame 2000: 0.0s - 600000.0fps
total time 5x400 frames jump: 0.0s

But since Spyder is not working with Python 3.11.0 I cant use this versions. I will just wait for future.

Since your versions are working as expected, I think the environment with Anaconda is somehow broken. But even I setup new environments with somehow there is something wrong.

Crackwitz: Thank you for supporting.

Another thing I have recognized in my program is, that the figures window is looking different:
This figure window is in the version Python-Version 3.8.15 + OpenCV-Version 4.0.1:

And the same code running on Python-Version 3.10.8 + OpenCV-Version 4.6.0 make a window with additional icons:

official OpenCV-Python builds do not come with the Qt-themed imshow window. you are running third party packages or your own build.

OpenCV-Python 4.7.0 is compatible with all Pythons starting from 3.6 onwards. If you have Python 3.8, you can use many many versions of OpenCV-Python, going back several years.

Thank you @Ersin_Dogan ! I was banging my head against a brick wall.

After adjusting my conda environment to have Python 3.11.5 with OpenCV 4.7.0, I can report the slowness is resolved, and going between frames takes reasonable time indifferent to the step or video extension.

I don’t know if it applies, but there is an issue about frame-correct seeking.