Videocapture set framenumber gives a None frame (not always)

Ramesh_Nuthalapati · May 21, 2025, 6:03pm

I’m using a simple python code to extract thumbnail from a video.
Using opencv headless version 4.6.0.66 as AWS lambda layer
AWS Lambda has 10240MB memory and 1024MB Ephemeral storage.
Runtime is Python 3.9
Architecture is x86_64

The video capture set is giving Frame as None sometimes. Not able to narrow it down to a specific setting / case as larger files are giving resutls and smaller files choke some times.

cam = cv2.VideoCapture(input_file_signed_url)
if not cam.isOpened():
    raise Exception("Error: Could not open video.")
cam.set(cv2.CAP_PROP_POS_FRAMES, thumbnail_frame_number)
_, frame = cam.read()
cv2.imwrite(tmp_thumbnail_filename, frame)

What could go wrong here?

crackwitz · May 22, 2025, 8:21am

can you provide video files for which this happened?

is the video even long enough to have the frame index you request?

run ffprobe (ffmpeg project) on the failing video files. what is the report?

videos only secondarily can be indexed by frame number. primarily they are indexed by timestamp.

try CAP_PROP_POS_MSEC. I’d expect that to do better.

Ramesh_Nuthalapati · May 22, 2025, 6:02pm

Thanks for your reply crackwitz. Yes the Video is long enough.

Unfortunately I can’t share the file. I can share the ffprobe output.

Also, like you suggested I have changed my approach to read frame with millisecs if not available with frame number, but no luck. When I set the millisecs its setting it to wrong millisecs rather than the one I set to

time1 = cam.get(cv2.CAP_PROP_POS_MSEC)
fps = cam.get(cv2.CAP_PROP_FPS)
total_frames = cam.get(cv2.CAP_PROP_FRAME_COUNT)
print(time1, fps, total_frames)
ms_per_frame = 1000 / fps
milliseconds = thumbnail_frame_number * ms_per_frame
cam.set(cv2.CAP_PROP_POS_FRAMES, thumbnail_frame_number)
_, frame = cam.read()
if frame is None:
     print(f"Warning: Could not read frame. Setting milliseconds to {milliseconds}")
     cam.set(cv2.CAP_PROP_POS_MSEC, milliseconds)
time2 = cam.get(cv2.CAP_PROP_POS_MSEC)
ressult, frame = cam.read()

Output is
0.0 25.0 138769.0
Warning: Could not read frame. Setting milliseconds to 122040.0
4881600.0 — wrong millis is set

ffprobe version 7.1.1 Copyright (c) 2007-2025 the FFmpeg developers
  built with Apple clang version 16.0.0 (clang-1600.0.26.6)
  configuration: --prefix=/opt/homebrew/Cellar/ffmpeg/7.1.1_2 --enable-shared --enable-pthreads --enable-version3 --cc=clang --host-cflags= --host-ldflags='-Wl,-ld_classic' --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libaribb24 --enable-libbluray --enable-libdav1d --enable-libharfbuzz --enable-libjxl --enable-libmp3lame --enable-libopus --enable-librav1e --enable-librist --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libspeex --enable-libsoxr --enable-libzmq --enable-libzimg --disable-libjack --disable-indev=jack --enable-videotoolbox --enable-audiotoolbox --enable-neon
  libavutil      59. 39.100 / 59. 39.100
  libavcodec     61. 19.101 / 61. 19.101
  libavformat    61.  7.100 / 61.  7.100
  libavdevice    61.  3.100 / 61.  3.100
  libavfilter    10.  4.100 / 10.  4.100
  libswscale      8.  3.100 /  8.  3.100
  libswresample   5.  3.100 /  5.  3.100
  libpostproc    58.  3.100 / 58.  3.100
Input #0, mxf, from 'opencv-test-file.mxf':
  Metadata:
    operational_pattern_ul: 060e2b34.04010101.0d010201.01010900
    uid             : 9c5eaf61-766b-11e9-83be-90e2ba8ac885
    generation_uid  : 9c5eaf62-766b-11e9-bd75-90e2ba8ac885
    company_name    : Telestream
    product_name    : Flip Technology
    product_version_num: 1.0.0.0.0
    product_version : 3.0
    application_platform: win32
    product_uid     : ffeeddcc-bbaa-9988-7766-554433221100
    toolkit_version_num: 4.5.4.0.3
    modification_date: 2019-05-14T17:13:36.824000Z
    material_package_umid: 0x060A2B340101010501010D1213000000B74FB20317860580B6DD90E2BA8AC885
    timecode        : 09:59:40:00
  Duration: 01:32:30.76, start: 0.000000, bitrate: 55081 kb/s
  Stream #0:0: Video: mpeg2video (4:2:2), yuv422p(tv, bt709, top first), 1920x1080 [SAR 1:1 DAR 16:9], 50000 kb/s, 25 fps, 25 tbr, 25 tbn
      Metadata:
        file_package_umid: 0x060A2B340101010501010D121363BB74864FB20317860580C65590E2BA8AC885
        file_package_name: Source Package
        track_name      : Track 1
      Side data:
        cpb: bitrate max/min/avg: 50000000/0/0 buffer size: 47185920 vbv_delay: N/A
  Stream #0:1: Audio: pcm_s24le, 48000 Hz, 1 channels, s32 (24 bit), 1152 kb/s
      Metadata:
        file_package_umid: 0x060A2B340101010501010D121363BB74864FB20317860580C65590E2BA8AC885
        file_package_name: Source Package
        track_name      : Track 2
  Stream #0:2: Audio: pcm_s24le, 48000 Hz, 1 channels, s32 (24 bit), 1152 kb/s
      Metadata:
        file_package_umid: 0x060A2B340101010501010D121363BB74864FB20317860580C65590E2BA8AC885
        file_package_name: Source Package
        track_name      : Track 3
  Stream #0:3: Audio: pcm_s24le, 48000 Hz, 1 channels, s32 (24 bit), 1152 kb/s
      Metadata:
        file_package_umid: 0x060A2B340101010501010D121363BB74864FB20317860580C65590E2BA8AC885
        file_package_name: Source Package
        track_name      : Track 4
  Stream #0:4: Audio: pcm_s24le, 48000 Hz, 1 channels, s32 (24 bit), 1152 kb/s
      Metadata:
        file_package_umid: 0x060A2B340101010501010D121363BB74864FB20317860580C65590E2BA8AC885
        file_package_name: Source Package
        track_name      : Track 5

cudawarped · May 23, 2025, 9:15am

If this is a bug we’ll need a video file which reproduces it to track down the cause. I assume that everything works if you manually decode all the frames?

If you can’t share the video then maybe you can find out what the difference is between the actual frame returned and the frame you request. i.e. If you request frame 10,20,30,40 etc. which frame do you actually get when seeking? If it works for low numbered frames where does he process brake down?

Ramesh_Nuthalapati · May 23, 2025, 1:16pm

The lower number frames doesn’t work as well. The Frame returned is None.

cudawarped · May 23, 2025, 1:35pm

Does that mean it works when you don’t seek?
Which frame does seeking to frame 0,1,2,..,10,20 actually return?

crackwitz · May 23, 2025, 2:06pm

when in doubt, use PyAV.

there is still this

github.com/opencv/opencv

CvCapture_FFMPEG::setProperty(CAP_PROP_POS_*) sets `picture_pts` inconsistently

opened 12:48AM - 04 Jan 23 UTC

crackwitz

bug category: videoio

This code wasn't touched in a long time. I don't know if there are tests for it.… https://github.com/opencv/opencv/blob/9208dcb07c015e1fda44e40bb07b43c700b4bf46/modules/videoio/src/cap_ffmpeg_impl.hpp#L1957-L1978 I just noticed some inconsistencies in seeking that might be caused by this. 1. the `seek()` call has overloads for `double` (seconds) and `int64_t` (frame number) but in the POS_AVI_RATIO case it's called like it took a timestamp (in timebase) 2. the `picture_pts` value is set wrong _all_ cases Test video, 23.976 fps, about 10 seconds, specifics not important): `$ ffmpeg -f lavfi -i testsrc=duration=10:size=320x100:rate=24000/1001:decimals=3 -c:v libx264 -b:v 1M -y testsrc.mov` Test script: ```python import cv2 as cv vid = cv.VideoCapture("testsrc.mov", apiPreference=cv.CAP_FFMPEG) assert vid.isOpened() print("seek: frame 0") vid.set(cv.CAP_PROP_POS_FRAMES, 0) print("at:", vid.get(cv.CAP_PROP_POS_FRAMES), "frames,", vid.get(cv.CAP_PROP_POS_MSEC), "msec") for k in range(5): (rv, frame) = vid.read() assert rv print("read") #imshow(frame) print("at:", vid.get(cv.CAP_PROP_POS_FRAMES), "frames,", vid.get(cv.CAP_PROP_POS_MSEC), "msec") print("seek: frame 3") vid.set(cv.CAP_PROP_POS_FRAMES, 3) #print("seel: msec 130") #vid.set(cv.CAP_PROP_POS_MSEC, 130) print("at:", vid.get(cv.CAP_PROP_POS_FRAMES), "frames,", vid.get(cv.CAP_PROP_POS_MSEC), "msec") for k in range(5): (rv, frame) = vid.read() assert rv #print("read") #imshow(frame) print("at:", vid.get(cv.CAP_PROP_POS_FRAMES), "frames,", vid.get(cv.CAP_PROP_POS_MSEC), "msec") ``` That emits this: ``` seek: frame 0 at: 0.0 frames, 0.0 msec at: 1.0 frames, 0.0 msec at: 2.0 frames, 41.708333333333336 msec at: 3.0 frames, 83.41666666666667 msec at: 4.0 frames, 125.12499999999999 msec at: 5.0 frames, 166.83333333333334 msec seek: frame 3 at: 3.0 frames, 0.125 msec ### NOTICE HERE at: 4.0 frames, 125.12499999999999 msec at: 5.0 frames, 166.83333333333334 msec at: 6.0 frames, 208.54166666666666 msec at: 7.0 frames, 250.24999999999997 msec at: 8.0 frames, 291.9583333333333 msec ``` As you can see in the implementation of `CvCapture_FFMPEG::setProperty()`, seeking with `CAP_PROP_POS_FRAMES` puts the frame number into `picture_pts`, instead of an actual timestamp. This plain number is then interpreted as a timestamp (using timebase `1001/24000`), which results in `get(CAP_PROP_POS_MSEC)` reporting a number that's clearly off (uses `dts_to_sec(picture_pts) * 1000`, sensible). The following read() fixes that up though because seeking succeeded and `picture_pts` is just a "cached" value. Seeking with `CAP_PROP_POS_MSEC` is similarly broken. Some calculations, comparing actual timestamps and frame numbers interpreted as timestamps: ``` TB = 1/24000 # same as vid.get(cv.CAP_PROP_POS_AVI_RATIO) # one frame of time is an increment of 1001 ticks dts_to_sec = lambda ts: ts * TB dts_to_sec(3003) * 1000 # 125.12499999999999 dts_to_sec(3120) * 1000 # 130 dts_to_sec(7007) * 1000 # 291.9583333333333 # seek(POS_FRAMES, 3) dts_to_sec(3) * 1000 # 0.125 # seek(POS_MSEC, 130) dts_to_sec(130) * 1000 # 5.416666666666667 ``` I should mention the `CAP_PROP_POS_AVI_RATIO` is a complete mess too. `get()` returns the timebase (useful if we could seek using timestamps), set() calculates a timestamp from a 0..1 float and the video's total duration (in timebase), so that doesn't match up at all. If you think that warrants discussion, I could open a separate issue for that. I haven't given this enough of a look to propose a patch yet. I'd like some opinions on this, some guidance on design decisions that need to be made and documented in the course of fixing this. The AVI_RATIO stuff needs to be brought in line with what the docs state (value range 0 to 1). I'm inclined to introduce `CAP_PROP_TIMEBASE` and `CAP_PROP_POS_TIMESTAMP` just for completeness/passthrough but that wouldn't necessarily map cleanly onto backends besides FFMPEG. I am planning to propose a separate patch that would let me seek *quickly*, bypassing the frame-accurate positioning that happens in `CvCapture_FFMPEG::seek` (code after `avcodec_flush_buffers` call). I have some use cases that would greatly benefit from quick seeking and then querying the actual achieved position/timestamp.

Ramesh_Nuthalapati · May 23, 2025, 2:35pm

@crackwitz - not able to set mills in cam.set(cv2.CAP_PROP_POS_MSEC, milliseconds) is a known bug? Thanks.

Topic		Replies	Views
cap.set(cv2.CAP_PROP_POS_MSEC) changes to unreachable timestamp? Python videoio	1	3506	December 4, 2021
Opencv-python video capture set method not working as expected Python ffmpeg , videoio	4	3082	April 10, 2023
Can’t grab frame, what happend?	1	3636	March 14, 2022
Why does cv2.VideoCapture.read() return the same frame from a camera Python videoio	1	2643	May 28, 2023
Opencv-python version4.5.x get(cv2.CAP_PROP_POS_MSEC)'s problem Python videoio	0	635	February 17, 2022

Videocapture set framenumber gives a None frame (not always)

Related topics