tamiri
April 8, 2023, 8:36am
1
Hi,
OS: Pop!_OS 22.04 LTS
Code (just the relevant part):
Any help will be appreciated!
opened 12:48AM - 04 Jan 23 UTC
This code wasn't touched in a long time. I don't know if there are tests for it.…
https://github.com/opencv/opencv/blob/9208dcb07c015e1fda44e40bb07b43c700b4bf46/modules/videoio/src/cap_ffmpeg_impl.hpp#L1957-L1978
I just noticed some inconsistencies in seeking that might be caused by this.
1. the `seek()` call has overloads for `double` (seconds) and `int64_t` (frame number) but in the POS_AVI_RATIO case it's called like it took a timestamp (in timebase)
2. the `picture_pts` value is set wrong _all_ cases
Test video, 23.976 fps, about 10 seconds, specifics not important):
`$ ffmpeg -f lavfi -i testsrc=duration=10:size=320x100:rate=24000/1001:decimals=3 -c:v libx264 -b:v 1M -y testsrc.mov`
Test script:
```python
import cv2 as cv
vid = cv.VideoCapture("testsrc.mov", apiPreference=cv.CAP_FFMPEG)
assert vid.isOpened()
print("seek: frame 0")
vid.set(cv.CAP_PROP_POS_FRAMES, 0)
print("at:", vid.get(cv.CAP_PROP_POS_FRAMES), "frames,", vid.get(cv.CAP_PROP_POS_MSEC), "msec")
for k in range(5):
(rv, frame) = vid.read()
assert rv
print("read")
#imshow(frame)
print("at:", vid.get(cv.CAP_PROP_POS_FRAMES), "frames,", vid.get(cv.CAP_PROP_POS_MSEC), "msec")
print("seek: frame 3")
vid.set(cv.CAP_PROP_POS_FRAMES, 3)
#print("seel: msec 130")
#vid.set(cv.CAP_PROP_POS_MSEC, 130)
print("at:", vid.get(cv.CAP_PROP_POS_FRAMES), "frames,", vid.get(cv.CAP_PROP_POS_MSEC), "msec")
for k in range(5):
(rv, frame) = vid.read()
assert rv
#print("read")
#imshow(frame)
print("at:", vid.get(cv.CAP_PROP_POS_FRAMES), "frames,", vid.get(cv.CAP_PROP_POS_MSEC), "msec")
```
That emits this:
```
seek: frame 0
at: 0.0 frames, 0.0 msec
at: 1.0 frames, 0.0 msec
at: 2.0 frames, 41.708333333333336 msec
at: 3.0 frames, 83.41666666666667 msec
at: 4.0 frames, 125.12499999999999 msec
at: 5.0 frames, 166.83333333333334 msec
seek: frame 3
at: 3.0 frames, 0.125 msec ### NOTICE HERE
at: 4.0 frames, 125.12499999999999 msec
at: 5.0 frames, 166.83333333333334 msec
at: 6.0 frames, 208.54166666666666 msec
at: 7.0 frames, 250.24999999999997 msec
at: 8.0 frames, 291.9583333333333 msec
```
As you can see in the implementation of `CvCapture_FFMPEG::setProperty()`, seeking with `CAP_PROP_POS_FRAMES` puts the frame number into `picture_pts`, instead of an actual timestamp. This plain number is then interpreted as a timestamp (using timebase `1001/24000`), which results in `get(CAP_PROP_POS_MSEC)` reporting a number that's clearly off (uses `dts_to_sec(picture_pts) * 1000`, sensible). The following read() fixes that up though because seeking succeeded and `picture_pts` is just a "cached" value.
Seeking with `CAP_PROP_POS_MSEC` is similarly broken.
Some calculations, comparing actual timestamps and frame numbers interpreted as timestamps:
```
TB = 1/24000 # same as vid.get(cv.CAP_PROP_POS_AVI_RATIO)
# one frame of time is an increment of 1001 ticks
dts_to_sec = lambda ts: ts * TB
dts_to_sec(3003) * 1000 # 125.12499999999999
dts_to_sec(3120) * 1000 # 130
dts_to_sec(7007) * 1000 # 291.9583333333333
# seek(POS_FRAMES, 3)
dts_to_sec(3) * 1000 # 0.125
# seek(POS_MSEC, 130)
dts_to_sec(130) * 1000 # 5.416666666666667
```
I should mention the `CAP_PROP_POS_AVI_RATIO` is a complete mess too. `get()` returns the timebase (useful if we could seek using timestamps), set() calculates a timestamp from a 0..1 float and the video's total duration (in timebase), so that doesn't match up at all. If you think that warrants discussion, I could open a separate issue for that.
I haven't given this enough of a look to propose a patch yet. I'd like some opinions on this, some guidance on design decisions that need to be made and documented in the course of fixing this. The AVI_RATIO stuff needs to be brought in line with what the docs state (value range 0 to 1). I'm inclined to introduce `CAP_PROP_TIMEBASE` and `CAP_PROP_POS_TIMESTAMP` just for completeness/passthrough but that wouldn't necessarily map cleanly onto backends besides FFMPEG.
I am planning to propose a separate patch that would let me seek *quickly*, bypassing the frame-accurate positioning that happens in `CvCapture_FFMPEG::seek` (code after `avcodec_flush_buffers` call). I have some use cases that would greatly benefit from quick seeking and then querying the actual achieved position/timestamp.
I don’t know if they ignore me automatically or manually, or they’re that backlogged, but they haven’t even triaged the issue yet (slapped appropriate labels on it)
1 Like
tamiri
April 8, 2023, 6:29pm
3
Thanks for the comment.
looks fairly easy to do, from what I remember of my investigation of the code. all the ffmpeg stuff is in that file.
just search and highlight a few of the variables relating to picture_pts
there are multiple issues in there, separate but related. it’s not going to be a single-line fix.
tamiri
April 10, 2023, 10:51pm
5
@crackwitz got a response from one of the moderators, and provided additional information they requested.video capture .set() method not working as expected · Issue #23472 · opencv/opencv · GitHub