VideoCapture doesn't start with FFMPEG

C++
1

I’m trying to speed up reading frames, and since cudacodec::VideoReader is no more, the only way (as far as I’ve read) to get hardware-accelerated decoding is with FFMPEG.

I’ve compiled FFMPEG 4.4 following NVidia’s instructions, compiled OpenCV with FFMPEG as videoio plugin, but setting the capture API to CAP_FFMPEG yields no results (the feed doesn’t start, read returns false).
I’m using C++ and opening with direct path (such as “/dev/video2”), and it works with other APIs but it takes around 50ms to read a single frame (I hope to get 60fps).

I use Logitech Brio and C920 webcams (connected by USB).

Below is the configuration

-- General configuration for OpenCV 4.5.5-dev =====================================
--   Version control:               4.5.5-63-g6778e19710
-- 
--   Extra modules:
--     Location (extra):            /home/moddingear/opencv_contrib/modules
--     Version control (extra):     4.5.5-5-gbb3048d5
-- 
--   Platform:
--     Timestamp:                   2022-03-10T18:13:00Z
--     Host:                        Linux 5.13.0-35-generic x86_64
--     CMake:                       3.16.3
--     CMake generator:             Ninja
--     CMake build tool:            /usr/bin/ninja
--     Configuration:               RELEASE
-- 
--   CPU/HW features:
--     Baseline:                    SSE SSE2 SSE3
--       requested:                 SSE3
--     Dispatched code generation:  SSE4_1 SSE4_2 FP16 AVX AVX2 AVX512_SKX
--       requested:                 SSE4_1 SSE4_2 AVX FP16 AVX2 AVX512_SKX
--       SSE4_1 (16 files):         + SSSE3 SSE4_1
--       SSE4_2 (1 files):          + SSSE3 SSE4_1 POPCNT SSE4_2
--       FP16 (0 files):            + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 AVX
--       AVX (4 files):             + SSSE3 SSE4_1 POPCNT SSE4_2 AVX
--       AVX2 (31 files):           + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 FMA3 AVX AVX2
--       AVX512_SKX (5 files):      + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 FMA3 AVX AVX2 AVX_512F AVX512_COMMON AVX512_SKX
-- 
--   C/C++:
--     Built as dynamic libs?:      YES
--     C++ standard:                11
--     C++ Compiler:                /usr/bin/c++  (ver 9.4.0)
--     C++ flags (Release):         -fsigned-char -ffast-math -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections  -msse -msse2 -msse3 -fvisibility=hidden -fvisibility-inlines-hidden -fopenmp -O3 -DNDEBUG  -DNDEBUG
--     C++ flags (Debug):           -fsigned-char -ffast-math -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections  -msse -msse2 -msse3 -fvisibility=hidden -fvisibility-inlines-hidden -fopenmp -g  -O0 -DDEBUG -D_DEBUG
--     C Compiler:                  /usr/bin/cc
--     C flags (Release):           -fsigned-char -ffast-math -W -Wall -Werror=return-type -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections  -msse -msse2 -msse3 -fvisibility=hidden -fopenmp -O3 -DNDEBUG  -DNDEBUG
--     C flags (Debug):             -fsigned-char -ffast-math -W -Wall -Werror=return-type -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections  -msse -msse2 -msse3 -fvisibility=hidden -fopenmp -g  -O0 -DDEBUG -D_DEBUG
--     Linker flags (Release):      -Wl,--exclude-libs,libippicv.a -Wl,--exclude-libs,libippiw.a   -Wl,--gc-sections -Wl,--as-needed  
--     Linker flags (Debug):        -Wl,--exclude-libs,libippicv.a -Wl,--exclude-libs,libippiw.a   -Wl,--gc-sections -Wl,--as-needed  
--     ccache:                      NO
--     Precompiled headers:         NO
--     Extra dependencies:          m pthread cudart_static dl rt nppc nppial nppicc nppidei nppif nppig nppim nppist nppisu nppitc npps cublas cufft -L/usr/local/cuda/lib64 -L/usr/lib/x86_64-linux-gnu
--     3rdparty dependencies:
-- 
--   OpenCV modules:
--     To be built:                 alphamat aruco barcode bgsegm bioinspired calib3d ccalib core cudaarithm cudabgsegm cudacodec cudafeatures2d cudafilters cudaimgproc cudalegacy cudaobjdetect cudaoptflow cudastereo cudawarping cudev datasets dnn dnn_objdetect dnn_superres dpm face features2d flann freetype fuzzy gapi hdf hfs highgui img_hash imgcodecs imgproc intensity_transform line_descriptor mcc ml objdetect optflow phase_unwrapping photo plot python3 quality rapid reg rgbd saliency sfm shape stereo stitching structured_light superres surface_matching text tracking ts video videoio videostab viz wechat_qrcode xfeatures2d ximgproc xobjdetect xphoto
--     Disabled:                    world
--     Disabled by dependency:      -
--     Unavailable:                 cvv java julia matlab ovis python2
--     Applications:                perf_tests apps
--     Documentation:               NO
--     Non-free algorithms:         YES
-- 
--   GUI:                           GTK2
--     GTK+:                        YES (ver 2.24.32)
--       GThread :                  YES (ver 2.64.6)
--       GtkGlExt:                  NO
--     VTK support:                 YES (ver 9.1.0)
-- 
--   Media I/O: 
--     ZLib:                        /usr/lib/x86_64-linux-gnu/libz.so (ver 1.2.11)
--     JPEG:                        /usr/lib/x86_64-linux-gnu/libjpeg.so (ver 80)
--     WEBP:                        build (ver encoder: 0x020f)
--     PNG:                         /usr/lib/x86_64-linux-gnu/libpng.so (ver 1.6.37)
--     TIFF:                        build (ver 42 - 4.2.0)
--     JPEG 2000:                   build (ver 2.4.0)
--     OpenEXR:                     /usr/lib/x86_64-linux-gnu/libImath.so /usr/lib/x86_64-linux-gnu/libIlmImf.so /usr/lib/x86_64-linux-gnu/libIex.so /usr/lib/x86_64-linux-gnu/libHalf.so /usr/lib/x86_64-linux-gnu/libIlmThread.so (ver 2_3)
--     HDR:                         YES
--     SUNRASTER:                   YES
--     PXM:                         YES
--     PFM:                         YES
-- 
--   Video I/O:
--     DC1394:                      YES (2.2.5)
--     FFMPEG:                      YES
--       avcodec:                   YES (59.23.100)
--       avformat:                  YES (59.18.101)
--       avutil:                    YES (57.22.100)
--       swscale:                   YES (6.5.100)
--       avresample:                YES (4.0.0)
--     GStreamer:                   YES (1.16.2)
--     v4l/v4l2:                    YES (linux/videodev2.h)
-- 
--   Parallel framework:            TBB (ver 2020.2 interface 11102)
-- 
--   Trace:                         YES (with Intel ITT)
-- 
--   Other third-party libraries:
--     Intel IPP:                   2020.0.0 Gold [2020.0.0]
--            at:                   /home/moddingear/opencv/build/3rdparty/ippicv/ippicv_lnx/icv
--     Intel IPP IW:                sources (2020.0.0)
--               at:                /home/moddingear/opencv/build/3rdparty/ippicv/ippicv_lnx/iw
--     VA:                          NO
--     Lapack:                      YES (/usr/lib/x86_64-linux-gnu/liblapack.so /usr/lib/x86_64-linux-gnu/libcblas.so /usr/lib/x86_64-linux-gnu/libatlas.so)
--     Eigen:                       YES (ver 3.3.7)
--     Custom HAL:                  NO
--     Protobuf:                    build (3.19.1)
-- 
--   NVIDIA CUDA:                   YES (ver 11.6, CUFFT CUBLAS FAST_MATH)
--     NVIDIA GPU arch:             50 52 61 75 86
--     NVIDIA PTX archs:
-- 
--   OpenCL:                        YES (no extra features)
--     Include path:                /home/moddingear/opencv/3rdparty/include/opencl/1.2
--     Link libraries:              Dynamic load
-- 
--   Python 3:
--     Interpreter:                 /usr/bin/python3 (ver 3.8.10)
--     Libraries:                   /usr/lib/x86_64-linux-gnu/libpython3.8.so (ver 3.8.10)
--     numpy:                       /home/moddingear/.local/lib/python3.8/site-packages/numpy/core/include (ver 1.22.2)
--     install path:                /usr/lib/python3/dist-packages/cv2/python-3.8
-- 
--   Python (for build):            /usr/bin/python2.7
-- 
--   Java:                          
--     ant:                         NO
--     JNI:                         NO
--     Java wrappers:               NO
--     Java tests:                  NO
-- 
--   Install to:                    /usr
-- -----------------------------------------------------------------

I don’t know what is wrong, and it’s probably something stupid.
Any help will be greatly appreciated.

2

just to confirm… you want to read from a video device, not from a file, right?

then hardware acceleration will hardly matter, and ffmpeg isn’t one of the backends for reading from a device (V4L is). there is nothing to decode, or the decoding (MJPEG, maybe H.264) costs very little.

what might limit your frame rate is the (edit:) format of the video data. OpenCV doesn’t request compressed modes by default. you have to ask for that. an USB 2 webcam like the C920 simply can’t push FullHD at 30 fps over the wire. it would have to compress, or lower the frame rate.

set CAP_PROP_FOURCC to VideoWriter_fourcc(*"MJPG") (or the C++ equivalent) and then any other properties (frame size, fps)

3

Yes, I want to read from a video device (webcam).

I tried using MJPEG with the C920 to get the 30fps, but just to check that it wasn’t the decompression that was slowing down everything, I got a logitech Brio that can do 30 fps in YUYV thanks to USB3. Both get about 15fps, in MJPEG, and the Brio also gets 15 fps in YUYV.

Still, that’s half of what the webcam is technically capable, and other applications (such as guvcviewer) manage a solid 30 fps with the C920 and 60fps with the Brio, so there must be some acceleration possible somewhere on my hardware.

4

I’m gonna need to see a minimal example of how exactly you use an VideoCapture instance. perhaps there’s something undisclosed that you do that should be done differently

5

I open my VideoCaptures like this :

feed = new VideoCapture();
cout << "Opening device at \"" << DevicePath << "\" with API id " << ApiID << endl;
feed->open(DevicePath, ApiID);
feed->set(CAP_PROP_FOURCC, VideoWriter::fourcc('M', 'J', 'P', 'G'));
//feed->set(CAP_PROP_FOURCC, VideoWriter::fourcc('Y', 'U', 'Y', 'V'));
feed->set(CAP_PROP_FRAME_WIDTH, CaptureSize.width);
feed->set(CAP_PROP_FRAME_HEIGHT, CaptureSize.height);
feed->set(CAP_PROP_FPS, fps);
feed->set(CAP_PROP_BUFFERSIZE, 1);

with ApiID equal to CAP_FFMPEG, and DevicePath equal to “/dev/video2” (a valid device path for my USB-connected webcam).
I use the new keyword because the capture is all abstracted away in a struct instead of being in the main but otherwise it’s the same as you’d find in any tutorial.
Then when I want a frame I do

feed->read(frame);

In parallel to this, I’m looking for options in GStreamer, and I’ve managed to compile nvdec but I can’t manage to start OpenCV’s GStreamer either… (using ApiID = CAP_GSTREAMER and DevicePath = “/dev/video2”)

6

again, try CAP_V4L2/CAP_V4L. that’s made for video devices.

CAP_FFMPEG is intended for files, not video devices.

7

CAP_V4L2 works but is slow, that’s why I’m trying to use something else.

8

does your CPU show increased utilization? if not, then the reason for any slowness won’t be resource usage, but the ability of the device, or a bug in some programming.