Not able to read Video from Capture Card with GPU (Unsupported Format) while Reading with CPU works fine

Hello @cudawarped,
i copied my project to Windows to try out your original solution for using ffmpeg with dshow:

Build

OS: Windows 10 Pro (21H2)
OpenCV: 4.6.0
Cuda: 11.7

CMake Config
General configuration for OpenCV 4.6.0 =====================================
  Version control:               unknown

  Extra modules:
    Location (extra):            C:/Users/NvidiaPC/OpenCV/opencv_contrib-4.6.0/modules
    Version control (extra):     unknown

  Platform:
    Timestamp:                   2022-08-11T11:28:03Z
    Host:                        Windows 10.0.19044 AMD64
    CMake:                       3.24.0-rc5
    CMake generator:             Visual Studio 17 2022
    CMake build tool:            C:/Program Files/Microsoft Visual Studio/2022/Community/MSBuild/Current/Bin/amd64/MSBuild.exe
    MSVC:                        1932
    Configuration:               Release

  CPU/HW features:
    Baseline:                    SSE SSE2 SSE3
      requested:                 SSE3
    Dispatched code generation:  SSE4_1 SSE4_2 FP16 AVX AVX2 AVX512_SKX
      requested:                 SSE4_1 SSE4_2 AVX FP16 AVX2 AVX512_SKX
      SSE4_1 (18 files):         + SSSE3 SSE4_1
      SSE4_2 (2 files):          + SSSE3 SSE4_1 POPCNT SSE4_2
      FP16 (1 files):            + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 AVX
      AVX (5 files):             + SSSE3 SSE4_1 POPCNT SSE4_2 AVX
      AVX2 (33 files):           + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 FMA3 AVX AVX2
      AVX512_SKX (8 files):      + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 FMA3 AVX AVX2 AVX_512F AVX512_COMMON AVX512_SKX

  C/C++:
    Built as dynamic libs?:      YES
    C++ standard:                11
    C++ Compiler:                C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.32.31326/bin/Hostx64/x64/cl.exe  (ver 19.32.31332.0)
    C++ flags (Release):         /DWIN32 /D_WINDOWS /W4 /GR  /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _SCL_SECURE_NO_WARNINGS /Gy /bigobj /Oi  /fp:fast     /EHa /wd4127 /wd4251 /wd4324 /wd4275 /wd4512 /wd4589 /MP  /MD /O2 /Ob2 /DNDEBUG 
    C++ flags (Debug):           /DWIN32 /D_WINDOWS /W4 /GR  /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _SCL_SECURE_NO_WARNINGS /Gy /bigobj /Oi  /fp:fast     /EHa /wd4127 /wd4251 /wd4324 /wd4275 /wd4512 /wd4589 /MP  /MDd /Zi /Ob0 /Od /RTC1 
    C Compiler:                  C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.32.31326/bin/Hostx64/x64/cl.exe
    C flags (Release):           /DWIN32 /D_WINDOWS /W3  /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _SCL_SECURE_NO_WARNINGS /Gy /bigobj /Oi  /fp:fast     /MP   /MD /O2 /Ob2 /DNDEBUG 
    C flags (Debug):             /DWIN32 /D_WINDOWS /W3  /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _SCL_SECURE_NO_WARNINGS /Gy /bigobj /Oi  /fp:fast     /MP /MDd /Zi /Ob0 /Od /RTC1 
    Linker flags (Release):      /machine:x64  /INCREMENTAL:NO 
    Linker flags (Debug):        /machine:x64  /debug /INCREMENTAL 
    ccache:                      NO
    Precompiled headers:         NO
    Extra dependencies:          cudart_static.lib nppc.lib nppial.lib nppicc.lib nppidei.lib nppif.lib nppig.lib nppim.lib nppist.lib nppisu.lib nppitc.lib npps.lib cublas.lib cufft.lib -LIBPATH:C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.7/lib/x64
    3rdparty dependencies:

  OpenCV modules:
    To be built:                 aruco barcode bgsegm bioinspired calib3d ccalib core cudaarithm cudabgsegm cudacodec cudafeatures2d cudafilters cudaimgproc cudalegacy cudaobjdetect cudaoptflow cudastereo cudawarping cudev datasets dnn dnn_objdetect dnn_superres dpm face features2d flann fuzzy gapi hfs highgui img_hash imgcodecs imgproc intensity_transform line_descriptor mcc ml objdetect optflow phase_unwrapping photo plot quality rapid reg rgbd saliency shape stereo stitching structured_light superres surface_matching text tracking ts video videoio videostab wechat_qrcode world xfeatures2d ximgproc xobjdetect xphoto
    Disabled:                    -
    Disabled by dependency:      -
    Unavailable:                 alphamat cvv freetype hdf java julia matlab ovis python2 python3 sfm viz
    Applications:                tests perf_tests apps
    Documentation:               NO
    Non-free algorithms:         NO

  Windows RT support:            NO

  GUI: 
    Win32 UI:                    YES
    VTK support:                 NO

  Media I/O: 
    ZLib:                        build (ver 1.2.12)
    JPEG:                        build-libjpeg-turbo (ver 2.1.2-62)
    WEBP:                        build (ver encoder: 0x020f)
    PNG:                         build (ver 1.6.37)
    TIFF:                        build (ver 42 - 4.2.0)
    JPEG 2000:                   build (ver 2.4.0)
    OpenEXR:                     build (ver 2.3.0)
    HDR:                         YES
    SUNRASTER:                   YES
    PXM:                         YES
    PFM:                         YES

  Video I/O:
    DC1394:                      NO
    FFMPEG:                      YES (prebuilt binaries)
      avcodec:                   YES (58.134.100)
      avformat:                  YES (58.76.100)
      avutil:                    YES (56.70.100)
      swscale:                   YES (5.9.100)
      avresample:                YES (4.0.0)
    GStreamer:                   NO
    DirectShow:                  YES
    Media Foundation:            YES
      DXVA:                      YES

  Parallel framework:            Concurrency

  Trace:                         YES (with Intel ITT)

  Other third-party libraries:
    Intel IPP:                   2020.0.0 Gold [2020.0.0]
           at:                   C:/Users/NvidiaPC/OpenCV/build/3rdparty/ippicv/ippicv_win/icv
    Intel IPP IW:                sources (2020.0.0)
              at:                C:/Users/NvidiaPC/OpenCV/build/3rdparty/ippicv/ippicv_win/iw
    Lapack:                      NO
    Eigen:                       NO
    Custom HAL:                  NO
    Protobuf:                    build (3.19.1)

  NVIDIA CUDA:                   YES (ver 11.7, CUFFT CUBLAS FAST_MATH)
    NVIDIA GPU arch:             86
    NVIDIA PTX archs:

  cuDNN:                         NO

  OpenCL:                        YES (NVD3D11)
    Include path:                C:/Users/NvidiaPC/OpenCV/opencv-4.6.0/3rdparty/include/opencl/1.2
    Link libraries:              Dynamic load

  Python (for build):            NO

  Java:                          
    ant:                         NO
    JNI:                         NO
    Java wrappers:               NO
    Java tests:                  NO

  Install to:                    C:/Users/NvidiaPC/OpenCV/build/install
-----------------------------------------------------------------

Configuring done
Generating done

Unfortunately it does not work on first try:

cv::VideoCapture cam(0, cv::VideoCaptureAPIs::CAP_FFMPEG);

leads to the following error message:

Error Message:

OpenCV: terminate handler is called! The last OpenCV error is:
OpenCV(4.6.0) Error: Assertion failed (size.width>0 && size.height>0) in cv::imshow, file ...\OpenCV\opencv-4.6.0\modules\highgui\src\window.cpp, line 967

Also it is not clear to me how you set these variable in Windows. Is it via UI or PowerShell? Although in both cases the format of this variable seems strange to me.

and yes the format is strange because the one who implemented that in OpenCV decided that it needs to be strange. there is probably no reason for it.

1 Like

As mentioned in

  1. You need to specify the device name given in device manager, i.e. “HD Webcam”, not the index as you would when using the Direct Show backend (CAP_DSHOW).
  1. You need to build against FFmpeg not use the pre-built binaries, these won’t include any changes you made to cap_ffmpeg_impl.hpp.
1 Like

Hi,
could you please elaborate on your workflow for installing ffmpeg on Windows? How can I prevent cmake from using the prebuild binaries?

Thank you :slight_smile:

Sure thing, see this post

Thanks for the quick reply :slight_smile:

I extracted the Build and created an directory /cmake containing the mentioned .cmake file.
Afterwards I set the variables via Windows UI, but Cmake (I use cmake-gui on windows) doesn’t catch the FFMpeg Build.

FFMPEG: Downloading opencv_videoio_ffmpeg.dll from https://raw.githubusercontent.com/opencv/opencv_3rdparty/65ec04d4573dcdfa4531f0b9e67f35d8ffff873e/ffmpeg/opencv_videoio_ffmpeg.dll
FFMPEG: Downloading opencv_videoio_ffmpeg_64.dll from https://raw.githubusercontent.com/opencv/opencv_3rdparty/65ec04d4573dcdfa4531f0b9e67f35d8ffff873e/ffmpeg/opencv_videoio_ffmpeg_64.dll
FFMPEG: Downloading ffmpeg_version.cmake from https://raw.githubusercontent.com/opencv/opencv_3rdparty/65ec04d4573dcdfa4531f0b9e67f35d8ffff873e/ffmpeg/ffmpeg_version.cmake

I don’t use the GUI.

How did you set the CMake flag OPENCV_FFMPEG_USE_FIND_PACKAGE=ON?

How did you set CMAKE_PREFIX_PATH or FFMPEG_DIR environmental variables?

I used the Environment Editor for the varibles

Same here, althought i wasn’t sure if this would actually work.
How do you use cmake?

Inside CMake?

From the command line, that way I can
set FFMPEG_DIR=...
and then run
cmake.exe ... .. OPENCV_FFMPEG_USE_FIND_PACKAGE=ON

Although I would have thought you should be able to achieve the same thing in the GUI if you add OPENCV_FFMPEG_USE_FIND_PACKAGE using Add Entry not Environment. The only problem is you then have to configure again and it might not pick up this additional variable.

Yes inside cmake-gui you can edit environment variables. I previously set the varibles in the Windows UI for environmental variables, which led them to appear in the Environment Editor of Cmake-gui. With cmd set you can achieve the same.

This might be the issue. I tried opening the gui with cmd, but added the OPENCV_FFMPEG_USE_FIND_PACKAGE=ON Flag. No difference.

Works for me in the GUI, if I add OPENCV_FFMPEG_USE_FIND_PACKAGE as a boolean using Add Entry, even if I press configure twice.

I configured first without the setting OPENCV_FFMPEG_USE_FIND_PACKAGE and got the default downloaded version of FFmpeg

FFMPEG:                      YES (prebuilt binaries)
  avcodec:                   YES (58.134.100)
  avformat:                  YES (58.76.100)
  avutil:                    YES (56.70.100)
  swscale:                   YES (5.9.100)
  avresample:                YES (4.0.0)

then I set OPENCV_FFMPEG_USE_FIND_PACKAGE using Add Entry, pressed configure again and CMake picked up my FFmpeg install

FFMPEG:                      YES (find_package)
  avcodec:                   YES (59.18.100)
  avformat:                  YES (59.16.100)
  avutil:                    YES (57.17.100)
  swscale:                   YES (6.4.100)

Sorry, yes it works, but do you also have these errors?:

CMake Warning at modules/videoio/cmake/detect_ffmpeg.cmake:6 (find_package):
Found package configuration file:

C:/Users/NvidiaPC/FFMPEG_BUILD/cmake/ffmpeg-config.cmake

but it set FFMPEG_FOUND to FALSE so package “FFMPEG” is considered to be
NOT FOUND.
Call Stack (most recent call first):
modules/videoio/cmake/init.cmake:7 (include)
modules/videoio/cmake/init.cmake:11 (add_backend)
cmake/OpenCVModule.cmake:298 (include)
cmake/OpenCVModule.cmake:361 (_add_modules_1)
cmake/OpenCVModule.cmake:385 (ocv_glob_modules)
CMakeLists.txt:931 (ocv_register_modules)

Did you see the above in the output, did you download the FFmpeg build?

I downloaded and extracted the FFmpeg Build like you described in the other post.
Yes, I have the (find_package) instead of the (prebuild binaries):

FFMPEG:                      YES (find_package)
      avcodec:                   YES (58.134.100)
      avformat:                  YES (58.76.100)
      avutil:                    YES (56.70.100)
      swscale:                   YES (5.9.100)
      avresample:                YES (4.0.0)

I don’t know what you are doing differently, I just tried again from scratch setting OPENCV_FFMPEG_USE_FIND_PACKAGE=ON and FFMPEG_DIR to the directory containing the cmake script inside of the cmake gui and it picked up FFmpeg straight away.

Does your FFmpeg directory sturcture look similar to the below?

bin
 - avcodec-59.dll
- ...
cmake
 - ffmpeg-config.cmake
doc
...

I must have casually extracted the folder there and forgot about the original zip folder, so the dir cmake was not with bin in one dir.
However, now I moved them in an structure like this:

\bin\
\cmake\
  ffmpeg-config.cmake
\docs\
...

This gives me a new output, FFmpeg is found, but disabled:

Found FFMPEG: C:/Users/NvidiaPC/FFMPEG_BUILD/ffmpeg-n5.1-latest-win64-gpl-shared-5.1/include (found version "5.1-9-g4e4cc6e56a-20220821") found components: avutil swresample swscale avcodec avformat avfilter avdevice 
FFMPEG is disabled. Can't find suitable libavcodec library (minimal 54.35.0, found #define LIBAVCODEC_VERSION_MINOR  37;#define LIBAVCODEC_VERSION_MICRO 100.37.100).
FFMPEG is disabled. Can't find suitable libavformat library (minimal 54.20.4, found #define LIBAVFORMAT_VERSION_MINOR  27;#define LIBAVFORMAT_VERSION_MICRO 100.27.100).
FFMPEG is disabled. Can't find suitable libswscale library (minimal 2.1.1, found #define LIBSWSCALE_VERSION_MINOR   7;#define LIBSWSCALE_VERSION_MICRO 100.7.100).
FFMPEG libraries version check failed (minimal libav release 9.20, minimal FFMPEG release 1.1.16).

    FFMPEG:                      NO (find_package)
      avcodec:                   YES (#define LIBAVCODEC_VERSION_MINOR  37 #define LIBAVCODEC_VERSION_MICRO 100.37.100)
      avformat:                  YES (#define LIBAVFORMAT_VERSION_MINOR  27 #define LIBAVFORMAT_VERSION_MICRO 100.27.100)
      avutil:                    YES (57.28.100)
      swscale:                   YES (#define LIBSWSCALE_VERSION_MINOR   7 #define LIBSWSCALE_VERSION_MICRO 100.7.100)
      avresample:                NO

Sorry it looks like they moved things around in the headers, try
https://github.com/BtbN/FFmpeg-Builds/releases/download/autobuild-2022-06-30-12-32/ffmpeg-n5.0.1-7-g7389a49fd3-win64-gpl-shared-5.0.zip

Thanks, now FFmpeg is found without erros:

Found FFMPEG: C:/Users/NvidiaPC/FFMPEG_BUILD/include (found version "5.0.1-7-g7389a49fd3-20220630") found components: avutil swresample swscale avcodec avformat avfilter avdevice

    FFMPEG:                      YES (find_package)
      avcodec:                   YES (59.18.100)
      avformat:                  YES (59.16.100)
      avutil:                    YES (57.17.100)
      swscale:                   YES (6.4.100)
      avresample:                NO

Except avresample, but I don’t think this will be a problem.

I can build OpenCV now, but I can’t use ist anymore.
On Windows I use VS Code with a CMakeLists.txt Configuration.
I can run normal C++ Code with no problem so far, but as soon as I add an OpenCV function (or even cv::Mat), the build exits with 0, but the whole code isn’t executed anymore, there is also no error message.