Trouble using NVDIA hardware decoder when streaming from Camera

We are trying to offload encoding work from the CPU in our application. We built OpenCV 4.5.1 to enable CUDA, we can get cuda functions to run on there, but video decode does not use it.

We found a reference that said you need to set the environment variable to:

OPENCV_FFMPEG_CAPTURE_OPTIONS=“hwaccel;cuvid|video_codec;h264_cuvid|vsync;0”

This did work when reading from a file, but not when reading from an RTSP camera.

Another source had OPENCV_FFMPEG_CAPTURE_OPTIONS=“video_codec;h264_cuvid|rtsp_transport;tcp”

as the way to read from RTSP but this did not work

Any one know what the setup is to read from RTSP camera into NVIDIA decoder into Opencv?

can you get that (rtsp + hw decode) working on a standalone ffmpeg/ffplay?

The RTSP input works without the decode (no options) that’s how we are doing it. Shouldn’t this be known technology at this point, though?

I downloaded a recent ffmpeg , the most recent and did get it to work and use both CUDA and the NVDEC blocks. It wanted to use a different name for the hwaccel block tag and its name, now cuda. It did not like the rtsp stuff but worked without it. So we can use at least a modern version of FFMPEG to read it.

I’m working with 4.5.1 so its version of the libraries might be different and it did not work with the same data string in the OPENCV_FFMPEG_(etc) environment variable as did work for reading from a file.

ffmpeg -hwaccel_output_format cuda -vcodec h264_cuvid -i rtsp://admin:Margin719@192.168.0.101/Src/MediaInput/stream_1 cap001.mp4
ffmpeg version 2022-03-07-git-e645a1ddb9-full_build-www.gyan.dev Copyright (c) 2000-2022 the FFmpeg developers
  built with gcc 11.2.0 (Rev7, Built by MSYS2 project)
  configuration: --enable-gpl --enable-version3 --enable-static --disable-w32threads --disable-autodetect --enable-fontconfig --enable-iconv --enable-gnutls --enable-libxml2 --enable-gmp --enable-bzlib --enable-lzma --enable-libsnappy --enable-zlib --enable-librist --enable-libsrt --enable-libssh --enable-libzmq --enable-avisynth --enable-libbluray --enable-libcaca --enable-sdl2 --enable-libdav1d --enable-libdavs2 --enable-libuavs3d --enable-libzvbi --enable-librav1e --enable-libsvtav1 --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxvid --enable-libaom --enable-libopenjpeg --enable-libvpx --enable-mediafoundation --enable-libass --enable-frei0r --enable-libfreetype --enable-libfribidi --enable-libvidstab --enable-libvmaf --enable-libzimg --enable-amf --enable-cuda-llvm --enable-cuvid --enable-ffnvcodec --enable-nvdec --enable-nvenc --enable-d3d11va --enable-dxva2 --enable-libmfx --enable-libshaderc --enable-vulkan --enable-libplacebo --enable-opencl --enable-libcdio --enable-libgme --enable-libmodplug --enable-libopenmpt --enable-libopencore-amrwb --enable-libmp3lame --enable-libshine --enable-libtheora --enable-libtwolame --enable-libvo-amrwbenc --enable-libilbc --enable-libgsm --enable-libopencore-amrnb --enable-libopus --enable-libspeex --enable-libvorbis --enable-ladspa --enable-libbs2b --enable-libflite --enable-libmysofa --enable-librubberband --enable-libsoxr --enable-chromaprint
  libavutil      57. 22.100 / 57. 22.100
  libavcodec     59. 22.100 / 59. 22.100
  libavformat    59. 17.103 / 59. 17.103
  libavdevice    59.  5.100 / 59.  5.100
  libavfilter     8. 27.100 /  8. 27.100
  libswscale      6.  5.100 /  6.  5.100
  libswresample   4.  4.100 /  4.  4.100
  libpostproc    56.  4.100 / 56.  4.100
Input #0, rtsp, from 'rtsp://admin:Margin719@192.168.0.101/Src/MediaInput/stream_1':
  Metadata:
    title           : Media Presentation
  Duration: N/A, start: 0.100000, bitrate: N/A
  Stream #0:0: Video: h264 (High), yuvj420p(pc, bt709, progressive), 2560x1440 [SAR 1:1 DAR 16:9], 10 fps, 10 tbr, 90k tbn
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (h264_cuvid) -> h264 (libx264))
Press [q] to stop, [?] for help
[libx264 @ 000002908c3c0040] using SAR=1/1
[libx264 @ 000002908c3c0040] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2 AVX512
[libx264 @ 000002908c3c0040] profile High, level 5.0, 4:2:0, 8-bit
[libx264 @ 000002908c3c0040] 264 - core 164 r3094 bfc87b7 - H.264/MPEG-4 AVC codec - Copyleft 2003-2022 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=24 lookahead_threads=4 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=10 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to 'cap001.mp4':
  Metadata:
    title           : Media Presentation
    encoder         : Lavf59.17.103
  Stream #0:0: Video: h264 (avc1 / 0x31637661), nv12(pc, bt709, progressive), 2560x1440 [SAR 1:1 DAR 16:9], q=2-31, 10 fps, 10240 tbn
    Metadata:
      encoder         : Lavc59.22.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A
frame=  889 fps= 10 q=25.0 size=   30976kB time=00:01:21.80 bitrate=3102.1kbits/s speed=0.943x

I’m also a bit confused by the rtsp_transport … I didn’t make it up, there are documentation for it and other parameters passed in thru the command line.

4.5.1 might be too old. try again with the latest version.

some related links:

On 4.5.1 the default is to stream via tcp so rtsp_transport;tcp shouldn’t have any effect at all.

Did it work from rtsp at all?

If you built with CUDA then you could try using cudacodec::VideoCapture();

Unless anything has changed the most recent version of FFMpeg won’t work with OpenCV so even if it works stand alone it won’t help you.

I will check again but I don’t remember having any issues using older versions of FFMpeg with video_codec;h264_cuvid for streaming from RTSP as long as they were compiled with NVDEC support. Are you sure the version of FFMpeg you linked OpenCV against was compiled with NVDEC support?

1 Like

It runs NVDEC properly using Opencv/FFMPEG from a file, only RTSP inputs do not work. Therefore I think the FFMPEG was built with support for that.

Yes, on the run above with the parameters I was able to run with the most recent ffmpeg from rtsp, just by using the command line parameters in the post. It didn’t even recognize rtsp_transport at all which I found surprising as I have seen it listed in other sources as a parameter to allow selection of udp.

Our CUDA device is 7.5 which other sources have said is not supported with cudacodec. I think we got an error when we tried that.

That is odd considering rtsp_transport it is one of the listed options in the official documentation.

Can you post a link to the sources as it should be supported on all valid compute capabilities as long as you download and build against the Video Codec SDK?

which sources - we built against 4.5.1 official release source. Again, since it worked for reading from file. Or do you mean the command line FFMPEG? That was downloaded from the most recent build.

I’m now trying 4.5.5 as per the other gentleman’s posts.

I just tried it with the latest OpenCV. Now I get that h264_cuvid decoder is not recognized, which worked fine with the old version, and the most up to date FFMPEG.

Blockquote
url: rtsp://admin:Margin719@192.168.0.101/Src/MediaInput/stream_1
url: rtsp://admin:Margin719@192.168.0.102/Src/MediaInput/stream_1
url: rtsp://admin:Margin719@192.168.0.103/Src/MediaInput/stream_1
url: rtsp://admin:Margin719@192.168.0.104/Src/MediaInput/stream_1
[h264 @ 000001ddb063f300] cabac_init_idc 5 overflow
[h264 @ 000001ddb063f300] decode_slice_header error
[h264 @ 000001ddb063f300] no frame!
[h264 @ 000001ddb063f300] deblocking filter parameters 0 8 out of range
[h264 @ 000001ddb063f300] decode_slice_header error
[h264 @ 000001ddb063f300] no frame!
[h264 @ 000001ddb063f300] deblocking_filter_idc 9 out of range
[h264 @ 000001ddb063f300] decode_slice_header error
[h264 @ 000001ddb063f300] no frame!
[h264 @ 000001ddb063f300] deblocking filter parameters 0 8 out of range
[h264 @ 000001ddb063f300] decode_slice_header error
[h264 @ 000001ddb063f300] no frame!
[h264 @ 000001ddb063f300] deblocking_filter_idc 9 out of range
[h264 @ 000001ddb063f300] decode_slice_header error
[h264 @ 000001ddb063f300] no frame!
[h264 @ 000001ddb063f300] QP 4294967279 out of range
[h264 @ 000001ddb063f300] decode_slice_header error
[h264 @ 000001ddb063f300] no frame!
[h264 @ 000001ddb063f300] Missing reference picture, default is 0
[h264 @ 000001ddb063f300] decode_slice_header error
[ERROR:0@5.915] global /build/opencv/modules/videoio/src/cap_ffmpeg_impl.hpp (1117) open Could not find decoder ‘h264_cuvid’
[ERROR:0@5.916] global /build/opencv/modules/videoio/src/cap_ffmpeg_impl.hpp (1140) open VIDEOIO/FFMPEG: Failed to initialize VideoCapture
Could not open reference rtsp://admin:Margin719@192.168.0.101/Src/MediaInput/stream_1
FS: w: 0 h: 0
FPS0: 0 ex:
FPS: 0
Wait for Go:
< < < Game over! > > >
[h264 @ 000001de6634aa40] deblocking_filter_idc 9 out of range
[h264 @ 000001de6634aa40] decode_slice_header error
[h264 @ 000001de6634aa40] no frame!
[h264 @ 000001de6634aa40] QP 4294967279 out of range
[h264 @ 000001de6634aa40] decode_slice_header error
[h264 @ 000001de6634aa40] no frame!
[h264 @ 000001de6634aa40] Missing reference picture, default is 0
[h264 @ 000001de6634aa40] decode_slice_header error
[ERROR:0@9.991] global /build/opencv/modules/videoio/src/cap_ffmpeg_impl.hpp (1117) open Could not find decoder ‘h264_cuvid’
[ERROR:0@9.992] global /build/opencv/modules/videoio/src/cap_ffmpeg_impl.hpp (1140) open VIDEOIO/FFMPEG: Failed to initialize VideoCapture
Could not open reference rtsp://admin:Margin719@192.168.0.102/Src/MediaInput/stream_1
FS: w: 0 h: 0
FPS0: 0 ex:
FPS: 0
Wait for Go:
< < < Game over! > > >
[ERROR:0@13.011] global /build/opencv/modules/videoio/src/cap_ffmpeg_impl.hpp (1117) open Could not find decoder ‘h264_cuvid’
[ERROR:0@13.011] global /build/opencv/modules/videoio/src/cap_ffmpeg_impl.hpp (1140) open VIDEOIO/FFMPEG: Failed to initialize VideoCapture
Could not open reference rtsp://admin:Margin719@192.168.0.103/Src/MediaInput/stream_1
FS: w: 0 h: 0
FPS0: 0 ex:
FPS: 0
Wait for Go:
< < < Game over! > > >
[ERROR:0@16.033] global /build/opencv/modules/videoio/src/cap_ffmpeg_impl.hpp (1117) open Could not find decoder ‘h264_cuvid’
[ERROR:0@16.033] global /build/opencv/modules/videoio/src/cap_ffmpeg_impl.hpp (1140) open VIDEOIO/FFMPEG: Failed to initialize VideoCapture
Could not open reference rtsp://admin:Margin719@192.168.0.104/Src/MediaInput/stream_1
FS: w: 0 h: 0
FPS0: 0 ex:
FPS: 0
Wait for Go:
< < < Game over! > > >

I was using the old value that worked for a file just as a quick test:

OPENCV_FFMPEG_CAPTURE_OPTIONS=“hwaccel;cuvid|video_codec;h264_cuvid|vsync;0”

But it is complaining that it doesn’t know the video codec. Now, it knows this codec in the latest FFMPEG, and in the old 4.5.1 code. What is the proper value of the options to the NVIDIA NVDECODE from RSTP?

I did get some activity in the Video Decode block of the GPU before the streams failed.

Oh also the reference to the file doesn’t appear to refer to any file I can find in the installed distribution, or the built source.

I was refering to the source of information, sorry for the confusion. The point is cv::cudacodec::VideoReader() should work on all versions of CUDA any information to the contrary may be for specific circumstances which I would be interested to investigate if you have a link to the information?

Anyway this thread is getting confusing, I have just performed some tests on my build of 4.5.5 to confirm everything is working as it should, and it appears to be. I will try to compress the relevent information below.

Essentially for Nvidia hardware decoding you have two (possibly three) easy approaches:

  1. Build OpenCV against FFMpeg built against the Nvidia Video Codec SDK.
  2. Use cv::cudacodec::VideoReader().
  3. Use GStreamer - I haven’t personally tested this with OpenCV and RTSP but I have used the Nvidia plugins when not streaming from RTSP outside of OpenCV without any issues.

In my opinion the pros and cons for each are.

Build OpenCV against FFMpeg built against Nvidia Video Codec SDK.
Pros:

  • You can use the VideoCapture() api as is, frames are returned as Mat’s etc., so essentially the hardware decoding is hidden from you and should work without issues with existing code which uses `VideoCapture().

Cons:

  • FFMpeg needs to be built against the Nvidia Video Codec SDK - if you get similar errors to the following from VideoCapture::open()
    [ERROR:0@1.112] global /build/opencv/modules/videoio/src/cap_ffmpeg_impl.hpp (1117) open Could not find decoder 'h264_cuvid'
    then this is not the case.
  • If you use the old api pre OpenCV 4.5.2 you need to set environmental variables and make sure you know which codec you require in advance, e.g. setting
    OPENCV_FFMPEG_CAPTURE_OPTIONS=video_codec;h264_cuvid and trying to decode a h265 file will result in the following error
    [h264_cuvid @ 00000269ae74a000] Codec type or id mismatches
  • Decoding can be as slow as using the CPU (although this may have changed with the latest api), you simply get the benefit of reducing CPU utilization.

Procedure:
This vaires depending on the version of OpenCV you are using. For all versions of OpenCV adding the environmenal variables:

  • OPENCV_FFMPEG_CAPTURE_OPTIONS=video_codec;h264_cuvid
  • OPENCV_FFMPEG_CAPTURE_OPTIONS=video_codec;hevc_cuvid
    or even
  • OPENCV_FFMPEG_CAPTURE_OPTIONS=video_codec;h264_qsv
    if you have an Intel chip.

should work as long as the version of FFMpeg you are building against supports these codecs.

From 4.5.2 onwards this was made simpler but it is still fairly new and I am not sure if everything works. You no longer need to add an environmental variable and opening an RTSP source as

VideoCapture cap("rtsp://....", cv::CAP_FFMPEG, { CAP_PROP_HW_ACCELERATION, cv::VIDEO_ACCELERATION_ANY });

should result in hardware decoding and debug output similar to below

[ INFO:0@1.315] global repos/opencv/modules/videoio/src/cap_ffmpeg_hw.hpp (276) hw_check_device FFMPEG: Using d3d11va video acceleration on device: NVIDIA GeForce GTX 1060
[ INFO:0@1.316] global repos/opencv/modules/videoio/src/cap_ffmpeg_hw.hpp (566) hw_create_device FFMPEG: Created video acceleration context (av_hwdevice_ctx_create) for d3d11va on device 'default'

In addition this should work out of the box on windows as it is baked into opencv_videoio_ffmpegxxx_64.dll where as video_codec;h264_cuvidisn’t and would requir hacking the cmake files to link directly to FFMpeg libs built with hardware support.

cv::cudacodec::VideoReader().
Pros:

  • The output is a cv::cuda::GPUMat. This could also be a con if you require a cv::Mat.
  • You can get the raw encoded output at the same time as the decoded frame if required. This is useful if you want to record without the overhead of encoding by writing the encoded data to a file.
  • You don’t need to build or find a version of FFMpeg which is built against the Nvidia Video Codec SDK.

Cons:

  • OpenCV needs to be built against both the CUDA and Nvidia Video Codec SDK, if you are not using CUDA for anything else this is probably overkill.
  • You don’t need to mess around with environmental variables when using versions of OpenCV pre 4.5.2.

Procedure:

  • Build OpenCV again CUDA and the Nvidia VideoCodec SDK.

To summarize:

  • If you have the latest version of OpenCV, access to FFMpeg built against the Nvidia Video Codec SDK and don’t use CUDA for processing the resulting frames I would use approach 1).
  • If you can’t use a version of FFMpeg built against the Nvidia Video Codec SDK or use CUDA to process the resulting frames use approach 2).

Personally I use 2) because most of the work I do is on the GPU so I would add the following caveat. I am not sure if there is a limit on the number of concurrent hardware decoding sessions or how stable approach 1) is.

You can see the results of some comparisons I ran a while back here to get an idea of the decoding perfomance of each approach.

1 Like

Thank you for the information, I’m trying out some of them now, hopefully it gives me a path forward.

The place where I got the information about the codacodec::VideoReader is here:

among other places this exchange is relevant:

@ZiperRom1

ZiperRom1 commented on Jun 28, 2018

edited

So the cudacodec::createVideoReader to read video input directly in a GpuMat is deprecated since CUDA 7 right ?

@dapicard

ContributorAuthor

dapicard commented on Jun 28, 2018

Yes, this is actually what I understand. And I’ve never been able to use cudacodec video reader/writer with CUDA 7+

OK that makes sense what they are saying was probably true in 2018 but it is not since at least 2020, if you stick to the OpenCV 4.5 and above you should be fine. As a side note, they’re talking about CUDA SDK version 7 which is really old not compute capability 7 devices so you should be fine there aswell. If you decide to build cudacodec and are having any issues then I can provide you with build instructions if you want?