Recording camera video - fps drop?

,
1

Hello,
I’m working with OpenCV 4.5 on Jetson Nano 2GB [ CUDA 10 ] - Debian system.
I need to record video from my webcam and save it to .mov or .mp4, the problem is that I do not know which encoding to use, while I tried a variety of fourcc-s ( like “mp4v”, “h264”, “avc1”) all of them generate a drop of fps (from 25 to 8 :worried:) with resolution 1280x720. If I resize to a smaller size, it’s work’s better but I can’t go any lower :frowning:.

So I thought about two solutions:

  • Saving raw frames and encoding them only when a user would like to download recordings.
  • Use some faster encoding

But I have no idea how to do that, I could only try different combinations for the whole day and I found nearly nothing! :triumph:

I by chance saved once in MPEG-4 video format and 7400 kbps, and while I could not play that file with any program (Internal stream error) - I could read it by OpenCV again and then save it correctly (seems like my first idea), but I don’t know how I have done that. Now when I try to do the same I’m only getting MPEG-4 (Simple Profile) which is not working :frowning: . I have done that on lower resolution (640x480) so I’m not even sure if that’s really would solve my problem.

I’m working on Jetson nano and theoretically have access to GPU, but I could not find a working way to do this with it.

I had issues with video reading speed so I’m not using VideoCapture but NvAPI and I get cv::Mat from it. [Check Here]

That’s my first post here - sorry if I have made any mistakes.

2

Does your code work on a desktop/laptop computer?

OpenCV VideoWriter uses the underlying ffmpeg engine to encode the videos, so it’s not really an OpenCV related problem. Video encoding is a computationnaly intensive task, so the ARM CPUs don’t have enough power to do it in real time. They have to be done on the video hardware - and I’m not sure that ffmpeg is configured for this.

First, check if the encoding is done on CPU or GPU (using htop or system monitor for CPU and nvidia-smi for GPU).

Then, try to do the oncoding using NvAPI too.

To be honest, I never did video encoding on the Nvidia Nano, so these are only some general observations. Maybe there are some people who know this better. And probably the nvidia forum is a better place for this question, they know better that hardware.

3

If I use VideoCapture instead NvAPI then it’s working just fine - but computing power is on a different scale after all.

Oh, I checked FFmpeg forums but nearly all posts use commands so I thought that even thru is ffmpg in the background I want OpenCV based solution :sweat_smile: .

They have to be done on the video hardware - and I’m not sure that ffmpeg is configured for this.

I don’t know if it is, can I check somehow?

First, check if the encoding is done on CPU or GPU (using htop or system monitor for CPU and nvidia-smi for GPU).

When I set encoding to H264 (which should be supported by Jetson) it doesn’t use HW encoders or GPU at all - I used jtop to check that. It seem CPU only.

Then, try to do the oncoding using NvAPI too.

I found a NvAPI example that does that using HW encoders, but that API is crappy, and not really a thing that I would like to work with :worried: , I tried to execute that example on some files saved from OpenCV (that go to my first idea) that maybe I could save my video somehow raw and then encode using that example - but I don’t know how to save the video without encoding at all :frowning: . I tried anyway with a few fourcc-s, but it’s not allowed in example or I get shifted video :

image
image933×519 71.3 KB

Maybe there are some people who know this better. And probably the nvidia forum is a better place for this question, they know better that hardware.

I will ask there as well then :slight_smile:

4

Note that there might be a bottleneck in writing RAW data too. The microSD cards are not very fast. Raw data takes 1280x720x3 = ~3MB/image, 75MB/sec at 25fps.
You can reduce the data size if you can capture the bayer image from the camera and interpolate it later (this only works if this is the native resolution of the camera).
JPEG encoding won’t work on the CPU at this speed neither. libturbojpeg runs on CPU and it isn’t multithreaded, so it can use only one core.

5

Note that there might be a bottleneck in writing RAW data too, Raw data takes 1280x720x3 = ~3MB/image, 75MB/sec at 25fps.

That may be the reason why sometimes it’s working just fine at the beginning of the recording and is getting worse with each second?

I found something like that, here someone uses .resize(0,1) to send frame thru socket, can I do the same in my case (it’s that the raw I was looking for)? It’s doesn’t really change the size of Mat, just make it continuous, so I’m not sure if that will solve that issue you just mentioned :worried: …

You can reduce the data size if you can capture the bayer image from the camera and interpolate it later (this only works if this is the native resolution of the camera)

Are there some functions for that in OpenCV, so I can do a quick test?

6

Yes. It puts the frames in the cache (memory), and when the memory is filled up, it can’t continue until the cache is somewhat filled up.

Normally the captured frames are continuous, so no need to resize(). Just do:

ofstream outfile("image.dat", ios::out | ios::binary);
outfile.write(image.ptr(),image.rows*image.cols*image.channels());
outfile.close();

But this won’t reduce the file size.

capture.set(CAP_PROP_FORMAT,-1);

should work (if supported): documentation
Then use cvtColor(bayer,COLOR_BAYER...2BGR); to get a color image (choose the correct type of decoder for your camera)

That said, you should really use hardware acceleration for video encoding on ARM. If it’s important for your project, it’s worth the effort.

7

Thanks for the explantation :slight_smile:

I will try saving like that, maybe that file can be then thrown to NvAPI Example and HW encode to H264, we will see :stuck_out_tongue:

capture.set(CAP_PROP_FORMAT,-1);

But while I’m using frames directly from NvAPI there’s no capture to set :<
can I like do that other way around? cvtColor to BAYER and then save to file?

That said, you should really use hardware acceleration for video encoding on ARM. If it’s important for your project, it’s worth the effort.

Yup, I already asked on Nvidia forum, but have yet to get an answer - I thought that meanwhile, I could try smth :stuck_out_tongue:

8

I will try saving like that, maybe that file can be then thrown to NvAPI Example and HW encode to H264, we will see :stuck_out_tongue:

I tried, and as expected it’s starting to be laggy after a second - that too much for it, and even throwing that file to example to encode with H264 gives the wrong video.

I also tried to convert my frames to Bayer as shown here, and then save it to file with H264 encoding (that way of saving cause like 1 fps drop, I can deal with that) but after that when I tried to read that file (on my PC) I get error

[ INFO:0] global C:\build\master_winpack-build-win64-vc15\opencv\modules\videoio\src\videoio_registry.cpp (191) cv::`anonymous-namespace'::VideoBackendRegistry::VideoBackendRegistry VIDEOIO: Enabled backends(7, sorted by priority): FFMPEG(1000); GSTREAMER(990); INTEL_MFX(980); MSMF(970); DSHOW(960); CV_IMAGES(950); CV_MJPEG(940)
[ INFO:0] global C:\build\master_winpack-build-win64-vc15\opencv\modules\videoio\src\backend_plugin.cpp (370) cv::impl::getPluginCandidates Found 3 plugin(s) for FFMPEG
[ INFO:0] global C:\build\master_winpack-build-win64-vc15\opencv\modules\videoio\src\backend_plugin.cpp (175) cv::impl::DynamicLib::libraryLoad load H:\Moje\Pulpit\Praca\Git\TestProjects\OpenCVTest\x64\Debug\opencv_videoio_ffmpeg440_64d.dll => FAILED
[ INFO:0] global C:\build\master_winpack-build-win64-vc15\opencv\modules\videoio\src\backend_plugin.cpp (175) cv::impl::DynamicLib::libraryLoad load opencv_videoio_ffmpeg440_64d.dll => FAILED
[ INFO:0] global C:\build\master_winpack-build-win64-vc15\opencv\modules\videoio\src\backend_plugin.cpp (175) cv::impl::DynamicLib::libraryLoad load opencv_videoio_ffmpeg440_64.dll => OK
[ INFO:0] global C:\build\master_winpack-build-win64-vc15\opencv\modules\videoio\src\backend_plugin.cpp (236) cv::impl::PluginBackend::PluginBackend Video I/O: loaded plugin 'FFmpeg OpenCV Video I/O plugin'
[ INFO:0] global C:\build\master_winpack-build-win64-vc15\opencv\modules\videoio\src\backend_plugin.cpp (370) cv::impl::getPluginCandidates Found 2 plugin(s) for GSTREAMER
[ INFO:0] global C:\build\master_winpack-build-win64-vc15\opencv\modules\videoio\src\backend_plugin.cpp (175) cv::impl::DynamicLib::libraryLoad load H:\Moje\Pulpit\Praca\Git\TestProjects\OpenCVTest\x64\Debug\opencv_videoio_gstreamer440_64d.dll => FAILED
[ INFO:0] global C:\build\master_winpack-build-win64-vc15\opencv\modules\videoio\src\backend_plugin.cpp (175) cv::impl::DynamicLib::libraryLoad load opencv_videoio_gstreamer440_64d.dll => FAILED
[ INFO:0] global C:\build\master_winpack-build-win64-vc15\opencv\modules\videoio\src\backend_plugin.cpp (370) cv::impl::getPluginCandidates Found 2 plugin(s) for INTEL_MFX
[ INFO:0] global C:\build\master_winpack-build-win64-vc15\opencv\modules\videoio\src\backend_plugin.cpp (175) cv::impl::DynamicLib::libraryLoad load H:\Moje\Pulpit\Praca\Git\TestProjects\OpenCVTest\x64\Debug\opencv_videoio_intel_mfx440_64d.dll => FAILED
[ INFO:0] global C:\build\master_winpack-build-win64-vc15\opencv\modules\videoio\src\backend_plugin.cpp (175) cv::impl::DynamicLib::libraryLoad load opencv_videoio_intel_mfx440_64d.dll => FAILED
[ INFO:0] global C:\build\master_winpack-build-win64-vc15\opencv\modules\videoio\src\backend_plugin.cpp (370) cv::impl::getPluginCandidates Found 2 plugin(s) for MSMF
[ INFO:0] global C:\build\master_winpack-build-win64-vc15\opencv\modules\videoio\src\backend_plugin.cpp (175) cv::impl::DynamicLib::libraryLoad load H:\Moje\Pulpit\Praca\Git\TestProjects\OpenCVTest\x64\Debug\opencv_videoio_msmf440_64d.dll => FAILED
[ INFO:0] global C:\build\master_winpack-build-win64-vc15\opencv\modules\videoio\src\backend_plugin.cpp (175) cv::impl::DynamicLib::libraryLoad load opencv_videoio_msmf440_64d.dll => OK
[ INFO:0] global C:\build\master_winpack-build-win64-vc15\opencv\modules\videoio\src\backend_plugin.cpp (236) cv::impl::PluginBackend::PluginBackend Video I/O: loaded plugin 'Microsoft Media Foundation OpenCV Video I/O plugin'
[ INFO:0] global C:\build\master_winpack-build-win64-vc15\opencv\modules\videoio\src\cap_images.cpp (282) cv::icvExtractPattern Pattern: VJetson/rec_%04d_27_1_21.mp4 @ 1525
[ INFO:0] global C:\build\master_winpack-build-win64-vc15\opencv\modules\videoio\src\cap_images.cpp (313) cv::CvCapture_Images::open CAP_IMAGES: Stop scanning. Can't read image file: VJetson/rec_1525_27_1_21.mp4

Btw what about that:

I by chance saved once in MPEG-4 video format and 7400 kbps, and while I could not play that file with any program (Internal stream error) - but I could read it by OpenCV again and then save it correctly (seems like my first idea), but I don’t know how I have done that. Now when I try to do the same I’m only getting MPEG-4 (Simple Profile) which is not working :frowning: . I have done that on lower resolution (640x480) so I’m not even sure if that’s really would solve my problem.

Do you know how to do that again? At last, I could check if that would be ok as well in full resolution :sweat_smile: