Slow frame read from webcam

Hi,

As per the title, I’m seeing very slow frame reads from webcam. At a resolution at which it can output 120FPS, I’m only getting 20-40FPS using a very simple OpenCV application, while ffmpeg fed directly from the webcam gets ~60FPS while also doing h264 encoding and streaming, so something is definitely off. One other odd thing is that on one start it could stay locked at 50FPS, while in other instances it wouldn’t be able to get anything better than 20FPS for the duration of the run.

Any points on where to start looking please?

Thanks!

any waitKey() involved?

what API is involved, DSHOW/MSMF? V4L?

use ffmpeg to query the camera’s modes (available for dshow and v4l, that I know of). is there any mention of mjpeg or other compressed formats?

you’d have to set(CAP_PROP_FOURCC, ...) and see what works. "MJPG" fourcc has given me “higher” frame rates from some cameras. OpenCV doesn’t “try them all”.

No waitKey() involved :slight_smile: I’m measuring how long each operation takes using gettimeofday() calls around both reading the frames and encoding them in JPG (weird that’s required given that the output is MJPG).

I’m pretty sure it’s gstreamer, as I’m seeing some debug output as I start my app. I used capture.get(CAP_PROP_FOURCC) and confirmed the camera is set to output MJPG. As far as I can tell, I’ve checked everything.

ok then. try reading from the camera with a plain ffmpeg or gstreamer process from the terminal, without any opencv involved.

if either of that succeeds, it should be possible for opencv to do it too.

Double checked and ffmpeg uses v4l for interacting with the webcam, getting 40-45 FPS while doing encoding as well.

Input #0, video4linux2,v4l2, from '/dev/video2':
  Duration: N/A, start: 29580.442876, bitrate: N/A
    Stream #0:0: Video: mjpeg (Baseline), yuvj422p(pc, bt470bg/unknown/unknown), 1280x720, 120 fps, 120 tbr, 1000k tbn, 1000k tbc
Input #1, lavfi, from 'anullsrc':
  Duration: N/A, start: 0.000000, bitrate: 705 kb/s
    Stream #1:0: Audio: pcm_u8, 44100 Hz, stereo, u8, 705 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (mjpeg (native) -> h264 (libx264))
  Stream #1:0 -> #0:1 (pcm_u8 (native) -> aac (native))
Press [q] to stop, [?] for help
[libx264 @ 0xaaaadd0f7140] using cpu capabilities: ARMv8 NEON
[libx264 @ 0xaaaadd0f7140] profile High 4:2:2, level 3.1, 4:2:2, 8-bit
[libx264 @ 0xaaaadd0f7140] 264 - core 159 r2991M 1771b55 - H.264/MPEG-4 AVC codec - Copyleft 2003-2019 - http://www.videolan.org/x264.html - options: cabac=0 ref=1 deblock=0:0:0 analyse=0:0 me=dia subme=0 psy=1 psy_rd=1.00:0.00 mixed_ref=0 me_range=16 chroma_me=1 trellis=0 8x8dct=0 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=0 threads=9 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=0 weightp=0 keyint=250 keyint_min=25 scenecut=0 intra_refresh=0 rc=crf mbtree=0 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=0
Output #0, flv, to 'rtmp://a.rtmp.youtube.com/live2/a6ve-eh4g-v70c-6zwq-77c8':
  Metadata:
    encoder         : Lavf58.29.100
    Stream #0:0: Video: h264 (libx264) ([7][0][0][0] / 0x0007), yuvj422p(pc, progressive), 1280x720, q=-1--1, 30 fps, 1k tbn, 30 tbc
    Metadata:
      encoder         : Lavc58.54.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1
    Stream #0:1: Audio: aac (LC) ([10][0][0][0] / 0x000A), 44100 Hz, stereo, fltp, 128 kb/s
    Metadata:
      encoder         : Lavc58.54.100 aac
[video4linux2,v4l2 @ 0xaaaadd0ed7c0] Thread message queue blocking; consider raising the thread_queue_size option (current value: 8)
[flv @ 0xaaaadd0f5e50] Failed to update header with correct duration.377.1kbits/s speed=1.36x
[flv @ 0xaaaadd0f5e50] Failed to update header with correct filesize.
frame=  841 fps= 41 q=-1.0 Lsize=   21833kB time=00:00:28.00 bitrate=6387.4kbits/s speed=1.37x

The weird thing is that I checked and OpenCV uses V4L2 as well, as
capture.getBackendName() revealed.

I put up a post with the ffmpeg output, but was locked by the antispam filter… Essentially that also uses a V4L2 backend, but gets 40-45 FPS even while doing encoding and streaming.

Input #0, video4linux2,v4l2, from '/dev/video2':
  Duration: N/A, start: 29580.442876, bitrate: N/A
    Stream #0:0: Video: mjpeg (Baseline), yuvj422p(pc, bt470bg/unknown/unknown), 1280x720, 120 fps, 120 tbr, 1000k tbn, 1000k tbc
Input #1, lavfi, from 'anullsrc':
  Duration: N/A, start: 0.000000, bitrate: 705 kb/s
    Stream #1:0: Audio: pcm_u8, 44100 Hz, stereo, u8, 705 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (mjpeg (native) -> h264 (libx264))
  Stream #1:0 -> #0:1 (pcm_u8 (native) -> aac (native))
Press [q] to stop, [?] for help
[libx264 @ 0xaaaadd0f7140] using cpu capabilities: ARMv8 NEON
[libx264 @ 0xaaaadd0f7140] profile High 4:2:2, level 3.1, 4:2:2, 8-bit
[libx264 @ 0xaaaadd0f7140] 264 - core 159 r2991M 1771b55 - H.264/MPEG-4 AVC codec - Copyleft 2003-2019 - http://www.videolan.org/x264.html - options: cabac=0 ref=1 deblock=0:0:0 analyse=0:0 me=dia subme=0 psy=1 psy_rd=1.00:0.00 mixed_ref=0 me_range=16 chroma_me=1 trellis=0 8x8dct=0 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=0 threads=9 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=0 weightp=0 keyint=250 keyint_min=25 scenecut=0 intra_refresh=0 rc=crf mbtree=0 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=0
Output #0, flv, to 'rtmp://a.rtmp.youtube.com/live2/a6ve-eh4g-v70c-6zwq-77c8':
  Metadata:
    encoder         : Lavf58.29.100
    Stream #0:0: Video: h264 (libx264) ([7][0][0][0] / 0x0007), yuvj422p(pc, progressive), 1280x720, q=-1--1, 30 fps, 1k tbn, 30 tbc
    Metadata:
      encoder         : Lavc58.54.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1
    Stream #0:1: Audio: aac (LC) ([10][0][0][0] / 0x000A), 44100 Hz, stereo, fltp, 128 kb/s
    Metadata:
      encoder         : Lavc58.54.100 aac
[video4linux2,v4l2 @ 0xaaaadd0ed7c0] Thread message queue blocking; consider raising the thread_queue_size option (current value: 8)
[flv @ 0xaaaadd0f5e50] Failed to update header with correct duration.377.1kbits/s speed=1.36x
[flv @ 0xaaaadd0f5e50] Failed to update header with correct filesize.
frame=  841 fps= 41 q=-1.0 Lsize=   21833kB time=00:00:28.00 bitrate=6387.4kbits/s speed=1.37x

OpenCV also uses V4L2, as shown by the capture.getBackendName() output.

Any ideas about how to approach this? I find it odd that both OpenCV and FFMpeg would use the same underlying module (V4L) for accessing the webcam with such different results.

Is it possible that they’d use different version of the Video4Linux module?

You probably need to do multithreading. Try to just capture the frames, without any processing of the frame, to check if you can read the frames from the webcam as fast as you expect. If this works, move the processing of the frames to another/other threads.