Cannot read RTSP Stream with less than 1 FPS

Python
1

Hi, I am trying to read a RTSP Stream from either an IP Camera (I am using HIKVISION) or from a streaming server (example:
RTSP Simple Server ) using this syntax:
cap = cv2.VideoCapture(‘rtsp://{rtsp-src}’, cv2.CAP_FFMPEG)

This works fine on both sources, but when I set the IP Camera FPS or Stream Server FPS through the vendor app (not from OpenCV) to value less than 1. OpenCV cannot read the stream. It often produces the error message like this:

[h264 @ 000002689cfbe1c0] missing picture in access unit with size 22
[h264 @ 000002689cfbe1c0] no frame!

or this

[h264 @ 0x5602ded1bcc0] error while decoding MB 55 121, bytestream -15

Surprisingly, it can read the Stream on Low-end PC and Embedded Devices like Nvidia Jetson, with different OS (Windows and Ubuntu) and OpenCV version (I have tested 4.0.0, 4.1.1, 4.5.5).

I have also noticed some strange patterns:

  • The time it takes to complete this line of code cap = cv2.VideoCapture(‘rtsp://{rtsp-src}’, cv2.CAP_FFMPEG) increases when the FPS is lower (10 FPS is 1 sec, 1 FPS is 7 secs, and 0.25 FPS is 30secs).

  • The working devices have less than 8 CPU Threads. The non working devices have 12 and 56 threads. They also have 64GB RAM.

  • In low FPS Setting, I try to shut down the stream when OpenCV is capturing the input. It outputs several frames at once before exiting the program.

My goal is to get it working on the Ubuntu OS and High-end PC or server. Any advice or suggestions appreciated.

2

On the machine which it doesn’t work on, does it work with the ffmpeg application? How often do you send a key frame? Is the delay always present even after streaming for a while, I am asking because I wonder if the key frame is being sent every say 5 frames and for some reason the first key frame is missed, so you have to wait for the next one?

3

Hi, I tested and FFMPEG works with both types of stream. But it has some error messages too, I will attach the logs.

How often do you send a key frame?

In the camera setting, I see there is I Frame Interval config, is this keyframe you mention? I tried 1 and 15, but it didn’t work.

Thank for your response.

4
FFMPEG IPCAMERA LOG 


ffmpeg version 4.2.2 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 7.3.0 (crosstool-NG 1.23.0.449-a04d0)
  configuration: --prefix=/home/hoa/.conda/envs/OCV --cc=/tmp/build/80754af9/ffmpeg_1587154242452/_build_env/bin/x86_64-conda_cos6-linux-gnu-cc --disable-doc --enable-avresample --enable-gmp --enable-hardcoded-tables --enable-libfreetype --enable-libvpx --enable-pthreads --enable-libopus --enable-postproc --enable-pic --enable-pthreads --enable-shared --enable-static --enable-version3 --enable-zlib --enable-libmp3lame --disable-nonfree --enable-gpl --enable-gnutls --disable-openssl --enable-libopenh264 --enable-libx264
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
[tcp @ 0x55957e9c1ec0] Starting connection attempt to 1.2.3.5 port 554
[tcp @ 0x55957e9c1ec0] Successfully connected to 1.2.3.5 port 554
[rtsp @ 0x55957e9bfbc0] SDP:
v=0
o=- 1657556507010180 1657556507010180 IN IP4 1.2.3.5
s=Media Presentation
e=NONE
b=AS:5050
t=0 0
a=control:rtsp://1.2.3.5:554/
m=video 0 RTP/AVP 96
c=IN IP4 0.0.0.0
b=AS:5000
a=recvonly
a=x-dimensions:3840,2160
a=control:rtsp://1.2.3.5:554/trackID=1
a=rtpmap:96 H264/90000
a=fmtp:96 profile-level-id=420029; packetization-mode=1; sprop-parameter-sets=Z00AM4qKUB4AIff+AAIAAtgI,aO48gA==
a=Media_header:MEDIAINFO=494D4B48010300000400000100000000000000000000000000000000000000000000000000000000;
a=appversion:1.0

[rtsp @ 0x55957e9bfbc0] setting jitter buffer size to 500
[h264 @ 0x55957e9c3880] Reinit context to 3840x2160, pix_fmt: yuvj420p
[rtsp @ 0x55957e9bfbc0] max delay reached. need to consume packet
[rtsp @ 0x55957e9bfbc0] RTP: missed 396 packets
[h264 @ 0x55957e9c3880] error while decoding MB 79 22, bytestream -9
[h264 @ 0x55957e9c3880] concealing 27090 DC, 27090 AC, 27090 MV errors in I frame
[rtsp @ 0x55957e9bfbc0] max delay reached. need to consume packet
[rtsp @ 0x55957e9bfbc0] RTP: missed 227 packets
[h264 @ 0x55957e9c3880] concealing 26169 DC, 26169 AC, 26169 MV errors in I frame
[rtsp @ 0x55957e9bfbc0] max delay reached. need to consume packet
[rtsp @ 0x55957e9bfbc0] RTP: missed 147 packets
[h264 @ 0x55957e9c3880] cabac decode of qscale diff failed at 28 25
[h264 @ 0x55957e9c3880] error while decoding MB 28 25, bytestream 240819
[h264 @ 0x55957e9c3880] concealing 26421 DC, 26421 AC, 26421 MV errors in I frame
[rtsp @ 0x55957e9bfbc0] max delay reached. need to consume packet
[rtsp @ 0x55957e9bfbc0] RTP: missed 209 packets
[h264 @ 0x55957e9c3880] left block unavailable for requested intra mode
[h264 @ 0x55957e9c3880] error while decoding MB 0 26, bytestream 150346
[h264 @ 0x55957e9c3880] concealing 26209 DC, 26209 AC, 26209 MV errors in I frame
[rtsp @ 0x55957e9bfbc0] max delay reached. need to consume packet
[rtsp @ 0x55957e9bfbc0] RTP: missed 153 packets
[h264 @ 0x55957e9c3880] cabac decode of qscale diff failed at 146 24
[h264 @ 0x55957e9c3880] error while decoding MB 146 24, bytestream 235832
[h264 @ 0x55957e9c3880] concealing 26543 DC, 26543 AC, 26543 MV errors in I frame
[rtsp @ 0x55957e9bfbc0] max delay reached. need to consume packet
[rtsp @ 0x55957e9bfbc0] RTP: missed 159 packets
Input #0, rtsp, from 'rtsp://username:password@1.2.3.5:554':
  Metadata:
    title           : Media Presentation
  Duration: N/A, start: 0.079000, bitrate: N/A
    Stream #0:0: Video: h264 (Main), 1 reference frame, yuvj420p(pc, progressive, left), 3840x2160 [SAR 1:1 DAR 16:9], 1 tbr, 90k tbn, 180k tbc
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
[h264 @ 0x55957ea030c0] Reinit context to 3840x2160, pix_fmt: yuvj420p
[h264 @ 0x559580c40440] error while decoding MB 79 22, bytestream -9
[h264 @ 0x559580c40440] concealing 27090 DC, 27090 AC, 27090 MV errors in I frame
[h264 @ 0x55957eceef40] concealing 26169 DC, 26169 AC, 26169 MV errors in I frame
[h264 @ 0x55957eccbcc0] cabac decode of qscale diff failed at 28 25
[h264 @ 0x55957eccbcc0] error while decoding MB 28 25, bytestream 240819
[h264 @ 0x55957eccbcc0] concealing 26421 DC, 26421 AC, 26421 MV errors in I frame
[h264 @ 0x55957ecd2400] left block unavailable for requested intra mode
[h264 @ 0x55957ecd2400] error while decoding MB 0 26, bytestream 150346
[h264 @ 0x55957ecd2400] concealing 26209 DC, 26209 AC, 26209 MV errors in I frame
[h264 @ 0x55957ecb3100] cabac decode of qscale diff failed at 146 24
[h264 @ 0x55957ecb3100] error while decoding MB 146 24, bytestream 235832
[h264 @ 0x55957ecb3100] concealing 26543 DC, 26543 AC, 26543 MV errors in I frame
[h264 @ 0x55957ed19000] concealing 26406 DC, 26406 AC, 26406 MV errors in I frame
[rtsp @ 0x55957e9bfbc0] max delay reached. need to consume packet
[rtsp @ 0x55957e9bfbc0] RTP: missed 313 packets
[h264 @ 0x559580a76ec0] error while decoding MB 49 25, bytestream -11
[h264 @ 0x559580a76ec0] concealing 26400 DC, 26400 AC, 26400 MV errors in I frame
[graph 0 input from stream 0:0 @ 0x559580a3d540] w:3840 h:2160 pixfmt:yuvj420p tb:1/90000 fr:1/1 sar:1/1 sws_param:flags=2
Output #0, null, to 'pipe:':
  Metadata:
    title           : Media Presentation
    encoder         : Lavf58.29.100
    Stream #0:0: Video: wrapped_avframe, 1 reference frame, yuvj420p(left), 3840x2160 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 1 fps, 1 tbn, 1 tbc
    Metadata:
      encoder         : Lavc58.54.100 wrapped_avframe
rtsp://username:password@1.2.3.5:554: corrupt decoded frame in stream 025x    
rtsp://username:password@1.2.3.5:554: corrupt decoded frame in stream 013x     
rtsp://username:password@1.2.3.5:554: corrupt decoded frame in stream 051x    
rtsp://username:password@1.2.3.5:554: corrupt decoded frame in stream 042x    
rtsp://username:password@1.2.3.5:554: corrupt decoded frame in stream 008x    
rtsp://username:password@1.2.3.5:554: corrupt decoded frame in stream 057x    
frame=    8 fps=0.2 q=-0.0 Lsize=N/A time=00:00:25.00 bitrate=N/A speed=0.626x    
video:4kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Input file #0 (rtsp://username:password@1.2.3.5:554):
  Input stream #0:0 (video): 23 packets read (10371722 bytes); 9 frames decoded; 
  Total: 23 packets (10371722 bytes) demuxed
Output file #0 (pipe:):
  Output stream #0:0 (video): 8 frames encoded; 8 packets muxed (4288 bytes); 
  Total: 8 packets (4288 bytes) muxed
FFMPEG RTSP SERVER LOG 
ffmpeg version 4.2.2 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 7.3.0 (crosstool-NG 1.23.0.449-a04d0)
  configuration: --prefix=/home/hoa/.conda/envs/OCV --cc=/tmp/build/80754af9/ffmpeg_1587154242452/_build_env/bin/x86_64-conda_cos6-linux-gnu-cc --disable-doc --enable-avresample --enable-gmp --enable-hardcoded-tables --enable-libfreetype --enable-libvpx --enable-pthreads --enable-libopus --enable-postproc --enable-pic --enable-pthreads --enable-shared --enable-static --enable-version3 --enable-zlib --enable-libmp3lame --disable-nonfree --enable-gpl --enable-gnutls --disable-openssl --enable-libopenh264 --enable-libx264
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
[tcp @ 0x557472023ec0] Starting connection attempt to 1.2.3.4 port 8554
[tcp @ 0x557472023ec0] Successfully connected to 1.2.3.4 port 8554
[rtsp @ 0x557472021bc0] SDP:
v=0
o=- 0 0 IN IP4 127.0.0.1
s=Stream
c=IN IP4 0.0.0.0
t=0 0
m=video 0 RTP/AVP 96
a=rtpmap:96 H264/90000
a=fmtp:96 packetization-mode=1; sprop-parameter-sets=Z00AM4qKUB4AIff+AAIAAtgI,aO48gA==; profile-level-id=4D0033
a=control:trackID=0

[rtsp @ 0x557472021bc0] method SETUP failed: 461 Unsupported Transport
[rtsp @ 0x557472021bc0] setting jitter buffer size to 0
[h264 @ 0x5574720255c0] Reinit context to 3840x2160, pix_fmt: yuvj420p
Input #0, rtsp, from 'rtsp://1.2.3.4:8554/stream0':
  Metadata:
    title           : Stream
  Duration: N/A, start: 3.909567, bitrate: N/A
    Stream #0:0: Video: h264 (Main), 1 reference frame, yuvj420p(pc, progressive, left), 3840x2160 [SAR 1:1 DAR 16:9], 1 tbr, 90k tbn, 180k tbc
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
[h264 @ 0x5574720444c0] Reinit context to 3840x2160, pix_fmt: yuvj420p
[graph 0 input from stream 0:0 @ 0x5574747e7d00] w:3840 h:2160 pixfmt:yuvj420p tb:1/90000 fr:1/1 sar:1/1 sws_param:flags=2
Output #0, null, to 'pipe:':
  Metadata:
    title           : Stream
    encoder         : Lavf58.29.100
    Stream #0:0: Video: wrapped_avframe, 1 reference frame, yuvj420p(left), 3840x2160 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 1 fps, 1 tbn, 1 tbc
    Metadata:
      encoder         : Lavc58.54.100 wrapped_avframe
frame=    9 fps=0.2 q=-0.0 Lsize=N/A time=00:00:29.00 bitrate=N/A speed=0.519x    
video:5kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Input file #0 (rtsp://1.2.3.4:8554/stream0):
  Input stream #0:0 (video): 24 packets read (12696414 bytes); 10 frames decoded; 
  Total: 24 packets (12696414 bytes) demuxed
Output file #0 (pipe:):
  Output stream #0:0 (video): 9 frames encoded; 9 packets muxed (4824 bytes); 
  Total: 9 packets (4824 bytes) muxed
5

Is the first log streaming from the hik camera and the second from an rtsp server with the same ip address?

6

Yes, the first one is HIK camera. I run ffmpeg and python code on the same machine that hosts RTSP server.

7

OK so you don’t get any errors from the RTSP server and you get mostly errors when streaming from the IP camera.

Therefore I would start investigating why you can’t stream from the RTSP server at 1 fps with OpenCV when you can with FFMpeg. There is no point in trying to stream from the IP camera with OpenCV when you can’t with FFMpeg. Which streaming server are you using, how are you setting 1fps?

8

I am using RTSP Simple Server as a proxy server to read streams from other IP Cameras. So the FPS of the camera will be the same as the FPS of the stream I read from RTSP server. I only need to set FPS on the camera, which is fairly simple.

9

Are you streaming directly from the hik camera or through a proxy server when you have issues?

When you say RTSP server do you mean a seperate RTSP (live555 etc.) server or the proxy?

10

I mean reading from proxy server, sorry for the confusion. In my setup, all the cameras will stream to the proxy server, and the application will use OpenCV Python to read and process streams from the proxy server only.
At first, I thought the proxy server was the problem, so I experimented with reading stream directly from camera. But it turned out that only high-end PCs had this issue with both types of stream (< 1 FPS of course, no problem with higher FPS), when other devices worked just fine.

The problem is just confusing and strange.

11

OK so when you set 1fps on the IP camera and read from it with the FFMpeg application you get a lot of errors?

In the other log FFMPEG RTSP SERVER LOG you didn’t get any errors when using the FFMpeg application. Do you get errors in this case when using OpenCV?

12

OpenCV outputs one of the two error messages and cannot read both type of streams. Although FFMPEG also outputs the errors, it can read two streams afterward. I read that there is some “probing” at the start of capturing the stream, maybe FFMPEG passes that phase, but OpenCV does not?

13

Update: Maybe my issue is similar to this one:
https://github.com/opencv/opencv/issues/20002

It is related to the number of decoding threads ffmpeg uses.

2 Likes
14

It makes sense, I’ll be intereseted to find out the results you get when you apply the fix.

1 Like
15

I modified the source code to set the FFMPEG threads count to 1, and it worked perfectly ! However, it may not work well with other case, setting the value through environment variable will be more appropriate.

1 Like