Lot of Delay with my RTSP cam with OpenCV on Python

I had a similar problem. I switched to cpp and it works fine. After reading a lot I think it may have something to do with pipes in python.

it really doesnā€™t. nothing here involves pipes and thereā€™s no reason to suspect python.

Whilst not due to python there could have been a small delay caused if Nahum_Nir did not pass the image as an argument. i.e. calling

retval, image = cv.VideoCapture.read()

instead of

retval, _= cv.VideoCapture.read( image )

with a high resolution stream. I have seen over a 50% increase in execution time on 4K h264 due to the allocation of image on every call when calling cv.VideoCapture.read() using the former approach.

3 Likes

@crackwitz
ip cam :
HIKVISION DS-2CD2023G0-I(2.8MM) , 1920x1080 @ 30fps , H.265 +, H.265, H.264 +, H.264
webcam : AUKEY webcam 1080p 30fps

@cudawarped It Only work on my webcam. It work too with my ip cam but with a lot of delay. I think that it work correctly like with 30 fps.

I have seen somewhere that a videocapture::grab can decrease my delay
could this be a way to investigate or not at all?

I find a solution too where i can use Gstreamer instead of VideoCapture. But itā€™s really different. Is it a good solution ?

To clarify are you saying that:

  1. You can run the dnn on your webcam without any delay. In that case have you confirmed that you are processing at 30fps by calculating the number of processed frames every second? Without checking this everything else is meaningless.
  2. You specified delay in the title but then talked about frame loss, is the problem delay, frame loss or both?
  3. Do you get delay on your IP camera when you donā€™t use the dnn?
  4. Do you get frame loss on your IP camera when you donā€™t use the dnn?
  5. What is the measured fps on your IP camera with and without the dnn?

Iā€™m asking because Iā€™m lost as to what your problem is. Streaming from RTSP and a webcam behave differently but I canā€™t think of a reason why you canā€™t get the same performance from both.

Regarding delay:

  1. RTSP streams with FFmpeg in my experience always have a delay of maybe 1 second. Probably due to TCP/reordering etc.
  2. If you call
cap = cv2.VideoCapture('rtsp://[username]:[password]@[IP]:554/Streaming/Channels/1/')

and then perform some initialization you will see a significant delay on certain IP cameras. This is because the camera can que up the frames and you have to call VideoCapture.read() to drain the que before you get to the current frame. As long as you donā€™t have any significant delay in

cv2.waitKey(1)

this shouldnā€™t last long. However if your dnn takes a while you may never drain the que.

1 Like

1 - Yeah i can say that because in my code i have process where i can see the fps in real time. itā€™s between 20-30 fps.
2- When i use the code with an IP cam at the begining i can see that i have like 7-9 fps. When i wait a bit i can see that the video freez like one second. I can see the delay increase like in 2 minutes of streaming i have a delay of 20 seconds sometimes itā€™s about 40 seconds. and sometimes i got some enconding error (you can see the error in a previous post). So i think itā€™s both.
3/4 - Without dnn i have any problem. Streaming is fluid. any delay and any frame loss
5- The fps is the same of my webcam without dnn.

1 second of delay is not a problem for me. But if my stream is freezing all the time itā€™s not usable for my project. i tried too up the cv2.waitKey(1) but i have any changes.

Thatā€™s very strange.

If we try to calculate the maximum frame time for the combination of your ip camera with the dnn using the worst case scenario:

  1. Assume 20fps for your webcam when processing its frames through the dnn and assuming that the decoding of your frame takes 0ms gives a dnn time of 50ms/frame (1000/20).
  2. Assume your CPU is working 100% to decode each frame it receives from your IP camera gives a decoding time of 33.33ms/frame (1000/30).
  3. The combination of these two is the maximum possible time for OpenCV to decode a frame and pass it through the the dnn. This is 88.33ms/frame (50+33.33).

Therefore you should be seeing an fps of ~12. That is a worst case scenario so it should be way above 7-9fps can you check this?
What CPU are you using and can you check the CPU usage when decoding from your ip camera without the dnn to see if it is near to 100%?

If it is the decoding your options are limited (change the resolution) unless you have access to a GPU? Are you using Windows or Linux?

Ok i was wrong about the 7-9 fps iā€™m sorry itā€™s not all the time. I would say that the avarage of fps is about 17 fps. but itā€™s not regular. I have drops in fps that go from 18 fps to 12fps in my last test. Itā€™s pretty variable because sometimes i got some falls of 9 fps Itā€™s really strange because iā€™m sure that itā€™s very slow. I had a freeze too and when the stream resmued this error falls :

[h264 @ 00000254348000a0] left block unavailable for requested intra mode
[h264 @ 00000254348000a0] error while decoding MB 0 42, bytestream 667
FPS: 16.945313510019393
[NULL @ 0000025434533600] sps_id 1 out of range
[h264 @ 0000025434d24600] Invalid NAL unit 1, skipping.

My CPU : AMD Ryzen 5 1500X Quad-Core Processor
Sure below the tests :

With the DNN i can see 60% of my CPU usage
Without DNN : 3% of my CPU usage

I have two environment :

  • My Computer : Windows 10 with the Ryzen CPU and a Nvidia Geforce 1070 GTX
  • And a Jetson Xavier with an Ubuntu 18.04.

I tried on both of them and i have the same issues. The code should normally be used in production on the Jetson XAVIER.

Which set up are you referring to in this post. That is on which environment do you get 20fps on the webcam with dnn and 12-17fps on the ip cam?

I would guess 12fps could be iframe etc. or more dnn processing if you are using a cnn with a variable amount of processing, e.g. r-cnn etc.

Are you using the GPU for the dnn with 60% CPU usage?

Everything still points to the original problem, (see the post by crackwitz for the solution) that you are not requesting frames at the rate they are being produced at. From the numbers it looks like the web cam decoding uses slightly less processing than the IP cam but apart from that you are getting similar fps. When you use the webcam this is just not as obvious.

You said that you need to process every frame but I would suggest that this isnā€™t possible on the Jetson if you can only run the dnn @20fps on your 1070GTX.

note on english language: the word ā€œanyā€ alone only occurs in questions. negation in a statement is indicated by ā€œnot any Xā€ or ā€œno Xā€.

so, if you have no problem without dnn, you donā€™t have any problem without dnn.

ā€œany delay and any frame lossā€ is read as a question with an implied question mark.

if you want to state that you have no delay and no frame loss, you say it like I just did.

Hello @crackwitz thx for your answer.

I change a bit my project and i will finally use a usb camera. It work correctly soā€¦

I donā€™t know if i need to create a new topic but i have a question :

Does anyone have a site to share on which I could find a dnn allowing the detection of a license plate for blurring?

Thanks in advance.