VideoCapture get() returns old frame after long periods of time using 3 cameras

Python
1

I’m using OpenCV 4.4 in Python 3.6.9.

Problem:
I have written a program that will take frames from 3 cameras and save to disk. The program alternates between each camera in a round robin fashion on an interval of X seconds. So every X seconds, a new image is saved to disk taken from camera (Y + 1) % 3.

This seems fine for at least 1 hour. If I use imshow, I can see that the latest frame saved is about 3-4s behind real time which is fine. However after leaving this program running over the weekend, I see that the latest images saved are old frames from 2 days ago just 4-5 hours after I started the program. I run an image diff (pixel by pixel) on the latest images and images saved 2 days ago and they appear to be the exact same frame. Given the the number of frames saved to disk over 3 days, its hard to tell when exactly this issue arises.

Code:
I use gstreamer with 3 cameras and connect to each camera by initializing 3 VideoCapture objects:

gst_str = f"""nvarguscamerasrc sensor_id={file_cap_id} !
          video/x-raw(memory:NVMM), width={self.frame_width}, height={self.frame_height},
          format=(string)NV12, framerate=30/1 ! nvvidconv flip-method={flip_method} !
          video/x-raw, width={self.frame_width}, height={self.frame_height},
          format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR
          ! appsink"""
cap = cv2.VideoCapture(gst_str, cv2.CAP_GSTREAMER)
# ... some code later
self.cameras.append(cap)

I startup a separate thread to always get the latest frames from each camera’s buffer. I do this because frames saved to disk should be on an interval of X seconds and we need these frames to be the latest:

  def read_frames(self):
    ''' Function to read frames forever '''
    while True:
      num_cameras = len(self.cameras)
      with self.frame_lock:
        for i in range(num_cameras):
          curr_cap = self.cameras[i]
          ret, image = curr_cap.read()
          self.frames[i] = copy.deepcopy(image)
      # Show frame if enabled.
      if self.args['show_frame']: 
        self._show_frame(self.frames[0])
      # Wait key to keep real-time.
      key = cv2.waitKey(1)

And I process these frames in the main thread:

  def process_frames(self):
    frame_idx = -1
    while True:
      frame_idx = (frame_idx + 1) % len(self.cameras)
      with self.frame_lock:      
        orig_image = copy.deepcopy(self.frames[frame_idx])
      if orig_image is None or len(orig_image) is 0:
        continue
      self.save_frame(frame_idx, orig_image)

Eg.:

  def run(self):
    ''' Main run function '''
    self._thread = Thread(target=self.read_frames)
    self._thread.start()
    self.process_frames()

I suspect this might have to do with trying to use 3 cameras. Best I can think up of what is happening is there is some kind of frame “drift” and the camera buffer gets too large?

Edit: I’ll update this post when I can test 1 camera. Testing for this takes awhile given the nature of the problem.

2

no, this has to do with forcefully throttling those cameras.

you absolutely must read every frame a camera generates. do not delay. you can discard them after you’ve read them, but you must read them.

you can run a thread per camera and keep the latest frame around. or you can use a somewhat new API in OpenCV that’s called waitAny, which is unfortunately not available to python (yet)

1 Like
3

Hey, thanks for the reply.

When you say “forcefully throttling” are you referring to the 3 cameras being read from in a single separate thread from main? I suppose I’m confused how a thread per camera instead of a thread for 3 cameras would differ? Because in both instances as far as I can tell we are reading from the cameras as fast as possible.

4

no. I mean you’re reading a frame whenever you feel like it, while the camera produces frames at its fixed frame rate. they queue up and will eventually be discarded by the driver, but that alone causes the 3-4 seconds of latency (stale frames in the queue) and drivers may decide that you’re abusing them and simply throw an error, if you do that.

and then, cameras aren’t perfect, nothing is. their timekeeping may run at a few parts per million differently from other cameras. if you read one frame from each camera, in lockstep, eventually one will be clearly the slowest, and all the others are queueing up frames, that you don’t read in time, …

2 Likes
5

I see, this makes sense. So letting Python decide and switch threads when the current thread is possibly waiting for a frame removes the throttling issue.

I also tested this:

cap.set(cv2.CAP_PROP_BUFFERSIZE, 1)

For each VideoCapture and after testing for two days it looks like the issue is gone. I’m even doing a lot of processing with PyTorch now while grabbing these frames. But I’m going to go ahead and implement your suggestion too.

6

Hey, have an update with the issue persisting. After a long period of time, the frame retrieved from the camera will be from hours before.

I’ve changed the code so I start a thread per camera. I deep copy the newest frame from each camera to a list of frames. Each element in this list is protected with a unique lock:

  def read_from_camera(self, idx):
    ''' Three threads are created and started running this function '''
    cap = self.cameras[idx]
    image = None
    ret = False
    while True:
      with self.frame_locks[idx]:
        if ret:
          self.frames[idx] = copy.deepcopy(image)
      ret, image = cap.read()
      key = cv2.waitKey(1)
    return

And in the main thread I process the frame:

  def process_frames(self):
    frame_idx = -1
    while True:
      frame_idx = (frame_idx + 1) % len(self.cameras)
      with self.frame_locks[frame_idx]:      
        orig_image = copy.deepcopy(self.frames[frame_idx])
      if orig_image is None or len(orig_image) is 0:
        continue
      # Save frame to disk.
      filename, filepath = self.save_frame(frame_idx, orig_image)
      # Sleep
      time.sleep(2)

Also, my previous post solution resulted in an eventual seg fault. I must still be missing something?

7

you don’t need a “deepcopy” of numpy arrays. whatever you get from VideoCapture.read is a new array/matrix object (unless you passed one in…) and it’s all refcounted and garbage-collected and you are only ever reading that data, not changing it (and even then, no trouble).

you don’t need locking if you use python’s queue (from the queue module). it’s already thread-safe.

8

The problem remains. I often see 1 camera is frozen on a frame while the others return realtime frames.

I’ve tried queues, a list of frames where each element corresponds to the latest frame from a camera, removing locks and deep copies, and I’ve tried removing time.sleep from the main thread (instead counting elapsed time to decide whether to save the frame to disk or not).

I’ve also tried the solution at this link with 3 threads (1 per camera). This results in the same behavior.

Maybe this is a gstreamer issue and I should refer to the Nvidia forums?

Edit - I’ve done some more testing. When the repeated frame occurs, I believe cap.read is hanging or blocking indefinitely and not retrieving anything now, because I cannot get the thread to stop by updating a ‘stopped’ variable that is checked in the camera thread’s loop. This would explain the ‘repeated’ frames because the thread would never update the frame.

9

throw a ton of logging output (print) in your code. prefix each line with a timestamp (wall clock time or time.perf_counter())

because it’s a bunch of threads, make a “logging function” that acquires and releases a lock before writing, so you get whole lines with carriage return, instead of some interleaved mess.

10

Unfortunately Only available for V4l2

1 Like