Inconsistent Frame Interval with 60fps VideoCapture using CSI Camera on NVIDIA Jetson Orin Nano

Python
, , ,
1

Hello everyone,

I am currently working on a project where I am using an NVIDIA Jetson Orin Nano with a Raspberry Pi 2 camera module connected via CSI interface. I have configured the camera to capture video at 1280x720 resolution at 60fps using OpenCV.

To verify that I am receiving frames at the correct rate, I am measuring the time interval between frames using cv2.VideoCapture.read() and Python’s time module. I expected to get a consistent 16ms interval between frames since the camera is set to 60fps. However, I am observing inconsistent intervals such as 25ms, 9ms, and other varying values.

Here is a simplified version of my code:

# MIT License
# Copyright (c) 2019-2022 JetsonHacks

# Using a CSI camera (such as the Raspberry Pi Version 2) connected to a
# NVIDIA Jetson Nano Developer Kit using OpenCV
# Drivers for the camera and OpenCV are included in the base image

import cv2
import time

""" 
gstreamer_pipeline returns a GStreamer pipeline for capturing from the CSI camera
Flip the image by setting the flip_method (most common values: 0 and 2)
display_width and display_height determine the size of each camera pane in the window on the screen
Default 1920x1080 displayd in a 1/4 size window
"""

def gstreamer_pipeline(
    sensor_id=0,
    capture_width=1280,
    capture_height=720,
    display_width=960,
    display_height=540,
    framerate=60,
    flip_method=0,
):
    return (
        "nvarguscamerasrc sensor-id=%d ! "
        "video/x-raw(memory:NVMM), width=(int)%d, height=(int)%d, framerate=(fraction)%d/1 ! "
        "nvvidconv flip-method=%d ! "
        "video/x-raw, width=(int)%d, height=(int)%d, format=(string)BGRx ! "
        "videoconvert ! "
        "video/x-raw, format=(string)BGR ! appsink"
        % (
            sensor_id,
            capture_width,
            capture_height,
            framerate,
            flip_method,
            display_width,
            display_height,
        )
    )


def show_camera():
    window_title = "CSI Camera"

    print(gstreamer_pipeline(flip_method=2))
    video_capture = cv2.VideoCapture(gstreamer_pipeline(flip_method=0), cv2.CAP_GSTREAMER)
    prev_time = 0
    if video_capture.isOpened():
        try:
            while True:
                ret, frame = video_capture.read()

                current_time = time.time() - prev_time
                print(current_time)
                prev_time = time.time()

                cv2.imshow(window_title, frame)

        finally:
            video_capture.release()
            cv2.destroyAllWindows()
    else:
        print("Error: Unable to open camera")


if __name__ == "__main__":
    show_camera()

My questions are as follows:

1. Is it normal to experience such varying intervals even when the camera is set to 60fps?
2. Could this be due to hardware limitations or the way the Jetson Orin Nano handles CSI camera inputs?
3. Are there any known solutions or best practices to achieve a more consistent frame interval at 60fps?
4. Would using another method or API for capturing frames yield better results?

I would greatly appreciate any insights or suggestions that could help in diagnosing and resolving this issue.

Thank you for your assistance!

2

What’s the accuracy of your timer ±10ms?

3

Hello,

I have tested the frame interval measurement using both time.time() and time.perf_counter() in my application, where I am capturing video from a CSI camera (Raspberry Pi Camera Version 2) connected to an NVIDIA Jetson Orin Nano. Despite switching to time.perf_counter() for more precise timing, I still observe significant variations in the time intervals between frames.

For instance, when capturing at 60fps, the time intervals between frames should ideally be around 16.7ms. However, the measured intervals vary significantly, such as 4ms, 17ms, 21ms, and so on. This suggests that the issue may not be related to the accuracy of the timer itself.

Given these results, I am wondering if the inconsistent frame intervals could be due to other factors, such as:

  • Potential issues with the CSI camera or its driver.
  • Resource contention or CPU load on the Jetson Orin Nano affecting frame capture consistency.
  • GStreamer pipeline delays or buffering issues.
  • The camera not consistently providing frames at the expected rate.

Could these factors be contributing to the observed variability? Are there any known issues or recommended troubleshooting steps for ensuring consistent frame capture intervals on this hardware setup?

Any insights or suggestions would be greatly appreciated.

Thank you for your assistance!

4

time.time() has coarse resolution on some systems. use time.perf_counter() even if it happens to not make a difference in your situation, because if you keep using time.time() people will keep questioning that. it’s not made for that.

why doesn’t your code come with any kind of waitKey? without waitKey(), the imshow() will be pointless.

what is that imshow() doing there in the first place? you might have just measured video capture AND GUI stuff together, which can in some cases be impacted by any other GUI in the system. you can no longer be sure that you’re only measuring what you want. rip all of that out.

python is a garbage-collected language. don’t be surprised by occasional moments of GC. DO NOT mess with python’s GC. do not. any touching of the GC stuff usually does nothing. people are most likely to just use that hoping it’ll do anything, or it won’t actually harm them.

if it happens that Python’s GC is causing anything you can see with the naked eye or quantify with measurement equipment, try the same thing in a compiled language with different memory management (C, C++). if that improves the situation, go with that. if it doesn’t, Python is probably not to blame.

the way you’re accessing the hardware might cause such jitter. or it might not. you should try other means and compare. find out how gstreamer does it. see if you can use the same API from your own code. see if you can use another API from your own code.

5

Hello, @crackwitz

Thank you for your previous suggestions. First of all, I want to clarify that I have not made any modifications related to Python’s garbage collection (GC). To address the issue further, I transitioned from Python to C to better understand the problem.

However, even after switching to C, I’m still observing that the frames are often coming in faster than expected, resulting in an effective frame rate of around 80fps instead of the desired 60fps.

In my GStreamer pipeline, I’ve explicitly set the capture to 1280x720 resolution at 60fps, as shown below:

std::string pipeline = gstreamer_pipeline(1280, 720, 1280, 720, 60, 0);

Given this configuration, I expected the frames to be captured at exactly 60fps, but that doesn’t seem to be happening. I understand that I can drop frames or use a timer to artificially limit the frame rate to 60fps, but my question is more about why the camera is not adhering to the 60fps setting in the first place.

Could there be any underlying reason or configuration in GStreamer or the camera driver that might cause the frames to come in faster than the specified 60fps? Is there anything specific I should check or modify to ensure the camera operates strictly at the set 60fps?

Any insights would be greatly appreciated.

Thank you!

6

My first guess would be buffering and/or scheduling.

If you write a simple program that does nothing other than read the frames and discard them, do you end up with a long-term average of 16.66 msec per frame? In other words, are you truly getting a higher FPS, or just that some frames arrive earlier than you expect, but others arrive later than you expect?

One trick is to point the camera at a timer (I use the stop watch on my phone) and grab a sequence of images into memory, then save them to file and inspect. (Don’t save as you capture because of the computational and I/O load that could affect timings). Are the images being caputred at the 16.66 msec interval as expected, or do you see jitter in the images that correlates to the jitter in the times you are receiving them?

7

calculating that takes some thought.

say you expect a 2 fps average, and frames show up after 100 and 900 ms in turn (yes yes, trying to make a point). that would be 2 fps. however, if you simply averaged the “1/0.1 = 10 fps” and “1/0.9 = 1.11 fps” figures, you’d get 5.55 fps.

you need to average the intervals, not the rates.