VideoCapture not working with Gstreamer pipeline

Python
,
1

I have written a function to display two stereo cameras with different camera ids. I am later feeding those id’s into videocapture function. I can see that one camera is live-streaming and says both camera’s are open but rest of the program is not working. Can someone please direct me what I am missing here? Here is my program:

import cv2
import gi
gi.require_version("Gst", "1.0")
from gi.repository import Gst, GLib

#Initialize GStreamer
Gst.init(None)

def display_camera(camera_id):
  pipeline = Gst.Pipeline.new("video-display")

  # create element
  source = Gst.ElementFactory.make("qtiqmmfsrc", "qmmf-source")
framefilter = Gst.ElementFactory.make("capsfilter", "frame-filter")
sink = Gst.ElementFactory.make("waylandsink", "display")

if not pipeline or not source or not framefilter or not sink:
    print("Create element failed")
    return

source.set_property("camera", camera_id)

# Modify the properties for framefilter
if camera_id == 0:
    framefilter.set_property("caps",
                             Gst.caps_from_string("video/x-raw,format=NV12,framerate=30/1,width=1920,height=1080"))
else:
    framefilter.set_property("caps",
                             Gst.caps_from_string("video/x-raw,format=NV12,framerate=30/1,width=1280,height=720"))

# Modify the properties for sink
sink.set_property("x", 200)
sink.set_property("y", 200)
sink.set_property("width", 1280)
sink.set_property("height", 720)

# Build the pipeline
pipeline.add(source)
pipeline.add(framefilter)
pipeline.add(sink)

if not source.link(framefilter):
    print("ERROR: Could not link source to framefilter")
    return
if not framefilter.link(sink):
    print("ERROR: Could not link framefilter to sink")
    return

# Start playing
pipeline.set_state(Gst.State.PLAYING)
print("Started Liveview for camera %d" % camera_id)

return pipeline

# Start display pipelines for the cameras
camera_id1 = 1
pipeline_str = display_camera(camera_id1)

camera_id2 = 2
pipeline_str2 = display_camera(camera_id2)

# Open video capture for recording
cap1 = cv2.VideoCapture(display_camera(camera_id1))
cap2 = cv2.VideoCapture(display_camera(camera_id2))

#I tried this earlier but it is saying Videocapture takes only one argument so I used above    command
#cap1 = cv2.VideoCapture(pipeline_str, cv2.CAP_GSTREAMER)
#cap2 = cv2.VideoCapture(pipeline_str2, cv2.CAP_GSTREAMER)

# Check if the video capture objects were successfully opened
if not cap1.isOpened() or not cap2.isOpened():
    print("Failed to open cameras for recording")
    exit()

  num = 0

while True:
    success1, img1 = cap1.read()
    success2, img2 = cap2.read()

  if not success1 or not success2:
    print("Failed to read from cameras")
    break

k = cv2.waitKey(5)

if k == 27:
    break
elif k == ord('s'): # wait for 's' key to save and exit
    cv2.imwrite('images/stereoLeft/imageL' + str(num) + '.png', img1)
    cv2.imwrite('images/stereoright/imageR' + str(num) + '.png', img2)
    print("images saved!")
    num += 1

cv2.imshow('Camera 1', img1)
cv2.imshow('Camera 2', img2)

# Check for keyboard input
if cv2.waitKey(1) == 27:
    break

# Stop and cleanup the pipelines
pipeline_str.set_state(Gst.State.NULL)
pipeline_str2.set_state(Gst.State.NULL)

 # Release and destroy all windows before termination
cap1.release()
cap2.release()
cv2.destroyAllWindows()
2

most likely, it’s an usb bandwidth problem,
a single usb hub can barely handle a single hires cam.

try to use different usb hubs for the 2 cams, or reduce the resolution, if that works, you’ll know …

3

I am using MIPI OV9282 camera’s on Qualcomm’s rb5 board. They both have separate access. When I am running above code, it shows this log: Here is the log and I can see one window open with live camera access.

gbm_create_device(156): Info: backend name is: msm_drm
Started Liveview for camera 1
gbm_create_device(156): Info: backend name is: msm_drm
Started Liveview for camera 2
gbm_create_device(156): Info: backend name is: msm_drm
Started Liveview for camera 1

But I think rest of the code is not running, because probably OpenCVCapture is not taking livestream? I don’t know what I can change to fix this.

4

can someone help me with this please?

5

Hi @Fourier
I can unfortunately not comment on your question, but how do you feed a pipeline into cv2.VideoCapture? This should not be possible as the first argument should be a str and not a Pipeline. Not sure how your code sample is actually showing one stream.

6

Fourier’s code doesn’t even do that. his display_camera() function returns nothing.

that entire code should be questioned.

VideoCapture can take a string describing a pipeline, as is shown in all the examples that demonstrate gstreamer in opencv.