GStreamer backend

Python
, ,
1

I need to process live streams from twitch and other sites and the default ffmpeg backend has a bug which leads to stuttering so the recommendation is to use gstreamer instead.

I was trying for almost two weeks but cant get the gstreamer backend working.

I built OpenCV with gstreamer. Every DLL path is correct in cmake-gui, gstreamer directory is added to the path. It builds successfully then when I do import cv2 it cant find the DLLs. I tried everything I found about the topic but nothing worked so I finally copied the gstreamer DLLs to “site-packages\cv2\python-3.9” which is a dirty hack but at least now I can import cv2.

Then the problems continue… I can successfully show my webcam image with:

gst-launch-1.0 camerabin ! autovideosink

or show a twitch stream with:

gst-launch-1.0 [stream url] ! decodebin ! autovideosink

but nothing works with OpenCV. I tried ‘appsink’, BGR convert and every example I found but nothing works.

import cv2
import streamlink
import os
stream = 'https://www.twitch.tv/nokokopuffs'
url = streamlink.streams(stream)['480p'].to_url()
gs = f'{url} ! decodebin ! autovideosink'
os.system(f'gst-launch-1.0 {gs}')
print('CLOSING GST-LAUNCH\n')
cap = cv2.VideoCapture(gs, cv2.CAP_GSTREAMER)
while(True):
    ret, frame = cap.read()
    cv2.imshow('frame', frame)
    if cv2.waitKey(1) != -1: break
cap.release()
cv2.destroyAllWindows()

This code first shows the stream with gst-launch fine then crashes at the OpenCV part with the following output:

Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Got context from element 'souphttpsrc0': gst.soup.session=context, session=(SoupSession)NULL, force=(boolean)false;
Redistribute latency...
Redistribute latency...
Pipeline is PREROLLED ...      
Setting pipeline to PLAYING ...
New clock: GstSystemClock      
ERROR: from element /GstPipeline:pipeline0/GstAutoVideoSink:autovideosink0/GstD3DVideoSink:autovideosink0-actual-sink-d3dvideo: Output window was closed
Additional debug info:
../sys/d3dvideosink/d3dhelpers.c(1910): d3d_render_buffer (): /GstPipeline:pipeline0/GstAutoVideoSink:autovideosink0/GstD3DVideoSink:autovideosink0-actual-sink-d3dvideo
Execution ended after 0:00:01.722096000
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...
CLOSING GST-LAUNCH
[ WARN:0] global C:\opencv\opencv\modules\videoio\src\cap_gstreamer.cpp (480) cv::GStreamerCapture::isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created
[ERROR:0] global C:\opencv\opencv\modules\videoio\src\cap.cpp (116) cv::VideoCapture::open VIDEOIO(GSTREAMER): raised OpenCV exception:
OpenCV(4.2.0) C:\opencv\opencv\modules\videoio\src\cap_gstreamer.cpp:743: error: (-215:Assertion failed) uridecodebin in function 'cv::GStreamerCapture::open'
Traceback (most recent call last):
  File "d:\Work\2110 Imperfect Copies\vidgear_test.py", line 19, in <module>
    cv2.imshow('frame', frame)
cv2.error: OpenCV(4.2.0) C:\opencv\opencv\modules\highgui\src\window.cpp:376: error: (-215:Assertion failed) size.width>0 && size.height>0 in function 'cv::imshow'

Some help would be greatly appreciated as I don’t know what else to try. I built it many times with different options, tried many gstreamer pipeline options. Tried it on windows 10, 11, fedora, ubuntu… Nothing works…

2

First, try to pass only URI to the VideoCapture, it will try to construct pipeline on its own.

If you are writing your custom pipeline, then the sink element must be appsink. And the source should be uridecodebin or something similar in your case - uridecodebin uri="<your url>" ! .... Note that *decodebin elements can automatically demux and decode the stream, while combination of *src + *demux + *decode allow more granular control over this process.
Additionally you should assume that the sink only supports BGR image and you should convert it by adding videoconvert before it: uridecodebin uri=... ! videoconvert ! appsink. You can experiment with gst-launch by adding caps filter before the sink and checking if the conversion is needed: if ... ! video/x-raw,format=BGR ! fakesink does not work, try to add videoconvert somewhere.

3

Thank you very much for your reply!

gs = f'uridecodebin uri="{url}" ! videoconvert ! video/x-raw,format=BGR ! appsink'
cap = cv2.VideoCapture(gs, cv2.CAP_GSTREAMER)

Gives me the following error:

[ WARN:0] global C:\opencv\opencv\modules\videoio\src\cap_gstreamer.cpp (873) cv::GStreamerCapture::open OpenCV | GStreamer warning: Error opening bin: no element "uridecodebin"
[ WARN:0] global C:\opencv\opencv\modules\videoio\src\cap_gstreamer.cpp (616) cv::GStreamerCapture::isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created

Can it be that it can load some parts of gstreamer but not everything because of my hacky installation? As I mentioned I copied contents of gstreamer bin folder to “site-packages\cv2\python-3.9” as a last resort because otherwise I couldn’t get OpenCV to find the DLLs.

I cant understand why because every DLL path is correct in cmake-gui and if I run dependency walker on cv2.cp39-win_amd64.pyd which I built it shows that all the DLLs are linked from the gstreamer folder but if use process monitor to see what is happening when I’m trying to import cv2 it shows that it’s looking for files in different folders but not in the gstreamer folder.

Also as I’m writing this I noticed that the errors point to “C:\opencv\opencv” which was my git clone folder for building which dosn’t exist anymore and my install folder is “C:\opencv\OpenCV-4.5.4”

4

Probably. AFAIK GStreamer needs to find a folder with plugins somehow. Try to install GStreamer as recommended by its guide, then add its location to the PATH environment variable (e.g. OpenCV with GStreamer and QT on Windows | by Imran Bangash | Towards Data Science). Also, python 3.8+ on windows require to call add_dll_directory for each extra dll location (windows - Can't import dll module in Python - Stack Overflow).

5

Thank you! Adding the following line seems to solve all my problems.

os.add_dll_directory(r'C:\gstreamer\1.0\msvc_x86_64\bin')