I have been struggling for a while now to read basic UDP RPI stream using gstreamer and opencv, and i am hoping i’ll be able to find a solution in this forum which i enjoy so much.
My architecture:
my need:
to send rtp stream over wireless network and catch it on the other side using opencv to then restream it to html format to use on a web app.
my problem:
receiver is stuck on cv2.VideoCapture()
what i’ve tried:
mainly tried chaning ports, IP addresses, and basic pipeline configuration like encoders or video converts
Sender.py on Jetson NX:
camset='v4l2src device=/dev/video0 ! video/x-raw,width=640,height=360,framerate=52/1 ! \
nvvidconv flip-method=0 ! video/x-raw(memory:NVMM), format=I420, width=640, height=360 ! \
nvvidconv ! video/x-raw, format=BGRx ! videoconvert ! video/x-raw, format=BGR ! queue ! appsink drop=1'
gst_str_rtp = "appsrc ! video/x-raw,format=BGR ! queue ! videoconvert ! video/x-raw,format=BGRx ! nvvidconv !\
video/x-raw(memory:NVMM),format=NV12,width=640,height=360,framerate=52/1 ! nvv4l2h264enc insert-sps-pps=1 \
insert-vui=1 idrinterval=30bitrate=1000000 EnableTwopassCBR=1 ! h264parse ! rtph264pay ! udpsink host=169.254.84.10 port=5004 auto-multicast=0"
out = cv2.VideoWriter(gst_str_rtp, cv2.CAP_GSTREAMER, 0, float(52), (frame_width, frame_height), True)
cap = cv2.VideoCapture(camset,cv2.CAP_GSTREAMER)
if not cap.isOpened():
print("Cannot capture from camera. Exiting.")
quit()
# Check writer
if not out:
print("Cannot write. Exiting.")
quit()
# Go
while True:
ret, frame = cap.read()
if ret == False:
break
out.write(frame)
cv2.imshow("sender", frame)
if cv2.waitKey(1) & 0xFF == ord('q'):
break
the sender works well.
receiver.py on windows 10:
camSet='udpsrc address=169.254.84.10 port=5004 auto-multicast=0 ! application/x-rtp,media=video,encoding-name=H264 ! \
rtpjitterbuffer latency=0 ! rtph264depay ! avdec_h264 ! videoconvert !\
video/x-raw,format=BGR ! appsink drop=1'
fail=1
print("after camset")
while fail:
cap = cv2.VideoCapture(camSet,cv2.CAP_GSTREAMER)
if cap.isOpened():
fail=0
else:
print("fail")
while True:
ret, frame = cap.read()
if ret:
cv2.imshow('stream',frame)
else:
print("Bad frame received")
#outvid.write(frame)
if cv2.waitKey(10) & 0xFF == ord("q"):
break
cap.release()
cv2.destroyAllWindows()
os._exit(0)
exit()
receiver gst-launch that works perfectly:
gst-launch-1.0 udpsrc address=169.254.84.10 port=5004 ! application/x-rtp,media=video,encoding-name=H264 ! queue ! rtph264depay ! h264parse ! avdec_h264 ! videoconvert !autovideosink
for some reason my receiver gets completely stuck on videocapture, it returns no errors, just gets stuck. i know the stream works because gst launch works, the problem i assume is the pipeline.
cv2.getBuiltInformation for reference:
General configuration for OpenCV 4.5.4 =====================================
Version control: unknown
Extra modules:
Location (extra): C:/opencv/opencv_contrib-4.5.4/modules
Version control (extra): unknown
Platform:
Timestamp: 2022-04-11T12:45:59Z
Host: Windows 10.0.19044 AMD64
CMake: 3.23.0-rc1
CMake generator: Visual Studio 17 2022
CMake build tool: C:/Program Files/Microsoft Visual Studio/2022/Community/MSBuild/Current/Bin/amd64/MSBuild.exe
MSVC: 1931
Configuration: Debug Release
CPU/HW features:
Baseline: SSE SSE2 SSE3
requested: SSE3
Dispatched code generation: SSE4_1 SSE4_2 FP16 AVX AVX2 AVX512_SKX
requested: SSE4_1 SSE4_2 AVX FP16 AVX2 AVX512_SKX
SSE4_1 (17 files): + SSSE3 SSE4_1
SSE4_2 (2 files): + SSSE3 SSE4_1 POPCNT SSE4_2
FP16 (1 files): + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 AVX
AVX (5 files): + SSSE3 SSE4_1 POPCNT SSE4_2 AVX
AVX2 (32 files): + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 FMA3 AVX AVX2
AVX512_SKX (8 files): + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 FMA3 AVX AVX2 AVX_512F AVX512_COMMON AVX512_SKX
C/C++:
Built as dynamic libs?: YES
C++ standard: 11
C++ Compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.31.31103/bin/Hostx64/x64/cl.exe (ver 19.31.31104.0)
C++ flags (Release): /DWIN32 /D_WINDOWS /W4 /GR /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _SCL_SECURE_NO_WARNINGS /Gy /bigobj /Oi /fp:precise /EHa /wd4127 /wd4251 /wd4324 /wd4275 /wd4512 /wd4589 /MP /MD /O2 /Ob2 /DNDEBUG
C++ flags (Debug): /DWIN32 /D_WINDOWS /W4 /GR /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _SCL_SECURE_NO_WARNINGS /Gy /bigobj /Oi /fp:precise /EHa /wd4127 /wd4251 /wd4324 /wd4275 /wd4512 /wd4589 /MP /MDd /Zi /Ob0 /Od /RTC1
C Compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.31.31103/bin/Hostx64/x64/cl.exe
C flags (Release): /DWIN32 /D_WINDOWS /W3 /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _SCL_SECURE_NO_WARNINGS /Gy /bigobj /Oi /fp:precise /MP /MD /O2 /Ob2 /DNDEBUG
C flags (Debug): /DWIN32 /D_WINDOWS /W3 /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _SCL_SECURE_NO_WARNINGS /Gy /bigobj /Oi /fp:precise /MP /MDd /Zi /Ob0 /Od /RTC1
Linker flags (Release): /machine:x64 /INCREMENTAL:NO
Linker flags (Debug): /machine:x64 /debug /INCREMENTAL
ccache: NO
Precompiled headers: YES
Extra dependencies:
3rdparty dependencies:
OpenCV modules:
To be built: aruco barcode bgsegm bioinspired calib3d ccalib core datasets dnn dnn_objdetect dnn_superres dpm face features2d flann fuzzy gapi hfs highgui img_hash imgcodecs imgproc intensity_transform line_descriptor mcc ml objdetect optflow phase_unwrapping photo plot python3 quality rapid reg rgbd saliency shape stereo stitching structured_light superres surface_matching text tracking ts video videoio videostab wechat_qrcode xfeatures2d ximgproc xobjdetect xphoto
Disabled: world
Disabled by dependency: -
Unavailable: alphamat cudaarithm cudabgsegm cudacodec cudafeatures2d cudafilters cudaimgproc cudalegacy cudaobjdetect cudaoptflow cudastereo cudawarping cudev cvv freetype hdf java julia matlab ovis python2 sfm viz
Applications: tests perf_tests apps
Documentation: NO
Non-free algorithms: YES
Windows RT support: NO
GUI: WIN32UI
Win32 UI: YES
VTK support: NO
Media I/O:
ZLib: build (ver 1.2.11)
JPEG: build-libjpeg-turbo (ver 2.1.0-62)
WEBP: build (ver encoder: 0x020f)
PNG: build (ver 1.6.37)
TIFF: build (ver 42 - 4.2.0)
JPEG 2000: build (ver 2.4.0)
OpenEXR: build (ver 2.3.0)
HDR: YES
SUNRASTER: YES
PXM: YES
PFM: YES
Video I/O:
DC1394: NO
FFMPEG: YES (prebuilt binaries)
avcodec: YES (58.134.100)
avformat: YES (58.76.100)
avutil: YES (56.70.100)
swscale: YES (5.9.100)
avresample: YES (4.0.0)
GStreamer: YES (1.20.1)
DirectShow: YES
Media Foundation: YES
DXVA: YES
Parallel framework: Concurrency
Trace: YES (with Intel ITT)
Other third-party libraries:
Intel IPP: 2020.0.0 Gold [2020.0.0]
at: C:/Users/ktbm/build/3rdparty/ippicv/ippicv_win/icv
Intel IPP IW: sources (2020.0.0)
at: C:/Users/ktbm/build/3rdparty/ippicv/ippicv_win/iw
Lapack: NO
Eigen: NO
Custom HAL: NO
Protobuf: build (3.5.1)
OpenCL: YES (NVD3D11)
Include path: C:/opencv/opencv-4.5.4/3rdparty/include/opencl/1.2
Link libraries: Dynamic load
Python 3:
Interpreter: C:/Program Files/Python310/python.exe (ver 3.10.2)
Libraries: C:/Program Files/Python310/libs/python310.lib (ver 3.10.2)
numpy: C:/Users/ktbm/AppData/Roaming/Python/Python310/site-packages/numpy/core/include (ver 1.22.3)
install path: C:/Program Files/Python310/Lib/site-packages/cv2/python-3.10
Python (for build): C:/Python27/python.exe
Java:
ant: NO
JNI: NO
Java wrappers: NO
Java tests: NO
Install to: C:/Users/ktbm/build/install
-----------------------------------------------------------------
I have opened a few camera streams using gstreamer so i’ll have to assume that the compilation of opencv with gstreamer worked fine.
I’d appreciate any help on the matter as it really is my only problem left on my project and it is a very big problem because i have to see that stream.
sidenote:
if anyone has any gst-launch pipeline that can stream the received frames into an http format for html to read, that could also solve my issue, but i can’t have more than 100-200ms delay in the stream