Opencv android using using CAP_MJPEG nor CAP_GSTREAMER unable to get frames

Android/Java
,
1

I am trying to capture frames from an esp32-cam set up to my android phone, hoping to be able to do object detection from that esp32-cam’s stream. I know my esp32 is also oputting a MJPEG stream, and that Gstreamer may be necessary to interface the video stream to opencv. However, using CAP_MJPEG nor CAP_GSTREAMER does not work.

  1. Trying to use the MJPEG stream directly

code if i were to use purely the url with ip address i.e. http://192.168.0.100 (ip addresses 100 to 105 are all valid, i can see the camera webserver if i open the URL in a web browser)

var urlWithIpAddress = getUrlWithIpAddressFromGStreamerCommand(streamAddress)
var fullGStreamerCommand = "gst-launch-1.0 " + streamAddress // gst-launch-1.0 souphttpsrc etc etc....
//actual code meant for reading from the gstreamer lib
lifecycleScope.launch {
    var videoCapturing = VideoCapture(urlWithIpAddress, CAP_OPENCV_MJPEG)
    videoCapturing.open(urlWithIpAddress, CAP_OPENCV_MJPEG)

Logcat output

app_function Creating pipeline in CustomData at 0xb40000709a195810
app_function custom pipeline string: souphttpsrc location=http://192.168.0.100 ! multipartdemux ! jpegdec ! videoconvert ! autovideosink
urlWithIpAddress is http://192.168.0.100
[ WARN:0@26.437] global cap.cpp:206 open VIDEOIO(CV_MJPEG): backend is generally available but can't be used to capture by name
[ WARN:0@26.438] global cap.cpp:206 open VIDEOIO(CV_MJPEG): backend is generally available but can't be used to capture by name
  1. Trying to use with GStreamer

i read somewhere that opencv may be unable to get frames directly unless paired with gstreamer or ffmpeg (though the post is from 2018)
when trying to get frames from my esp32 camera using the following Gstreamer pipeline commmand

souphttpsrc location=http://192.168.0.102 ! multipartdemux ! jpegdec ! videoconvert ! autovideosink

i encounter the problems failures as noted by my logging statements. Also, it seems like adding the “gst-launch-1.0” to the front of the statement, and specifying CAP_GSTREAMER did not help.

var urlWithIpAddress = getUrlWithIpAddressFromGStreamerCommand(streamAddress)
var fullGStreamerCommand = "gst-launch-1.0 " + streamAddress // gst-launch-1.0 souphttpsrc etc etc....
//actual code meant for reading from the gstreamer lib
lifecycleScope.launch {
    var videoCapturing = VideoCapture(fullGStreamerCommand, CAP_GSTREAMER)
    videoCapturing.open(fullGStreamerCommand, CAP_CAP_GSTREAMER)

here is what my log shows (i have also added other logging statements to see what i pass into the videocapture variable as well as to check if isOpened returns true or false.)

urlWithIpAddress is http://192.168.0.100
videoCapturing opening:  gst-launch-1.0 souphttpsrc location=http://192.168.0.100 ! multipartdemux ! jpegdec ! videoconvert ! autovideosink 
videoCapturing isOpened:  false 
videoCapturing isOpened():  false 
Video capture failed, no frames obtained. videoCapturing.isOpened returned false. 
video capture releasing!

Edit:
i found a new error in my logs. It doesnt always appear

OpenCV(4.8.0) Error: Requested object was not found (could not open directory: /data/app/~~h0G0FSTdUBM1n2pgU7YM8g==/com.pkgname.streamplayer-fI_rhcyXiIM5ZinT8OdKFg==/base.apk!/lib/x86_64) in glob_rec, file /home/ci/opencv/modules/core/src/glob.cpp, line 279

Could anyone weigh in on what I could perhaps try to do?
Thank you :slight_smile:

2

opencv’s android kit does NOT have gst or ffmpeg support by default.
(you have to build either locally (android!), then rebuild the opencv sdk with that)

besides, your gst code is entirely broken,

gst-launch is the binary, you would call on a (pc!) cmdline, this must not be part of the VideoCapture url

opening same thing twice …

if it really is plain mjpeg over http , you could still code your own capture
(it’s just a socket & multipart-form images; look it up…)