Can not opencv more than 20 Video Capture RTSP FFMPEG/GStreamer

OpenCV => 4.5.1

Trying to open 40 rtsp stream with VideoCapture, each VideoCapture is in one thread, can only opened 20 at the same time, the 20 other failed to get stream.
I tried with FFMPEG backend and Gstreamer (1.18.1).

With FFMPEG: After 20 VideoCapture opened, I got this error :

[rtsp @ 00000250169be640] method SETUP failed: 500 Internal Server Error

With GStreamer:

[ WARN:15] global C:\opencv\4.5.1\opencv\modules\videoio\src\cap_gstreamer.cpp (1825) cv::handleMessage OpenCV | GStream
er warning: Embedded video playback halted; module rtspsrc19 reported: Could not read from resource.
[ WARN:15] global C:\opencv\4.5.1\opencv\modules\videoio\src\cap_gstreamer.cpp (914) cv::GStreamerCapture::open OpenCV |
 GStreamer warning: unable to start pipeline
[ WARN:15] global C:\opencv\4.5.1\opencv\modules\videoio\src\cap_gstreamer.cpp (501) cv::GStreamerCapture::isPipelinePla
ying OpenCV | GStreamer warning: GStreamer: pipeline have not been created
Steps to reproduce
#include <QCoreApplication>
#include <QDebug>
#include <QThread>
#include <utility>
#include <QSettings>
#include <QJsonDocument>
#include <QJsonObject>
#include <QJsonArray>
#include <opencv2/opencv.hpp>
#include <QFile>
#include <thread>

class Camera {
public:
    Camera() = delete;
    Camera(QString url, QString name);

    void start() {
        auto t = new std::thread(&Camera::run, this);
    }

    void run();

private:
    QString url_;
    QString name_;
};

Camera::Camera(QString url, QString name): url_(std::move(url)), name_(std::move(name)) {
}

void Camera::run() {
    auto cap = cv::VideoCapture(url_.toStdString(), cv::CAP_FFMPEG);
    qDebug() << "Start new camera thread with name " << name_;
	cv::Mat frame;
    while (true) {
        bool ret = cap.read(frame);
        if (!ret) {
            qDebug() << "Camera with name " << name_ << " FAILED !";
            break;
        }
    }
}

int main(int argc, char *argv[]) {
    QCoreApplication a(argc, argv);

    // QString urlPattern = "rtspsrc location=<url> ! rtph264depay ! h264parse ! nvh264dec ! appsink";
    // QString urlPattern = "rtspsrc location=<url> ! decodebin ! videoconvert ! appsink";
    QString urlPattern = "<url>";

    QFile file;
    file.setFileName("cameras.json");
    file.open(QIODevice::ReadOnly);

    QJsonDocument jsonDoc = QJsonDocument::fromJson(file.readAll());
    qDebug() << jsonDoc;

    std::vector<std::unique_ptr<Camera>> camerasToLaunch;

    QList<QVariant> cameraList = jsonDoc.toVariant().toList();
    for (const auto& camera: cameraList) {
        qDebug() << camera;
        qDebug() << "Get url of camera";
        QString url = camera.toMap()["url"].toString();
        QString name = camera.toMap()["name"].toString();
        QString urlFull = urlPattern.replace("<url>", url);

        auto pCamera = std::make_unique<Camera>(urlFull, name);
        pCamera->start();
        camerasToLaunch.push_back(std::move(pCamera));
        std::this_thread::sleep_for(std::chrono::milliseconds(300));
    }

    return QCoreApplication::exec();
}

I tried to launch more than 20 process which create a VideoCapture and do “read”, I can notice that. only 20 processes work and other processes have 0% CPU.

I have the same issue on Ubuntu 18.04

If i launched 2 times my program above, the second program can not read any frame :

[ WARN:0] global C:\opencv\4.5.1\opencv\modules\videoio\src\cap_gstreamer.cpp (1825) cv::handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module rtspsrc0 reported: Could not write to resource.
[ WARN:0] global C:\opencv\4.5.1\opencv\modules\videoio\src\cap_gstreamer.cpp (914) cv::GStreamerCapture::open OpenCV | GStreamer warning: unable to start pipeline
[ WARN:0] global C:\opencv\4.5.1\opencv\modules\videoio\src\cap_gstreamer.cpp (501) cv::GStreamerCapture::isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created

But I can display the rtsp stream with vlc or gstreamer in command line

can you open more than 20 concurrent streams using any other program?

that should tell you something. the server doesn’t like what you’re doing. consider configuring the server to accept more connections per client.

1 Like

Yes, using VLC or Gstreamer command Line, i can opened more than 20 streams.

Last time I tested this I couldn’t get past 31 streams (that’s with cudacodec which uses ffmpeg to stream from rtsp), maybe its dependent on the version of the ffmpeg libs which opencv is built against. I would try testing the latest version of ffmpeg on its own and if I could get more streams try to build opencv against that version.

Is the CPU near 100% usage? VLC could be using the GPU for decoding, I don’t know about Gstreamer.

RAM/VRAM usage could also be peaking.

I use Gstreamer and Ffmpeg as backend for VideoCapture. And same result. I didn’t try to mixte between them

CPU and RAM are not full when I try and I tried on heavy configuration ( 72 cores and more than 100Go of Ram, server configuration)

you should definitely investigate that.

Ok but how ?
Do you think my network could be the fault ?

Have you tried ffmpeg.exe to see if you can get more than 20 streams?

I will try but with VLC and Gstreamer and I have not this issue

no, not your network.

that’s an error from the server you’re connecting to.

Ok thanks.

Can not understand why on the same computer, I can open more than 20 instances of VLC/Gstreamer. In my test, I opened 30 instances of VLC and use 3 differentes cameras. 10 VLC to camera1, 10 other to camera2 and 10 other to camera 3.

I run my program on 2 computers in same network. Computer A and computer B.
Computer A connected to 10 streams from camera 1 and 10 streams from camera 2 (can not opened more streams)
On computer B, I can connected to 20 streams like Computer A.

But on same computer, I can not opened more than 20 streams.
The camera can handle less than 20 connections, that why, I only opened 10 streams for each cameras.

But if I understand well, the issue doesn’t come from OpenCV ?

If it works with GStreamer but not with the OpenCV GStreamer backend it points to a configuration issue (they should both be compiled against the same libs) when using OpenCV. Have you tried streaming with both TCP and UDP. The FFmpeg backend defaults to TCP so that could be causing your problem. That is the server might allow more UDP than TCP connections. I guess with GStreamer you can just add the option to the commands you send. If you want to try with FFmpeg you need to set the following environmental variable before running your application.

set “OPENCV_FFMPEG_CAPTURE_OPTIONS=rtsp_transport;udp”

What happens if you run 20 streams with OpenCV and then try to launch additional streams with VLC, do they work?

1 Like

When you said “server”, you mean the camera which generate the rtsp stream ?

When I run 20 streams, I can launch additional streams with VLC or gstreamer. But not a program which uses OpenCV on the same computer.

I will try to mixte TCP and UDP

GStreamer and OpenCV Gstreamer use same libs (Gstreamer installed before Opencv compilation)

Yes.

If it was me I would wireshark the connection to see what the difference is between the messages passed by vanilla GStreamer and OpenCV GStreamer when you try to initialize the 21st stream.

Good idea ! I will try it today

Hi,

The issue come from my organisation network management. The problem doesn’t come from OpenCV … Sorry.

Thanks all for your help.

1 Like