Simple example of traditional inference using gapi

I want to compile and run a graph using the gapi module, with mimimal preprocessing, a simple net inference, without postprocessing, using tradition image I/O (not streaming). Unfortunately, there are no examples for that available. Can someone please share a working example?

this g-api tutorial covers dnn based face-detection, as well as gender/age classification

Thanks, but these examples are for video streaming. I want a very simple example, just doing one inference and compiling a graph, that can later be used to infere any cv::Mat.

indeed, it’s quite heavy on the streaming ;(

do you have OpenVino ? (it seems, it can only load ie models, bummer)

Yes, i compiled opencv (4.5.2) with inference engine (2020.04) and ngraph. I can infere the used net (an xml and a bin from OpenVINO) without gapi. However, when I try to put the inference into a graph, I get “Kernel org.opencv.dnn.infer was not found”. OS is Ubuntu 16.04.

imho, the example can be reduced for single images:


#include "opencv2/imgcodecs.hpp"
#include "opencv2/imgproc.hpp"

#include "opencv2/gapi.hpp"
#include "opencv2/gapi/core.hpp"
#include "opencv2/gapi/imgproc.hpp"
#include "opencv2/gapi/infer.hpp"
#include "opencv2/gapi/infer/ie.hpp"
#include "opencv2/gapi/cpu/gcpukernel.hpp"
#include "opencv2/gapi/streaming/cap.hpp"

const std::string about =
    "This is an OpenCV-based version of Security Barrier Camera example";
const std::string keys =
    "{ h help |   | print this help message }"
    "{ input  |   | Path to an input img file }"
    "{ fdm    |   | IE face detection model IR }"
    "{ fdw    |   | IE face detection model weights }"
    "{ fdd    |   | IE face detection device }";

namespace custom {

//! [G_API_NET]
// Face detector: takes one Mat, returns another Mat
G_API_NET(Faces, <cv::GMat(cv::GMat)>, "face-detector");
// SSD Post-processing function - this is not a network but a kernel.
// The kernel body is declared separately, this is just an interface.
// This operation takes two Mats (detections and the source image),
// and returns a vector of ROI (filtered by a default threshold).
// Threshold (or a class to select) may become a parameter, but since
// this kernel is custom, it doesn't make a lot of sense.
G_API_OP(PostProc, <cv::GArray<cv::Rect>(cv::GMat, cv::GMat)>, "custom.fd_postproc") {
    static cv::GArrayDesc outMeta(const cv::GMatDesc &, const cv::GMatDesc &) {
        // This function is required for G-API engine to figure out
        // what the output format is, given the input parameters.
        // Since the output is an array (with a specific type),
        // there's nothing to describe.
        return cv::empty_array_desc();
    }
};

// OpenCV-based implementation of the above kernel.
GAPI_OCV_KERNEL(OCVPostProc, PostProc) {
    static void run(const cv::Mat &in_ssd_result,
                    const cv::Mat &in_frame,
                    std::vector<cv::Rect> &out_faces) {
        const int MAX_PROPOSALS = 200;
        const int OBJECT_SIZE   =   7;
        const cv::Size upscale = in_frame.size();
        const cv::Rect surface({0,0}, upscale);

        out_faces.clear();

        const float *data = in_ssd_result.ptr<float>();
        for (int i = 0; i < MAX_PROPOSALS; i++) {
            const float image_id   = data[i * OBJECT_SIZE + 0]; // batch id
            const float confidence = data[i * OBJECT_SIZE + 2];
            const float rc_left    = data[i * OBJECT_SIZE + 3];
            const float rc_top     = data[i * OBJECT_SIZE + 4];
            const float rc_right   = data[i * OBJECT_SIZE + 5];
            const float rc_bottom  = data[i * OBJECT_SIZE + 6];

            if (image_id < 0.f) {  // indicates end of detections
                break;
            }
            if (confidence < 0.5f) { // a hard-coded snapshot
                continue;
            }

            // Convert floating-point coordinates to the absolute image
            // frame coordinates; clip by the source image boundaries.
            cv::Rect rc;
            rc.x      = static_cast<int>(rc_left   * upscale.width);
            rc.y      = static_cast<int>(rc_top    * upscale.height);
            rc.width  = static_cast<int>(rc_right  * upscale.width)  - rc.x;
            rc.height = static_cast<int>(rc_bottom * upscale.height) - rc.y;
            out_faces.push_back(rc & surface);
        }
    }
};
//! [Postproc]

} // namespace custom
int main(int argc, char *argv[])
{
    cv::CommandLineParser cmd(argc, argv, keys);
    cmd.about(about);
    if (cmd.has("help")) {
        cmd.printMessage();
        return 0;
    }
    const std::string input = cmd.get<std::string>("input");

    // Express our processing pipeline. Lambda-based constructor
    // is used to keep all temporary objects in a dedicated scope.
    //! [GComputation]
    cv::GComputation pp([]() {
            // Declare an empty GMat - the beginning of the pipeline.
            cv::GMat in;

            // Run face detection on the input frame. Result is a single GMat,
            // internally representing an 1x1x200x7 SSD output.
            // This is a single-patch version of infer:
            // - Inference is running on the whole input image;
            // - Image is converted and resized to the network's expected format
            //   automatically.
            cv::GMat detections = cv::gapi::infer<custom::Faces>(in);

            // Parse SSD output to a list of ROI (rectangles) using
            // a custom kernel. Note: parsing SSD may become a "standard" kernel.
            cv::GArray<cv::Rect> faces = custom::PostProc::on(detections, in);
            // Now specify the computation's boundaries - our pipeline consumes
            // one images and produces one output.
            return cv::GComputation(cv::GIn(in),
                                    cv::GOut(faces));
        });
    //! [GComputation]

    // Note: it might be very useful to have dimensions loaded at this point!
    // After our computation is defined, specify how it should be executed.
    // Execution is defined by inference backends and kernel backends we use to
    // compile the pipeline (it is a different step).

    // Declare IE parameters for FaceDetection network. Note here custom::Face
    // is the type name we specified in GAPI_NETWORK() previously.
    // cv::gapi::ie::Params<> is a generic configuration description which is
    // specialized to every particular network we use.
    //
    // OpenCV DNN backend will have its own parmater structure with settings
    // relevant to OpenCV DNN module. Same applies to other possible inference
    // backends...
    //! [Param_Cfg]
    auto det_net = cv::gapi::ie::Params<custom::Faces> {
        cmd.get<std::string>("fdm"),   // read cmd args: path to topology IR
        cmd.get<std::string>("fdw"),   // read cmd args: path to weights
        cmd.get<std::string>("fdd"),   // read cmd args: device specifier
    };
    // Form a kernel package (with a single OpenCV-based implementation of our
    // post-processing) and a network package (holding our three networks).
    auto kernels = cv::gapi::kernels<custom::OCVPostProc>();
    auto networks = cv::gapi::networks(det_net);

    { // inference !

        // Declare data objects we will be receiving from the pipeline.
        cv::Mat in_frame = cv::imread("input"); // the input !                      // The captured frame itself
        std::vector<cv::Rect> faces;        // Array of detected faces
        pp.apply(cv::gin(in_frame),
                     cv::gout(faces),
                     cv::compile_args(kernels, networks));

        // do something with the face rects
	}
	return 0;
}

however, without openvino, we get: G-API has been compiled without OpenVINO IE support

1 Like

This code works! Thank you for figuring this out.

1 Like

Next step would be figuring out, how to serialize the whole graph and deploy it somewhere, with no prior knowledge of the details. With a simple graph, it would be the following.

// Make simple graph
cv::GMat in;
cv:GMat out = cv::gapi::resize(in, cv::Size(), 0.01, 0.01);
auto p = cv::gapi::serialize(cv::GComputation(in, out));
std::ofstream fout(“gcomp.bin”, std::ios::out | std::ios::binary);
fout.write((const char*)&p[0], p.size()); fout.close();

// Load graph
cv::Mat imgIn = cv::Mat::ones(8000, 4000, CV_8U);
cv::Mat imgOut;
std::ifstream file(“gcomp.bin”, std::ios::binary);
file.unsetf(std::ios::skipws);
std::streampos fileSize;
file.seekg(0, std::ios::end);
fileSize = file.tellg();
file.seekg(0, std::ios::beg);
std::vector<char> vec;
vec.reserve(fileSize);
vec.insert(vec.begin(), std::istream_iterator<char>(file), std::istream_iterator<char>());
auto c = cv::gapi::deserialize<cv::GComputation>(vec);
c.apply(imgIn, imgOut);

Now this cannot be applied to the upper example, as the kernels are not known by the consumer. Maybe someone is also interested in this approach. I’ll post a solution if I find one. Trying ot contact contributors and authors on this issue (G-API : how is serialize/deserialize supposed to be used properly ? · Issue #17965 · opencv/opencv · GitHub).

1 Like