How properly to wrap OpenCV APIs that take cv::InputArray/cv::OutputArray/cv::InputOutputArray for P/Invoke (C#)

C++
1

Hi there!

I understand that my question not really suit this forum, but I don’t know any suitable one

I am writing my custom OpenCV wrapper (now I am using OpenCV 4.11) for C#

I am using DNN-based face detector for face detection (see cv::FaceDetectorYN), in particular detect method

This method has two input parameters: cv::InputArray and cv::OutputArray

My wrapped method (I expose two detect overloads)

C++ side:

extern "C" {
    WRAPPEROPENCV_DLL_API void* Create_FaceDetect_DNN(const char* model, const char* config,
        cv::Size input_size, float score_threshold, float nms_threshold, int top_k,
        int backend_id, int target_id) {
        cv::Ptr<cv::FaceDetectorYN>* faceDetectorYN = new cv::Ptr<cv::FaceDetectorYN>(
            cv::FaceDetectorYN::create(model, config, input_size, score_threshold,
                nms_threshold, top_k, backend_id, target_id));

        if (faceDetectorYN->empty()) {
            std::cerr << "Error: Failed to create FaceDetectorYN" << std::endl;
            return nullptr;
        }
        return static_cast<void*>(faceDetectorYN);
    }

    WRAPPEROPENCV_DLL_API int Detect_FaceDetect_DNN(void* faceDetectorYN, cv::_InputArray* image, cv::_OutputArray* faces) {
        if (!faceDetectorYN || !image || !faces) {
            std::cerr << "Error: Detect invalid input(s)" << std::endl;
            return 0;
        }
        try {
            cv::Ptr<cv::FaceDetectorYN>* detectorPtr = static_cast<cv::Ptr<cv::FaceDetectorYN>*>(faceDetectorYN);
            return (*detectorPtr)->detect(*image, *faces);
        }
        catch (cv::Exception e) {
            return 0;
        }
        
    }

    WRAPPEROPENCV_DLL_API int DetectMat_FaceDetect_DNN(void* faceDetectorYN, cv::Mat* image, cv::Mat* faces) {
        if (!faceDetectorYN || !image || !faces) {
            std::cerr << "Error: Detect invalid input(s)" << std::endl;
            return 0;
        }

        cv::Ptr<cv::FaceDetectorYN>* detectorPtr = static_cast<cv::Ptr<cv::FaceDetectorYN>*>(faceDetectorYN);
        return (*detectorPtr)->detect(*image, *faces);
    }

    WRAPPEROPENCV_DLL_API void Delete_FaceDetect_DNN(void* faceDetectorYN) {
        if (!faceDetectorYN) {
            std::cerr << "Error: Delete Invalid input - FaceDetectorYN" << std::endl;
            return;
        }
        delete static_cast<cv::Ptr<cv::FaceDetectorYN>*>(faceDetectorYN);
    }
}

C# side:

public class DNNFacedetect : DisposableObject
    {
        public DNNFacedetect(string model, string config, Size2i input_size, float score_threshold = 0.9f,
            float nms_threshold = 0.3f, int top_k = 5000, 
            Backend backend_id = Backend.DNN_BACKEND_DEFAULT, Target target_id = Target.DNN_TARGET_CPU)
        {
            NativeObj = ThrowIfNull(Create_FaceDetect_DNN(model, config, input_size, score_threshold, nms_threshold,
                top_k, (int)backend_id, (int)target_id));
        }

        public int Detect(InputArray image, OutputArray faces)
        {
            return Detect_FaceDetect_DNN(NativeObj, image.NativeObj, faces.NativeObj);
        }

        public int DetectMat(Mat image, Mat faces)
        {
            return DetectMat_FaceDetect_DNN(NativeObj, image.NativeObj, faces.NativeObj);
        }
        protected override void DisposeUnmanaged()
        {
            if (NativeObj == IntPtr.Zero)
                return;

            Delete_FaceDetect_DNN(NativeObj);
            base.DisposeUnmanaged();
        }


        [DllImport("WrapperOpenCV.dll", CallingConvention = CallingConvention.Cdecl, CharSet = CharSet.Ansi)]
        public static extern IntPtr Create_FaceDetect_DNN(string model, string config, Size2i input_size,
            float score_threshold, float nms_threshold, int top_k, int backend_id, int target_id);

        [DllImport("WrapperOpenCV.dll", CallingConvention = CallingConvention.Cdecl)]
        public static extern int Detect_FaceDetect_DNN(IntPtr faceDetectorYN, IntPtr image, IntPtr faces);

        [DllImport("WrapperOpenCV.dll", CallingConvention = CallingConvention.Cdecl)]
        public static extern int DetectMat_FaceDetect_DNN(IntPtr faceDetectorYN, IntPtr image, IntPtr faces);

        [DllImport("WrapperOpenCV.dll", CallingConvention = CallingConvention.Cdecl)]
        public static extern void Delete_FaceDetect_DNN(IntPtr faceDetectorYN);
    }

Example of usage:

      inMat = new Mat(new Size2i(width, height), MatType.CV_8UC3); //copy image to this Mat
            inResizedMat = new Mat();
            detectedFaceMat = new Mat(); 
               
            dnnFacedetect = new DNNFacedetect(modelPath, "", size320,
                    scoreThreshold, nmsThreshold, topK, Backend.DNN_BACKEND_CUDA, Target.DNN_TARGET_CUDA);

            Imgproc.Resize(inMat, inResizedMat, size320);
            dnnFacedetect?.DetectMat(inResizedMat, detectedFaceMat);

If I use public int DetectMat(Mat image, Mat faces) programm works fine (without crashes)

But if I use public int Detect(InputArray image, OutputArray faces) programm crashes after few seconds

What could be reason for such behavior?

Best regards,

2

crosspost:

3

Your best bet is to compile a library with a C ABI, then use P/Invoke from there. Basically, create your face detector in C++, return it to C# as an opaque pointer (and expose a corresponding Release function that gets invoked from C#). Then to detect faces, pass the raw pixel data (e.g. as a byte[]) to C via P/Invoke, and construct a mat from it to do the processing.

You effectively have to manage all the serialization and deserialization to go across the P/Invoke boundary. I’m building a tool that can automate this, by compiling your Python to a self-contained compiled binary (via C++). It has bindings for C#, so lmk if you’d be interested to learn more.

4

Hi!

I have already implemented that you described:

  1. Use C ABI (extern C) and build dll
  2. Return void* from dll when create cv::Mat and cv::_InputArray in C# program
  3. Use P/Invoke
  4. Wrap byte[ ] into cv::Mat and cv::_InputArray

My problem is that method public int Detect(InputArray image, OutputArray faces) throw exception while method public int DetectMat(Mat image, Mat faces) not throw

5

Can you share the exception? Usually, if your C/C++ code throws an exception, it won’t get caught in C#. Instead, you will get a hard crash.

6

Hi!

You are right - in C# program I just see SEHException, but I have placed critical part of opencv wrapper (C++) in try/catch block
So that exception from C++ code:

Error in Detect_FaceDetect_DNN: OpenCV(4.11.0) C:\Users\me\Documents\GitFlic\opencv 4.11\opencv-4.11.0\modules\core\src\matrix_wrap.cpp:831: error: (-213:The function/feature is not implemented) Unknown/unsupported array type in function ‘cv::_InputArray::type’

Error in Detect_FaceDetect_DNN: OpenCV(4.11.0) C:\Users\me\Documents\GitFlic\opencv 4.11\opencv-4.11.0\modules\core\src\matrix_wrap.cpp:1748: error: (-213:The function/feature is not implemented) Unknown/unsupported array type in function ‘cv::_OutputArray::release’

Error in Detect_FaceDetect_DNN: OpenCV(4.11.0) C:\Users\me\Documents\GitFlic\opencv 4.11\opencv-4.11.0\modules\core\src\matrix_wrap.cpp:1476: error: (-27:Null pointer) create() called for the missing output array in function ‘cv::_OutputArray::create’

These exceptions is throw approximately once every 80-140 frames (not all together, but one by one) and reason can’t be a format/size of byte[] since every frame have same size and same format (I have only one algorithm that convert frame from camera to cv::Mat)

Also there is no exception if I pass cv::Mat instead of cv::_InputArray