I want to import TensorFlow 2.x - Object Detection - Mask R-CNN Model in OpenCV DNN API

Hi all,

System information (version)

*TensorFlow => 2.4.1

  • OpenCV => 4.5.1
  • Operating System / Platform => Windows 10 64 Bit
  • Compiler => Visual Studio 2019

Detailed description

For my task I want to use the Mask R-CNN Inception ResNet V2 1024x1024 object detection model from the TensorFlow 2 Detection Zoo.

I already set up everything regarding TensorFlow by following this tutorial: Train a Mask R-CNN model with the Tensorflow Object Detection API

I can also use TensorFlow to run the inference, e.g. for a single image. So everything works on the TensorFlow side.

Now I want to port my SavedModel from TensorFlow to C++ and OpenCV DNN. And I’ve been failing at this for several days now, I’ve checked and tried quite a few forum posts and Github issues, but nothing could solve my problems.

But let’s start from the beginning:

I know that the OpenCV DNN module does not support the SavedModel format and does expect the FrozenGraph format instead. I have already some problems with the conversion from SavedModel to FrozenGraph:

To convert the SavedModel to a FrozenGraph I use the following python code:

import tensorflow as tf
from tensorflow.python.framework.convert_to_constants import convert_variables_to_constants_v2

loaded = tf.saved_model.load('inference_graph/saved_model')
infer = loaded.signatures['serving_default']

f = tf.function(infer).get_concrete_function(input_tensor=tf.TensorSpec(shape=[1, None, None, 3], dtype=tf.uint8))
f2 = convert_variables_to_constants_v2(f)
graph_def = f2.graph.as_graph_def()

# Export frozen graph
with tf.io.gfile.GFile('frozen_graph.pb', 'wb') as f:
   f.write(graph_def.SerializeToString())

I found this code as an answer to a stackoverflow issue.

‘inference_graph/saved_model’ is the path to the folder containing my SavedModel. This code saves the resulting FrozenGraph as frozen_graph.pb. Everything seems to work fine.
BUT when I run the following code with my newly gained FrozenGraph:

#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc.hpp>
#include <opencv2/dnn.hpp>
#include <iostream>

using namespace cv;
using namespace cv::dnn;
using namespace std;

int main()
{
	// Path to the FrozenGraph file
	String savedModelPath= ".\\pretrained_TF-model\\frozen_graph.pb";
	
	Net net = readNetFromTensorflow(savedModelPath);
	net.setPreferableBackend(DNN_BACKEND_OPENCV);
	net.setPreferableTarget(DNN_TARGET_CPU);

	Mat img = imread(".\\images\\Screenshot.png");
        Mat blobImg = blobFromImage(img, 1.0, Size(512, 512), Scalar(0, 0, 0),false,false, CV_32F);
		
	net.setInput(blobImg);
	Mat output = net.forward();

	return 0;
}

The readNetFromTensorflow() method throws an exception difficult to decipher for me:

OpenCV(4.5.1) Error: Unspecified error (Input layer not found: StatefulPartitionedCall/StatefulPartitionedCall/Const_1) in cv::dnn::dnn4_v20201117::`anonymous-namespace’::TFImporter::connect, file C:\build\master_winpack-build-win64-vc15\opencv\modules\dnn\src\tensorflow\tf_importer.cpp, line 553

However, there is also the possibility to give this method a second parameter, the so-called config file.
This config file needs to be generated using the FrozenGraph file. However, I do not manage to generate this file. If I use the “tf_text_graph_mask_rcnn.py” script as described [here] (TensorFlow Object Detection API · opencv/opencv Wiki · GitHub), I get the

KeyError: ‘first_stage_features_stride’ in line 41

no matter which of the two config files I specify: pipeline.config or mask_rcnn_inception_resnet_v2_1024x1024_coco17_gpu-8.config. I don’t know which of these two files is the correct one, different sources refer to different files. But these are the only *.config files I have.

Summary

No matter what I googled and tried, I kept running into problems. So I still don’t know where the error lies: with the conversion, or with my C++/OpenCV code. However, I strongly suspect that it is the conversion.

Can anyone help me?
Has anyone had similar problems?
Should I open an issue, because all the issues I have seen do not deal with this specific problem regarding the Mask R-CNN model? And if so, what kind of issue should I open, is it because of the conversion, or is the error with the readNetFromTensorflow method?

Thank you in advance,
Jonas

Unfortunately Tensorflow import (esp. TF2) in OpenCV is mostly unuseable as far as I know… correct me if I’m wrong.

I also know it’s difficult to find a DNN framework with good C++ bindings… So probably the easyest solution would be to keep using Tensorflow and Python - eventually with some Python-C++ binding.

Thank you for the quick reply.
Is it planned to support Tensorflow 2.x and especially the SavedModel format in future releases of OpenCV ?
I think the support would be useful for many developers, because by using C++/OpenCV on the one hand often a speedup can be achieved, on the other hand you have the advantage that for your final project only one framework has to be used, namely OpenCV, because you often need OpenCV anyway, e.g. for pre- and postprocessing steps.

I imagine it is planned (as the DNN module is one of the most used modules of OpenCV), but it’s quite a lot of work (implement every type of layer with every parameter, otherwise it will just give errors like yours).
As OpenCV is mostly a community project, it will be ready when somebody will make the effort to implement it.

[Edit] check this issue, maybe it can help: Load TensorFlow 2 model (SSD, Object Detection API) with OpenCV · Issue #19257 · opencv/opencv · GitHub

1 Like

This post looks interesting and mentions some problems and exceptions I have also encountered. However, it uses a different model, so I’m not sure if it works. I will still try to follow it step by step next week and then reply here if it worked. Thank you!

1 Like

I have the same problem, I have trained Mask RCNN with TF2 Object Detection API and now i find it impossible to optimize the savedmodel for faster inference whatever way i go for e.g OpenCV dnn module, TensorRT, TFLITE, ONNX and so on… Did anynone managed to resolve this issue yet?

No, unfortunately not.
I too have tried a number of things to port the model to OpenCV DNN. Unfortunately, all without success.

Did you use any other methods to optimize your model? (faster inference)

There’s another solution that might work, however I personally never tested it.

The ONNX format is an open and portable format for defining DNNs, and it’s compatible with different frameworks (including tensorflow 1.x, 2.x and keras, but also caffee, pytorch etc.)
OpenCV has a pretty good ONNX support.

Try to convert your tensorflow model to onnx (GitHub - onnx/tensorflow-onnx: Convert TensorFlow, Keras and Tflite models to ONNX), then use this model in OpenCV.

1 Like