Problems loading TensorFlow model in C++

lml2204 · July 3, 2023, 12:46pm

Hi,
I’m quite new to OpenCV, so excuse my noob questions.

I have to make my own object detection model, so far so good, I followed this colab tutorial(or rather notebook), and got a saved_model.pb.

From what I read online, this doesn’t work with OpenCV, so I tried converting it to frozen_graph.pb and frozen_graph.pbtxt.

The problem is, I can’t figure out how to do it properly.
So far I have tried 3 times, and all I get is “Unhandled exception at 0x00007FF84ABCCF19 in testyolo.exe: Microsoft C++ exception: cv::Exception at memory location 0x00000084A32FDE60.”
(ignore the “testyolo” part, it’s a testing program, I decided to go with tensorflow)

How do you convert saved_model.pb to frozen_inference_graph.pb and pbtxt to use the model in OpenCV?
The program works fine with “ssd_mobilenet_v2_coco_2018_03_29”, but I need a custom model for my application.
Or how does one retrain a model on a new dataset?

Again, sorry if it’s a stupid or easy question, I just can’t find a straight answer.

laurent.berger · July 3, 2023, 1:53pm

Do you run your model in C++ or python?

before loading your model try to insert in C++


#include <opencv2/core/utils/logger.hpp>


utils::logging::setLogLevel(utils::logging::LogLevel::LOG_LEVEL_VERBOSE);

then post full output of your program

lml2204 · July 3, 2023, 2:43pm

C++

[DEBUG:0@0.072] global tf_importer.cpp:2723 cv::dnn::dnn4_v20221220::`anonymous-namespace’::TFImporter::TFImporter DNN/TF: processing TensorFlow model from file: H:/proj/frozen_graph.pb
[DEBUG:0@0.163] global tf_importer.cpp:2728 cv::dnn::dnn4_v20221220::`anonymous-namespace’::TFImporter::TFImporter DNN/TF: processing TensorFlow config from file: H:/proj/frozen_graph.pbtxt
[ INFO:0@0.553] global tf_importer.cpp:3014 cv::dnn::dnn4_v20221220::`anonymous-namespace’::TFImporter::populateNet DNN/TF: parsing model produced by TF v987 (min_consumer=0). Number of nodes = 8792
[ INFO:0@0.553] global tf_importer.cpp:3021 cv::dnn::dnn4_v20221220::`anonymous-namespace’::TFImporter::populateNet DNN/TF: parsing config produced by TF v987 (min_consumer=0). Number of nodes = 8792
[DEBUG:0@0.568] global tf_importer.cpp:3024 cv::dnn::dnn4_v20221220::`anonymous-namespace’::TFImporter::populateNet DNN/TF: RemoveIdentityOps(model) => 7494 nodes
[DEBUG:0@0.580] global tf_importer.cpp:3026 cv::dnn::dnn4_v20221220::`anonymous-namespace’::TFImporter::populateNet DNN/TF: RemoveIdentityOps(config) => 7494 nodes

crackwitz · July 3, 2023, 7:28pm

catch it, inspect it.

lml2204 · July 4, 2023, 3:07pm

OpenCV(4.7.0) F:\ChromeDownloads3\opencv-4.7.0\opencv-4.7.0\modules\dnn\src\tensorflow\tf_graph_simplifier.cpp:1059: error: (-215:Assertion failed) permIds.size() == net.node_size() in function ‘cv::dnn::dnn4_v20221220::sortByExecutionOrder’

[ INFO:1@0.614] global plugin_loader.impl.hpp:74 cv::plugin::impl::DynamicLib::libraryRelease unload G:\opencv2\install\x64\vc17\bin\opencv_videoio_ffmpeg470_64.dll

lml2204 · July 9, 2023, 9:54am

Anyone?
Will 4.8.0 help?

crackwitz · July 9, 2023, 11:42am

if you think updating the library would help, then do it. you don’t need to ask.

github.com/opencv/opencv

Load TensorFlow 2 model (SSD, Object Detection API) with OpenCV

opened 04:24PM - 04 Jan 21 UTC

closed 08:55AM - 03 Feb 21 UTC

golden96371

feature category: dnn

##### System information (version) - OpenCV => 4.5.1 (python) - TensorFlow => …2.3.1 (python) - Operating System / Platform => Windows 10 (64 Bit) - Compiler => Visual Studio 2019 ##### Detailed description After training a SSD MobileNet v2 using the TensorFlow Object Detection API with TensorFlow2, I converted the SavedModel into a FrozenGraph in order to make it compatible with OpenCV. I am using following code to load the model: ``` python import numpy as np import cv2 as cv net = cv.dnn.readNet('frozen_graph.pb') ``` This gives following error message: ``` Traceback (most recent call last): File "<stdin>", line 1, in <module> cv2.error: OpenCV(4.5.1) C:\Users\appveyor\AppData\Local\Temp\1\pip-req-build-oduouqig\opencv\modules\dnn\src\tensorflow\tf_importer.cpp:578: error: (-2:Unspecified error) More than one input is Const op in function 'cv::dnn::dnn4_v20201117::`anonymous-namespace'::TFImporter::getConstBlob' ``` The script `opencv/tf_text_graph_ssd.py` runs into an assertion: ``` Traceback (most recent call last): File "tf_text_graph_ssd_original.py", line 405, in <module> createSSDGraph(args.input, args.config, args.output) File "tf_text_graph_ssd_original.py", line 292, in createSSDGraph assert(num_matched_layers == num_layers) AssertionError ``` After fixing the assertion by adapting the layer names in the script, the text graph could be created. After loading the model with ``` python net = cv.dnn.readNet('frozen_graph.pb', 'graph.pbtxt') ``` there is a different error message ``` Traceback (most recent call last): File "<stdin>", line 1, in <module> cv2.error: OpenCV(4.5.1) C:\Users\appveyor\AppData\Local\Temp\1\pip-req-build-oduouqig\opencv\modules\dnn\src\tensorflow\tf_graph_simplifier.cpp:1019: error: (-215:Assertion failed) permIds.size() == net.node_size() in function 'cv::dnn::dnn4_v20201117::sortByExecutionOrder' ``` I attach my frozen_graph.pb and my pipeline.config. [model.zip](https://github.com/opencv/opencv/files/5765665/model.zip) ##### Steps to reproduce 1.) Train a SSD MobileNet v2 using the [TensorFlow Object Detection API](https://github.com/tensorflow/models/tree/master/research/object_detection) and export it to a SavedModel ``` # Train the network python model_main_tf2.py --model_dir="training" --pipeline_config_path="training/pipeline.config" # Export the network python exporter_main_v2.py --input_type=image_tensor --pipeline_config_path="training/pipeline.config" --trained_checkpoint_dir="training" --output_directory="training/model" ``` 2.) Convert the SavedModel to a FrozenGraph ``` python import tensorflow as tf from tensorflow.python.framework.convert_to_constants import convert_variables_to_constants_v2 loaded = tf.saved_model.load('training/model/saved_model') infer = loaded.signatures['serving_default'] f = tf.function(infer).get_concrete_function(input_tensor=tf.TensorSpec(shape=[1, None, None, 3], dtype=tf.uint8)) f2 = convert_variables_to_constants_v2(f) graph_def = f2.graph.as_graph_def() # Export frozen graph with tf.io.gfile.GFile('frozen_graph.pb', 'wb') as f: f.write(graph_def.SerializeToString()) ``` 3.) Try to load the FrozenGraph with OpenCV as described above: failed Anyone can reproduce the issue and help me to fix it?

lml2204 · July 9, 2023, 12:48pm

Well, I have tried building it maybe 10 times, on 2 different computers, and on both I’m missing the install folder.
Any ideas?
I’m following Nielsen’s tutorial.
I’d ask him on his discord, however it’s private.

lml2204 · July 11, 2023, 11:26am

So, CUDA 12.2 is incompatible and caused the issue with the folder

I saw in the changenotes that there’s now tflite support, so I trained a tflite model, and I get:

G:\opencv\opencv-4.8.0\modules\dnn\src\tensorflow\tf_importer.cpp:2999: error: (-215:Assertion failed) netBinSize || netTxtSize in function ‘cv::dnn::dnn4_v20230620::`anonymous-namespace’::TFImporter::populateNet’

crackwitz · July 11, 2023, 12:49pm

you should look up that line in the opencv source and figure out the context of the assertion.

I would guess that one or both of the required input files were empty or not there.

lml2204 · July 11, 2023, 4:09pm

my bad, I hadn’t changed the function to readNetFromTFLite, as I thought it’s still Tensorflow

Error is actually

[ERROR:0@0.238] global tflite_importer.cpp:243 cv::dnn::dnn4_v20230620::TFLiteImporter::populateNet DNN/TFLite: Problem during import of operator [PACK]:(ssd_mobile_net_v2_fpn_keras_feature_extractor/FeatureMaps/top_down/nearest_neighbor_upsampling/nearest_neighbor_upsampling/w_stack) (65/157). Exception: OpenCV(4.8.0) G:\opencv\opencv-4.8.0\modules\dnn\src\tflite\tflite_importer.cpp:235: error: (-213:The function/feature is not implemented) Unsupported operator type PACK in function ‘cv::dnn::dnn4_v20230620::TFLiteImporter::populateNet’

OpenCV(4.8.0) G:\opencv\opencv-4.8.0\modules\dnn\src\tflite\tflite_importer.cpp:235: error: (-213:The function/feature is not implemented) Unsupported operator type PACK in function ‘cv::dnn::dnn4_v20230620::TFLiteImporter::populateNet’

lml2204 · July 11, 2023, 4:10pm

Line 235 is

lml2204 · July 11, 2023, 8:18pm

Tensorflow model error(not tflite)

[ERROR:0@2.428] global tf_importer.cpp:3172 cv::dnn::dnn4_v20230620::anonymous-namespace'::TFImporter::parseNode DNN/TF: Can't parse layer for node='StatefulPartitionedCall/StatefulPartitionedCall/Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/add_179' of type='Add'. Exception: OpenCV(4.8.0) G:\opencv\opencv-4.8.0\modules\dnn\src\tensorflow\tf_importer.cpp:2837: error: (-2:Unspecified error) More than one input is Const op in function 'cv::dnn::dnn4_v20230620::anonymous-namespace’::TFImporter::getConstBlob’

OpenCV(4.8.0) G:\opencv\opencv-4.8.0\modules\dnn\src\tensorflow\tf_importer.cpp:2837: error: (-2:Unspecified error) More than one input is Const op in function ‘cv::dnn::dnn4_v20230620::`anonymous-namespace’::TFImporter::getConstBlob’

lml2204 · July 11, 2023, 8:19pm

crackwitz · July 14, 2023, 1:47pm

4 posts were split to a new topic: AI-generated spam and ensuing confusion

Topic		Replies	Views
Problems importing TensorFlow Object Detection Models into OpenCV DNN C++ C++ dnn , tensorflow	2	1069	March 5, 2021
I want to import TensorFlow 2.x - Object Detection - Mask R-CNN Model in OpenCV DNN API C++ dnn , tensorflow	9	3604	December 14, 2021
Loading Tensorflow Models Using Opencv and C++ C++ dnn , tensorflow	2	1427	April 20, 2023
Use TF2 saved_model.pb with OpenCV dnn module Python dnn	0	394	July 22, 2021
Could not convert Tensorflow Freeze Model for OpenCV Dnn 4.5.0-dev Python dnn	0	494	December 30, 2020

Problems loading TensorFlow model in C++

Related topics