DNN::Network::forward() works in Python but not C++

I have a working ONNX model (originally from TensorFlow) that works fine in opencv2 python. Its a ‘textbook’ CNN mostly for learning.

In Python it works as expected as per the examples.

In C++ Net::forward() fails with the following error (it seems the rows/cols of my image is getting confused with the number of channels).

Here is the python code:


def opencv_predict(onnx_full_model_path, class_names, test_directories):

    all_test_files = []

    for test_directory in test_directories:
        for file in glob.glob(test_directory + '/*.jpg'):
            all_test_files.append(file)

    net = cv2.dnn.readNetFromONNX(onnx_full_model_path)
    net.setPreferableBackend(cv2.dnn.DNN_BACKEND_OPENCV)
    net.setPreferableTarget(cv2.dnn.DNN_TARGET_CPU)

    for test_file in all_test_files:

        image = cv2.imread(test_file)
        image = cv2.resize(image, (IMG_HEIGHT, IMG_WIDTH))
        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

        image = np.array([image]).astype('float32') / 255.0

        net.setInput(image)
        predictions = net.forward()

        print("PREDICTIONS", predictions)

And here is the C++ code:


int main(int argc, char** argv)
{
    std::string onnxFullModelPath = "model.onnx\\model.onnx";
    Net net = readNetFromONNX(onnxFullModelPath);
 
    string testFolder = "images_training_validate\\tests";
    vector <string> allTestFiles = getFilesWithExt(testFolder, ".jpg");
    int IMG_WIDTH = 224;
    int IMG_HEIGHT = 224;

    for (auto& testFile : allTestFiles)
    {
        Mat image = imread(testFile);

        Mat blob;
        blobFromImage(image, blob, 1.0f / 255.0f, Size(IMG_WIDTH, IMG_HEIGHT), Scalar(), true, false, CV_32F);
        net.setInput(blob);
        Mat prob = net.forward();
        cout << prob << endl;
    }
}

The TF / Keras model is:

    
      model = Sequential([

            Input(shape=(IMG_HEIGHT, IMG_WIDTH, 3)),

            Conv2D(64, 3, padding='same', activation='relu'),
            Conv2D(64, 3, padding='same', activation='relu'),
            MaxPooling2D(),

            Conv2D(128, 3, padding='same', activation='relu'),
            Conv2D(128, 3, padding='same', activation='relu'),
            MaxPooling2D(),

            Conv2D(256, 3, padding='same', activation='relu'),
            Conv2D(256, 3, padding='same', activation='relu'),
            MaxPooling2D(),

            Conv2D(512, 3, padding='same', activation='relu'),
            Conv2D(512, 3, padding='same', activation='relu'),
            MaxPooling2D(),

            GlobalAveragePooling2D(),
            Dense(1024, activation='relu'),
            Dropout(.50),

            Dense(len(class_names), activation='softmax')
        ])

And the call to keras2onnx is:


def keras2onnx(keras_model, onnx_full_model_path):

    onnx_output_rev = 6
    input_spec = (tf.TensorSpec((1, IMG_HEIGHT, IMG_WIDTH, 3), tf.float32, name="input"),)
    onnx_model, _ = tf2onnx.convert.from_keras(keras_model, input_signature=input_spec, opset=onnx_output_rev, output_path=onnx_full_model_path)

    return onnx_model

One more note: I can make forward() work (but not infer correctly not surprisingly) if I try resizing my blob so the order if COLS x ROWS x CHANNELS agrees with the order as expected by TF / Keras. (But neither are correct array sizes as each module expects them.)
The below works with excepting


        Mat blob;
        blobFromImage(image, blob, 1.0f / 255.0f, Size(IMG_WIDTH, IMG_HEIGHT), Scalar(), true, false, CV_32F);

        int sz[4] = {1, IMG_WIDTH, IMG_HEIGHT, 3};
        Mat blob2 = Mat(4, sz, CV_32F);
        blob2.data = blob.data;

        net.setInput(blob2);

        Mat prob = net.forward();

Thanks for the help!

would you be so kind to us, and replace the useless screenshot of the errors with a TEXT version ? thank you.

Thank you Berak for the quick reply. Sorry for the screen capture.

(I also forgot to mention that I’m using opencv built DNN-CUDA support at 4.5.5.)

Thanks again.

  1. Below is the final error as thrown by cv::Exception::what():

OpenCV(4.5.5) error: (-2:Unspecified error) Number of input channels should be multiple of 3 but got 224 in function ‘cv::dnn::ConvolutionLayerImpl::getMemoryShapes’

  1. The below are internal CV warnings along the way to the final output exception above:
    OpenCV(4.5.5) Error: Unspecified error (Number of input channels should be multiple of 3 but got 224) in cv::dnn::ConvolutionLayerImpl::getMemoryShapes, file C:\Users\ghauschildt\Desktop\OpenCV_4_5_5\OpenCV_4_5_5\opencv-4.5.5\modules\dnn\src\layers\convolution_layer.cpp, line 405
    [ERROR:0@0.218] global C:\Users…\Desktop\OpenCV_4_5_5\OpenCV_4_5_5\opencv-4.5.5\modules\dnn\src\dnn.cpp (3875) cv::dnn::dnn4_v20211220::Net::Impl::getLayerShapesRecursively OPENCV/DNN: [Convolution]:(sequential/conv2d/BiasAdd:0): getMemoryShapes() throws exception. inputs=1 outputs=0/1 blobs=2
    [ERROR:0@0.218] global C:\Users…\Desktop\OpenCV_4_5_5\OpenCV_4_5_5\opencv-4.5.5\modules\dnn\src\dnn.cpp (3878) cv::dnn::dnn4_v20211220::Net::Impl::getLayerShapesRecursively input[0] = [ 1 224 3 224 ]
    [ERROR:0@0.218] global C:\Users…\Desktop\OpenCV_4_5_5\OpenCV_4_5_5\opencv-4.5.5\modules\dnn\src\dnn.cpp (3886) cv::dnn::dnn4_v20211220::Net::Impl::getLayerShapesRecursively blobs[0] = CV_32FC1 [ 64 3 3 3 ]
    [ERROR:0@0.219] global C:\Users…\Desktop\OpenCV_4_5_5\OpenCV_4_5_5\opencv-4.5.5\modules\dnn\src\dnn.cpp (3886) cv::dnn::dnn4_v20211220::Net::Impl::getLayerShapesRecursively blobs[1] = CV_32FC1 [ 64 1 ]
    [ERROR:0@0.220] global C:\Users…\ghauschildt\Desktop\OpenCV_4_5_5\OpenCV_4_5_5\opencv-4.5.5\modules\dnn\src\dnn.cpp (3888) cv::dnn::dnn4_v20211220::Net::Impl::getLayerShapesRecursively Exception message: OpenCV(4.5.5) C:\Users\ghauschildt\Desktop\OpenCV_4_5_5\OpenCV_4_5_5\opencv-4.5.5\modules\dnn\src\layers\convolution_layer.cpp:405: error: (-2:Unspecified error) Number of input channels should be multiple of 3 but got 224 in function ‘cv::dnn::ConvolutionLayerImpl::getMemoryShapes’

  2. Below are the exceptions from the C++ call to readNetFromOnnx(…)
    OpenCV(4.5.5) Error: Unspecified error (Number of input channels should be multiple of 3 but got 224) in cv::dnn::ConvolutionLayerImpl::getMemoryShapes, file C:\Users\ghauschildt\Desktop\OpenCV_4_5_5\OpenCV_4_5_5\opencv-4.5.5\modules\dnn\src\layers\convolution_layer.cpp, line 405
    [ERROR:0@0.218] global C:\Users\ghauschildt\Desktop\OpenCV_4_5_5\OpenCV_4_5_5\opencv-4.5.5\modules\dnn\src\dnn.cpp (3875) cv::dnn::dnn4_v20211220::Net::Impl::getLayerShapesRecursively OPENCV/DNN: [Convolution]:(sequential/conv2d/BiasAdd:0): getMemoryShapes() throws exception. inputs=1 outputs=0/1 blobs=2
    [ERROR:0@0.218] global C:\Users\ghauschildt\Desktop\OpenCV_4_5_5\OpenCV_4_5_5\opencv-4.5.5\modules\dnn\src\dnn.cpp (3878) cv::dnn::dnn4_v20211220::Net::Impl::getLayerShapesRecursively input[0] = [ 1 224 3 224 ]
    [ERROR:0@0.218] global C:\Users\ghauschildt\Desktop\OpenCV_4_5_5\OpenCV_4_5_5\opencv-4.5.5\modules\dnn\src\dnn.cpp (3886) cv::dnn::dnn4_v20211220::Net::Impl::getLayerShapesRecursively blobs[0] = CV_32FC1 [ 64 3 3 3 ]
    [ERROR:0@0.219] global C:\Users\ghauschildt\Desktop\OpenCV_4_5_5\OpenCV_4_5_5\opencv-4.5.5\modules\dnn\src\dnn.cpp (3886) cv::dnn::dnn4_v20211220::Net::Impl::getLayerShapesRecursively blobs[1] = CV_32FC1 [ 64 1 ]
    [ERROR:0@0.220] global C:\Users\ghauschildt\Desktop\OpenCV_4_5_5\OpenCV_4_5_5\opencv-4.5.5\modules\dnn\src\dnn.cpp (3888) cv::dnn::dnn4_v20211220::Net::Impl::getLayerShapesRecursively Exception message: OpenCV(4.5.5) C:\Users\ghauschildt\Desktop\OpenCV_4_5_5\OpenCV_4_5_5\opencv-4.5.5\modules\dnn\src\layers\convolution_layer.cpp:405: error: (-2:Unspecified error) Number of input channels should be multiple of 3 but got 224 in function ‘cv::dnn::ConvolutionLayerImpl::getMemoryShapes’

OpenCV(4.5.5) C:\Users\ghauschildt\Desktop\OpenCV_4_5_5\OpenCV_4_5_5\opencv-4.5.5\modules\dnn\src\layers\convolution_layer.cpp:405: error: (-2:Unspecified error) Number of input channels should be multiple of 3 but got 224 in function ‘cv::dnn::ConvolutionLayerImpl::getMemoryShapes’

OpenCV(4.5.5) Error: Unspecified error (Number of input channels should be multiple of 3 but got 224) in cv::dnn::ConvolutionLayerImpl::getMemoryShapes, file C:\Users\ghauschildt\Desktop\OpenCV_4_5_5\OpenCV_4_5_5\opencv-4.5.5\modules\dnn\src\layers\convolution_layer.cpp, line 405
[ERROR:0@0.218] global C:\Users\ghauschildt\Desktop\OpenCV_4_5_5\OpenCV_4_5_5\opencv-4.5.5\modules\dnn\src\dnn.cpp (3875) cv::dnn::dnn4_v20211220::Net::Impl::getLayerShapesRecursively OPENCV/DNN: [Convolution]:(sequential/conv2d/BiasAdd:0): getMemoryShapes() throws exception. inputs=1 outputs=0/1 blobs=2
[ERROR:0@0.218] global C:\Users\ghauschildt\Desktop\OpenCV_4_5_5\OpenCV_4_5_5\opencv-4.5.5\modules\dnn\src\dnn.cpp (3878) cv::dnn::dnn4_v20211220::Net::Impl::getLayerShapesRecursively input[0] = [ 1 224 3 224 ]
[ERROR:0@0.218] global C:\Users\ghauschildt\Desktop\OpenCV_4_5_5\OpenCV_4_5_5\opencv-4.5.5\modules\dnn\src\dnn.cpp (3886) cv::dnn::dnn4_v20211220::Net::Impl::getLayerShapesRecursively blobs[0] = CV_32FC1 [ 64 3 3 3 ]
[ERROR:0@0.219] global C:\Users\ghauschildt\Desktop\OpenCV_4_5_5\OpenCV_4_5_5\opencv-4.5.5\modules\dnn\src\dnn.cpp (3886) cv::dnn::dnn4_v20211220::Net::Impl::getLayerShapesRecursively blobs[1] = CV_32FC1 [ 64 1 ]
[ERROR:0@0.220] global C:\Users\ghauschildt\Desktop\OpenCV_4_5_5\OpenCV_4_5_5\opencv-4.5.5\modules\dnn\src\dnn.cpp (3888) cv::dnn::dnn4_v20211220::Net::Impl::getLayerShapesRecursively Exception message: OpenCV(4.5.5) C:\Users\ghauschildt\Desktop\OpenCV_4_5_5\OpenCV_4_5_5\opencv-4.5.5\modules\dnn\src\layers\convolution_layer.cpp:405: error: (-2:Unspecified error) Number of input channels should be multiple of 3 but got 224 in function ‘cv::dnn::ConvolutionLayerImpl::getMemoryShapes’

OpenCV(4.5.5) C:\Users\ghauschildt\Desktop\OpenCV_4_5_5\OpenCV_4_5_5\opencv-4.5.5\modules\dnn\src\layers\convolution_layer.cpp:405: error: (-2:Unspecified error) Number of input channels should be multiple of 3 but got 224 in function ‘cv::dnn::ConvolutionLayerImpl::getMemoryShapes’

The following warnings occur during ONNX import in C++ only:
[ INFO:0@0.093] global C:\Users…\Desktop\OpenCV_4_5_5\OpenCV_4_5_5\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (726) cv::dnn::dnn4_v20211220::ONNXImporter::populateNet DNN/ONNX: loading ONNX v4 model produced by ‘tf2onnx’:1.9.3. Number of nodes = 29, initializers = 20, inputs = 1, outputs = 1
[ INFO:0@0.094] global C:\Users…\Desktop\OpenCV_4_5_5\OpenCV_4_5_5\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (642) cv::dnn::dnn4_v20211220::ONNXImporter::parseOperatorSet DNN/ONNX: ONNX opset version = 9
[ INFO:0@0.117] global C:\Users…\Desktop\OpenCV_4_5_5\OpenCV_4_5_5\opencv-4.5.5\modules\dnn\src\onnx\onnx_importer.cpp (2891) cv::dnn::dnn4_v20211220::ONNXImporter::parseCustomLayer DNN/ONNX: unknown node type, try using custom handler for node with 1 inputs and 1 outputs: [Softmax]:(dense_1)

the main difference between your c++/py programs is the network input.
the python version just appends a batch dimension (so it’s in the expected BHWC format), while blobFromImage (called from c++) returns BCHW.

and no, you cannot simply reshape that, memory needs to be reshuffled
(you also got W,H wrong there…)

try to skip the blobFromImage() call, and do it like:

  img.convertTo(img, CV_32F, 1.0/255);
  int sz[4] = {1, IMG_HEIGHT, IMG_WIDTH, 3};
  Mat blob = Mat(4, sz, CV_32F, img.data);
  net.setInput(blob); 

Thanks. That works perfect.

so it’s in the expected BHWC format), while blobFromImage (called from c++) returns BCHW.

Understanding the above will help me out in the future as well I’m sure.

Thanks again.