Compare `Mat` from disk to memory

C++
,
1

This code’s purpose is to test if the function extracts the second-largest object in the image (the first is the background), the second-largest object is known by running the same function (checking by eye), so sample.object is the result of running this function. which is not a good way to test functions, but since I can verify that it’s actually the largest by eye It’s ok.
Now when running this code, the eq is false, (mats are not equal), so the function result mat is not the same as the disk mat (read from object.jpg), my guessing is that when saving the image in .jpg information is lost, so the function result mat is not the same as the disk mat.

extract_object.cpp:

//
// Created by jamal on 5/1/2021.
//

#include <opencv2/opencv.hpp>
#include <impl/image_utils.hpp>


using namespace cv;


typedef std::tuple<int, int> size_g;

bool sortSizes(size_g a, size_g b) {
    return std::get<1>(a) > std::get<1>(b);
}

int calculate_diameter(int x, int y) {
    return (int) sqrt(pow(x, 2) + pow(y, 2));
}

image get_object_by_index(cimage frame, cimage stats, int index);

std::optional<image> ImageUtilsImpl::extract_object(const image &frame) {

    image threshold;
    adaptiveThreshold(frame, threshold, 255, ADAPTIVE_THRESH_GAUSSIAN_C, THRESH_BINARY, 301, -7);


    image labels;
    image stats;
    image centroids;

    int nLabels = connectedComponentsWithStats(threshold, labels, stats, centroids, 8, CV_32S);


    if (nLabels < 1) return {};

    std::vector<size_g> sizes;
    for (int label = 0; label < nLabels; ++label) {
        int width = stats.at<int>(label, CC_STAT_WIDTH);
        int height = stats.at<int>(label, CC_STAT_HEIGHT);

        int diameter = calculate_diameter(width, height);
        sizes.emplace_back(size_g(label, diameter));
    }

    // sort stats based on highest diameter = sqrt( (x)^2 + (y)^2 )
    std::sort(sizes.begin(), sizes.end(), sortSizes);

    // ignore index 0 since it's always the background (the main image)
    int largestObjectIndex = std::get<0>(sizes[1]);

    image object = get_object_by_index(frame, stats, largestObjectIndex);

    return object;
}


image get_object_by_index(cimage frame, cimage stats, int index) {
    int x = stats.at<int>(index, CC_STAT_LEFT);
    int y = stats.at<int>(index, CC_STAT_TOP);
    int h = stats.at<int>(index, CC_STAT_HEIGHT);
    int w = stats.at<int>(index, CC_STAT_WIDTH);

    Rect rect(x, y, w, h);

    return frame(rect);
}

read.cpp

image ImageUtilsImpl::read(std::string filename) {
    return cv::imread(filename, cv::IMREAD_GRAYSCALE);
}

test_extract_object.cpp

typedef cv::Mat image;

bool is_mat_eq(ImageUtils *utils, image im1, image im2) {
    cv::Mat diff = im1 != im2;
    utils->write("diff.jpg", diff);
    bool eq = cv::countNonZero(diff) == 0;
    return eq;
}
TEST_P(ExtractObjectFixture, ExtractsObjectCorrectly) {
    ExtractObjectSample sample = GetParam();
    auto imageUtils = this->getInjector()->get<ImageUtils *>();


    std::optional<image> resultObject = imageUtils->extract_object(imageUtils->read(sample.mainImage));

    ASSERT_TRUE(resultObject);

    image object = *resultObject;
    image sobject = imageUtils->read(sample.object);

    EXPECT_EQ(object.rows, sobject.rows);
    EXPECT_EQ(object.cols, sobject.cols);

    std::string object_filename= this->generate_unique_file_name();
    imageUtils->write(object_filename, sobject);

    auto diskObject = imageUtils->read(object_filename);

    bool eq = is_mat_eq(imageUtils, diskObject, sobject);

    EXPECT_TRUE(eq);
}

diff.jpg:
diff

2
  • finding connected components != object detection
  • your is_mat_eq function does not work, you can never expect absolute zero difference, due to noise, compression artefacts (esp. with jpg input !). it also seems to be the wrong tool, if you already search for components
  • AVOID pointers to cv::Mat, like here: image object = *resultObject;, you’re messing up internal refcounting
  • please don’t run wild on c++xx things, your code is horribly overcomplicated,
    – a lot of noise, but nothing achieved

can you show us the original images
(and try to explain better, what you want to find out here,
what is the goal of your program) ?

maybe then we can help you to find a better idea

1 Like
3

why aren’t you saving your pictures as PNG, or anything that isn’t JPEG?

also… how about doing this instead of an exact comparison:

    cv::Mat diff = cv::absdiff(im1,  im2);
4

  1. I’m trying to find the largest object (without bg) in the image. then it’s classified by Keras later on.

  2. If I’m using imencode, write the mat content to a buffer then turning it back to a mat, does it work?

  3. image object = *resultObject is not a pointer to an image, typedef cv::Mat image, resultObject is of type std::optional and this is how I access the content.

  4. I can’t show the original image due to privacy (images are from CCTV), but it’s a house entrance.

5

imencode/imwrite behave the same, apart from the obvious difference.

to reiterate, you have to use a file format that isn’t lossy. JPEG is lossy. PNG is lossless.

additionally… ImageUtils *utils, why do you need to instantiate something (singleton?) that ought to be just a bunch of functions in a namespace?

6

So my “observation” was correct. there is information loss
I have implemented the changes to PNG, and it’s working as expected.

test_extract_object.cpp:

bool is_mat_eq(image &im1, image &im2) {
    cv::Mat diff;
    cv::absdiff(im1, im2, diff);
    bool eq = cv::countNonZero(diff) == 0;
    return eq;
}

TEST_P(ExtractObjectFixture, ExtractsObjectCorrectly) {
    ExtractObjectSample sample = GetParam();
    auto imageUtils = this->getInjector()->get<ImageUtils *>();


    std::optional<image> resultObject = imageUtils->extract_object(imageUtils->read(sample.mainImage));

    ASSERT_TRUE(resultObject);

    image object = *resultObject;
    image sample_object = imageUtils->read(sample.object);
    image fake_object = imageUtils->read(sample.fake_object);

    EXPECT_EQ(object.rows, sample_object.rows);
    EXPECT_EQ(object.cols, sample_object.cols);

    bool eq = is_mat_eq(object, sample_object);
    EXPECT_TRUE(eq);

    eq = is_mat_eq(sample_object, fake_object);
    EXPECT_FALSE(eq);

    eq = is_mat_eq(object, fake_object);
    EXPECT_FALSE(eq);
}
7

I’m using IoC for tests, organizing the functions, and for shared configurations (without making them static) and because it’s a part of another class,
these codes are tests, not the real app impl

8

not at all, what your question title says…
you don’t need 2 images for that. what’s the second one for ? your question is already unclear about it
and what does your mat_eq function actually compare ?

why on earth would someone do this ?

whole idea look murky. either throw an exception, or return something valid

9

you asked what the application is trying to do.

this is a test case, (I’m writing unit testing) and to test if the result is correct I’m comparing a pre-computed results with current, (to check if the function still works)

I want to check if the result of the function is the anticipated result (
a = f(x)
b = x
checking if f(b) = a ) it’s for unit testing to make sure I don’t break anything.

My thinking was:
jpg = F(image) where F is imencode/imwrite
so If I want to check if image == image2 where image is unknown but image2 is known
I need to do:
jpg2 = F(image2)
bool eq = jpg2 == jpg; means F(image2) == F(image) which should result to image2 == image2,
but looks like it’s not the case with jpg (F is not linear)

I’ll look into that.