Java matchTemplate with mask and TM_CCORR_NORMED fails to find match (Infinity/NaN)

kinram · August 6, 2024, 10:22pm

Hi,

Note: I’m a new user on the forum, so I’m limited to 1 image for now. I need a lot of images to explain my use case, could any of the admins be so kind to lift that restriction so I can add the images? Pretty please, the only other way is that I would document it in a readme (using the same markdown) in a github repo and link to it from here, but I think that is worse.

I’ve been breaking my head over this for hours, and I either don’t understand how the mask template works, or there is some kind of bug. I tried with the last 5 versions of OpenCV (4.9.0, 4.8.1, 4.7.0, 4.6.0) but they all have the same behaviour.

Let me first explain what I’m trying to do, which is pretty straightforward: I want to find a template using a mask, using TM_CCORR_NORMED (to get values between 0 and 1), which means that I want to ignore some parts of the template (the transparent pixels).

This is the 20x20 template (needle):
needle_white
It looks like a window, it has a few white pixels (the borders), and a few transparent pixels.

Trying to find it in an image (haystack) and drawing a yellow border around it sometimes works fine.

4 examples that work fine:
todo: insert images, new user restrictions

However, it fails to find it on the red part:
todo: insert images, new user restrictions

I tried moving the red part to a different location, but it seems color-related rather than location-related, because this also fails to find it there:
todo: insert images, new user restrictions

I tried to investigate the results, and it looks like +Infinity and -Infinity (results of dividing by 0) and NaN (not sure why that occurs) are breaking the normalization.

Out of the 4 matches that seem to work fine, only 2 are “really” OK.

the one with the needle in the middle (over all the colors), where the best match is at the bottom left (note that the heatmap does not have the same size as the haystack, but haystack-needle, so in the haystack it’s more towards the top-left)
todo: insert images, new user restrictions
the one on black, peak at bottom left:
todo: insert images, new user restrictions

These are where I draw the top-left corner of the yellow box (± 1 pixel).

However, the heatmaps for the other 2 matches are black. but the location of the best match is correct. The last 2 don’t find the correct location either.

I added a lot of logging to get an idea of what is happening, so let me walk you through this use case:
todo: insert images, new user restrictions

Here is some code and logs:

First, I read both needle and haystack as IMREAD_COLOR, ignoring any alpha channel (transparency):

private static void writeImage(String prefix, String imageName, Mat img) {
    String imageFilename = prefix + imageName + ".png";
    Imgcodecs.imwrite(imageFilename, img);
    l.info("Wrote image '{}'", imageFilename);
}

Log:

2024-08-06 23:24:43 INFO Read image 'needle_white' using codec 1, result Mat: Mat [ 20*20*CV_8UC3, isCont=true, isSubmat=false, nativeObj=0x222f65fcbc0, dataAddr=0x222f65abdc0 ]
2024-08-06 23:24:43 INFO Read image 'haystack_white_needle_on_red' using codec 1, result Mat: Mat [ 50*50*CV_8UC3, isCont=true, isSubmat=false, nativeObj=0x222f65fc220, dataAddr=0x222f5e2ab40 ]

I then re-read the needle INCLUDING the alpha channel, and create a mask out of it. Code:

public static Mat createMask(String imageName) {
    Mat img = readImage(imageName, Imgcodecs.IMREAD_UNCHANGED);

    Mat mask = new Mat(img.size(), CvType.CV_8UC1);
    for (int y = 0; y < img.rows(); y++) {
        for (int x = 0; x < img.cols(); x++) {
            // OpenCV uses BGR order, so the alpha channel is the 4th channel
            double alphaValue = img.get(y, x)[3];
            if (alphaValue > 0) {
                mask.put(y, x, 255); // Consider this pixel
            } else {
                mask.put(y, x, 0); // Ignore this pixel
            }
        }
    }
    return mask;
}

Docs say that any non-0 value is not ignored, I used 0 and 255 but tried 0 and 1 too.

Logs:

2024-08-06 23:24:43 INFO Read image 'needle_white' using codec -1, result Mat: Mat [ 20*20*CV_8UC4, isCont=true, isSubmat=false, nativeObj=0x222f65fc300, dataAddr=0x222f6a810c0 ]

Note that 8UC4 (4th channel is alpha).

I then use matchTemplate, and also try normalizing it (0-100) to understand what happens.

Code:

Mat matchResult = new Mat();
Imgproc.matchTemplate(haystackImg, needleImg, matchResult, matchMethod, mask);
l.info("matchResult:           {}", matchResult);

Mat normalizedMatchResult = new Mat();
Core.normalize(matchResult, normalizedMatchResult, 0, 100, Core.NORM_MINMAX, CvType.CV_32F);
l.info("normalizedMatchResult: {}", matchResult);

Logs:

2024-08-06 23:24:43 INFO matchResult:           Mat [ 31*31*CV_32FC1, isCont=true, isSubmat=false, nativeObj=0x222f65fd790, dataAddr=0x222f6c9cb00 ]
2024-08-06 23:24:43 INFO normalizedMatchResult: Mat [ 31*31*CV_32FC1, isCont=true, isSubmat=false, nativeObj=0x222f65fd790, dataAddr=0x222f6c9cb00 ]

Nothing special so far. The heatmap generation also happens here, the black heatmap suggests that we’re not staying withing the 0-255 boundaries, so the normalization fails. Code:

Mat heatmap = new Mat();
Core.normalize(matchResult, heatmap, 0, 255, Core.NORM_MINMAX, CvType.CV_8U);
writeImage(prefix, "04_heatmap", heatmap);

I then use minMaxLoc as documented, and I tried both on the original matchResult and my custom-normalized normalizedMatchResult.

Code:

Core.MinMaxLocResult mmr = Core.minMaxLoc(matchResult);
l.info("minMaxLoc matchResult:           maxVal {}, maxLoc {}, minVal {}, minLoc {}", mmr.maxVal, mmr.maxLoc, mmr.minVal, mmr.minLoc);
Core.MinMaxLocResult mmr2 = Core.minMaxLoc(normalizedMatchResult);
l.info("minMaxLoc normalizedMatchResult: maxVal {}, maxLoc {}, minVal {}, minLoc {}", mmr2.maxVal, mmr2.maxLoc, mmr2.minVal, mmr2.minLoc);

Logs:

2024-08-06 23:24:43 INFO minMaxLoc matchResult:           maxVal Infinity, maxLoc {2.0, 27.0}, minVal -Infinity, minLoc {5.0, 30.0}
2024-08-06 23:24:43 INFO minMaxLoc normalizedMatchResult: maxVal -Infinity, maxLoc {0.0, 0.0}, minVal Infinity, minLoc {0.0, 0.0}

This isn’t right, maxVal should be between 0.0 and 1.0 when using TM_CCORR_NORMED, but it’s Infinity.

Also, `` is not correct, it’s close: it should be (3,28) and not (2,27), but there is more.

I custom-printed the values in the result. I replace +Infinity and -Infinity with “+In” and “-In”, I printed NaN as “nan”, and I rounded the values I expect to be between 0 and 1.
This is the result:

2024-08-06 23:24:43 INFO   y  0:  0,66  0,66  0,74  0,82  0,74  0,66  0,66  0,66  0,66  0,66  0,66  0,70  0,75  0,69  0,62  0,62  0,62  0,62  0,62  0,62  0,64  0,67  0,62  0,58  0,58  0,58  0,58  0,58  0,58  0,58  0,58 
2024-08-06 23:24:43 INFO   y  1:  0,66  0,66  0,75  0,83  0,75  0,66  0,66  0,66  0,66  0,66  0,66  0,71  0,75  0,69  0,62  0,62  0,62  0,62  0,62  0,62  0,65  0,67  0,63  0,58  0,58  0,58  0,58  0,58  0,58  0,58  0,58 
2024-08-06 23:24:43 INFO   y  2:  0,74  0,75  0,84  0,92  0,84  0,75  0,74  0,74  0,73  0,72  0,72  0,76  0,81  0,74  0,67  0,66  0,65  0,65  0,64  0,63  0,65  0,68  0,63  0,58  0,58  0,58  0,58  0,58  0,58  0,58  0,58 
2024-08-06 23:24:43 INFO   y  3:  0,82  0,83  0,92  1,00  0,92  0,83  0,82  0,80  0,79  0,78  0,77  0,81  0,86  0,79  0,71  0,70  0,69  0,67  0,66  0,64  0,66  0,68  0,63  0,58  0,58  0,58  0,58  0,58  0,58  0,58  0,58 
2024-08-06 23:24:43 INFO   y  4:  0,74  0,75  0,84  0,92  0,84  0,75  0,74  0,74  0,73  0,72  0,72  0,76  0,81  0,74  0,67  0,66  0,65  0,65  0,64  0,63  0,65  0,68  0,63  0,58  0,58  0,58  0,58  0,58  0,58  0,58  0,58 
2024-08-06 23:24:43 INFO   y  5:  0,66  0,66  0,75  0,83  0,75  0,66  0,66  0,66  0,66  0,66  0,66  0,71  0,75  0,69  0,62  0,62  0,62  0,62  0,62  0,62  0,65  0,67  0,63  0,58  0,58  0,58  0,58  0,58  0,58  0,58  0,58 
2024-08-06 23:24:43 INFO   y  6:  0,66  0,66  0,74  0,82  0,74  0,66  0,66  0,66  0,66  0,66  0,66  0,70  0,75  0,69  0,62  0,62  0,62  0,62  0,62  0,62  0,64  0,67  0,62  0,58  0,58  0,58  0,58  0,58  0,58  0,58  0,58 
2024-08-06 23:24:43 INFO   y  7:  0,64  0,64  0,71  0,78  0,71  0,64  0,64  0,64  0,64  0,65  0,65  0,69  0,73  0,68  0,62  0,63  0,63  0,63  0,64  0,64  0,66  0,69  0,65  0,61  0,62  0,62  0,62  0,62  0,62  0,62  0,62 
2024-08-06 23:24:43 INFO   y  8:  0,61  0,61  0,68  0,75  0,68  0,61  0,61  0,62  0,62  0,63  0,64  0,68  0,72  0,67  0,63  0,63  0,64  0,65  0,65  0,66  0,68  0,71  0,68  0,65  0,66  0,66  0,67  0,67  0,67  0,67  0,67 
2024-08-06 23:24:43 INFO   y  9:  0,60  0,60  0,67  0,73  0,67  0,60  0,60  0,61  0,62  0,63  0,64  0,67  0,71  0,67  0,63  0,63  0,64  0,65  0,66  0,67  0,69  0,71  0,68  0,65  0,66  0,67  0,68  0,68  0,68  0,68  0,68 
2024-08-06 23:24:43 INFO   y 10:  0,59  0,59  0,65  0,71  0,65  0,59  0,59  0,61  0,62  0,63  0,64  0,67  0,70  0,66  0,62  0,63  0,65  0,66  0,67  0,67  0,69  0,71  0,69  0,66  0,67  0,68  0,69  0,69  0,69  0,69  0,69 
2024-08-06 23:24:43 INFO   y 11:  0,63  0,64  0,70  0,75  0,70  0,64  0,63  0,65  0,66  0,66  0,66  0,69  0,72  0,69  0,65  0,65  0,66  0,68  0,68  0,68  0,70  0,71  0,69  0,67  0,67  0,69  0,71  0,71  0,71  0,71  0,71 
2024-08-06 23:24:43 INFO   y 12:  0,67  0,68  0,74  0,79  0,74  0,68  0,67  0,68  0,69  0,69  0,69  0,72  0,74  0,71  0,67  0,67  0,68  0,69  0,69  0,69  0,70  0,71  0,69  0,67  0,68  0,70  0,72  0,72  0,72  0,72  0,72 
2024-08-06 23:24:43 INFO   y 13:  0,60  0,60  0,66  0,71  0,66  0,60  0,60  0,62  0,64  0,64  0,64  0,67  0,70  0,67  0,64  0,64  0,66  0,68  0,68  0,68  0,70  0,71  0,70  0,68  0,69  0,71  0,73  0,73  0,73  0,73  0,73 
2024-08-06 23:24:43 INFO   y 14:  0,51  0,51  0,57  0,62  0,57  0,51  0,51  0,55  0,58  0,58  0,59  0,63  0,66  0,63  0,60  0,61  0,64  0,66  0,67  0,68  0,70  0,71  0,70  0,69  0,69  0,72  0,74  0,74  0,74  0,74  0,74 
2024-08-06 23:24:43 INFO   y 15:  0,50  0,50  0,55  0,60  0,55  0,50  0,50  0,54  0,58  0,58  0,59  0,62  0,65  0,62  0,60  0,61  0,64  0,67  0,68  0,68  0,70  0,72  0,70  0,69  0,70  0,73  0,75  0,75  0,75  0,75  0,75 
2024-08-06 23:24:43 INFO   y 16:  0,47  0,47  0,51  0,55  0,51  0,47  0,47  0,51  0,56  0,57  0,58  0,61  0,63  0,62  0,60  0,61  0,65  0,68  0,69  0,70  0,72  0,74  0,73  0,72  0,73  0,76  0,79  0,79  0,79  0,79  0,79 
2024-08-06 23:24:43 INFO   y 17:  0,43  0,43  0,47  0,50  0,47  0,43  0,43  0,49  0,53  0,55  0,57  0,59  0,62  0,61  0,61  0,62  0,66  0,70  0,71  0,72  0,74  0,76  0,76  0,75  0,77  0,80  0,83  0,83  0,83  0,83  0,83 
2024-08-06 23:24:43 INFO   y 18:  0,42  0,42  0,45  0,47  0,45  0,42  0,42  0,48  0,53  0,55  0,57  0,59  0,61  0,61  0,61  0,62  0,66  0,70  0,72  0,73  0,74  0,76  0,76  0,76  0,77  0,81  0,84  0,84  0,84  0,84  0,84 
2024-08-06 23:24:43 INFO   y 19:  0,41  0,41  0,42  0,44  0,42  0,41  0,41  0,47  0,53  0,55  0,57  0,58  0,60  0,60  0,61  0,62  0,67  0,71  0,72  0,73  0,75  0,76  0,76  0,77  0,78  0,81  0,85  0,85  0,85  0,85  0,85 
2024-08-06 23:24:43 INFO   y 20:  0,43  0,44  0,45  0,46  0,45  0,44  0,43  0,50  0,55  0,57  0,58  0,59  0,60  0,61  0,62  0,63  0,68  0,72  0,73  0,74  0,75  0,76  0,77  0,77  0,78  0,82  0,86  0,86  0,86  0,86  0,86 
2024-08-06 23:24:43 INFO   y 21:  0,45  0,46  0,47  0,47  0,47  0,46  0,45  0,52  0,57  0,58  0,59  0,60  0,61  0,62  0,63  0,64  0,69  0,73  0,74  0,75  0,75  0,76  0,77  0,78  0,79  0,83  0,87  0,87  0,87  0,87  0,87 
2024-08-06 23:24:43 INFO   y 22:  0,37  0,38  0,38  0,39  0,38  0,38  0,37  0,46  0,53  0,54  0,55  0,57  0,58  0,59  0,61  0,62  0,67  0,72  0,73  0,74  0,75  0,76  0,77  0,78  0,79  0,84  0,88  0,88  0,88  0,88  0,88 
2024-08-06 23:24:43 INFO   y 23:  0,27  0,27  0,27  0,27  0,27  0,27  0,27  0,39  0,48  0,50  0,52  0,53  0,55  0,57  0,58  0,60  0,66  0,72  0,73  0,74  0,75  0,76  0,78  0,79  0,80  0,85  0,89  0,89  0,89  0,89  0,89 
2024-08-06 23:24:43 INFO   y 24:  0,26  0,26  0,26  0,26  0,26  0,26  0,26  0,38  0,48  0,50  0,51  0,53  0,55  0,57  0,58  0,60  0,66  0,72  0,73  0,75  0,76  0,77  0,78  0,79  0,81  0,85  0,90  0,90  0,90  0,90  0,90 
2024-08-06 23:24:43 INFO   y 25:  0,18  0,18  0,18  0,18  0,18  0,18  0,18  0,34  0,45  0,48  0,50  0,52  0,54  0,57  0,59  0,60  0,67  0,73  0,75  0,76  0,78  0,79  0,81  0,82  0,84  0,89  0,93  0,93  0,93  0,93  0,93 
2024-08-06 23:24:43 INFO   y 26: -0,00 -0,00 -0,00  nan   nan   nan   nan   0,30  0,43  0,46  0,49  0,51  0,54  0,57  0,59  0,61  0,68  0,75  0,76  0,78  0,80  0,82  0,83  0,85  0,87  0,92  0,97  0,97  0,97  0,97  0,97 
2024-08-06 23:24:43 INFO   y 27:  nan   nan   +In   nan   nan   nan   nan   0,30  0,43  0,46  0,49  0,51  0,54  0,57  0,59  0,61  0,68  0,75  0,76  0,78  0,80  0,82  0,83  0,85  0,87  0,92  0,97  0,97  0,97  0,97  0,97 
2024-08-06 23:24:43 INFO   y 28:   0    0,00  0,00  nan   0,00  nan   nan   0,30  0,43  0,46  0,49  0,51  0,54  0,57  0,59  0,61  0,68  0,75  0,76  0,78  0,80  0,82  0,83  0,85  0,87  0,92  0,97  0,97  0,97  0,97  0,97 
2024-08-06 23:24:43 INFO   y 29:  nan   nan   nan   nan   nan   nan   nan   0,30  0,43  0,46  0,49  0,51  0,54  0,57  0,59  0,61  0,68  0,75  0,76  0,78  0,80  0,82  0,83  0,85  0,87  0,92  0,97  0,97  0,97  0,97  0,97 
2024-08-06 23:24:43 INFO   y 30:  0,00  0,00  0,00 -0,00   0    -In   -In   0,30  0,43  0,46  0,49  0,51  0,54  0,57  0,59  0,61  0,68  0,75  0,76  0,78  0,80  0,82  0,83  0,85  0,87  0,92  0,97  0,97  0,97  0,97  0,97

As you can see, the bottom left is where a bunch of infinity/NaN/0 are occuring, which I think is throwing off the minMaxLoc.

Since this example is still close, let’s look at the one where I moved the red part to the bottom right:
todo: insert images, new user restrictions

It has similar Infinity/NaN fields, but the best match location is even further off:

2024-08-06 23:24:43 INFO minMaxLoc matchResult:           maxVal Infinity, maxLoc {1.0, 30.0}, minVal -Infinity, minLoc {6.0, 28.0}
2024-08-06 23:24:43 INFO minMaxLoc normalizedMatchResult: maxVal -Infinity, maxLoc {0.0, 0.0}, minVal Infinity, minLoc {0.0, 0.0}

In the bottom right, similar strange things are happening:

2024-08-06 23:24:43 INFO   y  0:  0,97  0,97  0,97  0,97  0,97  0,97  0,97  0,93  0,90  0,89  0,88  0,87  0,86  0,85  0,84  0,83  0,79  0,75  0,74  0,73  0,72  0,71  0,69  0,68  0,67  0,62  0,58  0,58  0,58  0,58  0,58 
2024-08-06 23:24:43 INFO   y  1:  0,97  0,97  0,97  0,97  0,97  0,97  0,97  0,93  0,90  0,89  0,88  0,87  0,86  0,85  0,84  0,83  0,79  0,75  0,74  0,73  0,72  0,71  0,69  0,68  0,67  0,62  0,58  0,58  0,58  0,58  0,58 
2024-08-06 23:24:43 INFO   y  2:  0,97  0,97  0,97  0,97  0,97  0,97  0,97  0,93  0,90  0,89  0,88  0,87  0,86  0,85  0,84  0,83  0,79  0,75  0,74  0,73  0,72  0,71  0,69  0,68  0,67  0,62  0,58  0,58  0,58  0,58  0,58 
2024-08-06 23:24:43 INFO   y  3:  0,97  0,97  0,97  0,97  0,97  0,97  0,97  0,93  0,90  0,89  0,88  0,87  0,86  0,85  0,84  0,83  0,79  0,75  0,74  0,73  0,72  0,71  0,69  0,68  0,67  0,62  0,58  0,58  0,58  0,58  0,58 
2024-08-06 23:24:43 INFO   y  4:  0,97  0,97  0,97  0,97  0,97  0,97  0,97  0,93  0,90  0,89  0,88  0,87  0,86  0,85  0,84  0,83  0,79  0,75  0,74  0,73  0,72  0,71  0,69  0,68  0,67  0,62  0,58  0,58  0,58  0,58  0,58 
2024-08-06 23:24:43 INFO   y  5:  0,97  0,97  0,97  0,97  0,97  0,97  0,97  0,93  0,90  0,89  0,88  0,87  0,86  0,85  0,84  0,83  0,79  0,75  0,74  0,73  0,72  0,71  0,69  0,68  0,67  0,62  0,58  0,58  0,58  0,58  0,58 
2024-08-06 23:24:43 INFO   y  6:  0,97  0,97  0,97  0,97  0,97  0,97  0,97  0,93  0,90  0,89  0,88  0,87  0,86  0,85  0,84  0,83  0,79  0,75  0,74  0,73  0,72  0,71  0,69  0,68  0,67  0,62  0,58  0,58  0,58  0,58  0,58 
2024-08-06 23:24:43 INFO   y  7:  0,92  0,92  0,92  0,92  0,92  0,92  0,92  0,89  0,85  0,85  0,84  0,83  0,82  0,81  0,81  0,80  0,76  0,73  0,72  0,71  0,70  0,69  0,68  0,67  0,66  0,62  0,58  0,58  0,58  0,58  0,58 
2024-08-06 23:24:43 INFO   y  8:  0,87  0,87  0,87  0,87  0,87  0,87  0,87  0,84  0,81  0,80  0,79  0,79  0,78  0,78  0,77  0,77  0,73  0,70  0,69  0,69  0,68  0,67  0,67  0,66  0,66  0,62  0,58  0,58  0,58  0,58  0,58 
2024-08-06 23:24:43 INFO   y  9:  0,85  0,85  0,85  0,85  0,85  0,85  0,85  0,82  0,79  0,79  0,79  0,78  0,78  0,78  0,77  0,77  0,74  0,71  0,71  0,71  0,70  0,70  0,69  0,69  0,69  0,66  0,63  0,63  0,63  0,63  0,63 
2024-08-06 23:24:43 INFO   y 10:  0,83  0,83  0,83  0,83  0,83  0,83  0,83  0,81  0,78  0,78  0,78  0,78  0,78  0,77  0,77  0,77  0,75  0,73  0,73  0,72  0,72  0,72  0,72  0,72  0,72  0,70  0,67  0,68  0,68  0,68  0,67 
2024-08-06 23:24:43 INFO   y 11:  0,82  0,82  0,82  0,82  0,82  0,82  0,82  0,79  0,77  0,77  0,77  0,77  0,76  0,76  0,76  0,75  0,73  0,71  0,71  0,71  0,71  0,70  0,70  0,70  0,69  0,67  0,65  0,65  0,66  0,65  0,65 
2024-08-06 23:24:43 INFO   y 12:  0,80  0,80  0,80  0,80  0,80  0,80  0,80  0,78  0,76  0,76  0,76  0,76  0,75  0,75  0,74  0,74  0,71  0,69  0,69  0,70  0,69  0,69  0,68  0,67  0,67  0,64  0,62  0,63  0,64  0,63  0,62 
2024-08-06 23:24:43 INFO   y 13:  0,78  0,78  0,78  0,78  0,78  0,78  0,78  0,76  0,75  0,75  0,76  0,75  0,74  0,73  0,73  0,72  0,70  0,68  0,69  0,70  0,69  0,68  0,67  0,67  0,66  0,64  0,62  0,64  0,66  0,64  0,62 
2024-08-06 23:24:43 INFO   y 14:  0,76  0,76  0,76  0,76  0,76  0,76  0,76  0,75  0,73  0,74  0,75  0,74  0,73  0,72  0,72  0,71  0,69  0,68  0,69  0,70  0,69  0,67  0,67  0,66  0,65  0,64  0,62  0,65  0,67  0,65  0,62 
2024-08-06 23:24:43 INFO   y 15:  0,75  0,75  0,75  0,75  0,75  0,75  0,75  0,73  0,72  0,73  0,74  0,73  0,72  0,71  0,70  0,70  0,68  0,67  0,69  0,71  0,69  0,67  0,66  0,65  0,65  0,63  0,62  0,65  0,69  0,65  0,62 
2024-08-06 23:24:43 INFO   y 16:  0,68  0,68  0,68  0,68  0,68  0,68  0,68  0,67  0,66  0,68  0,69  0,68  0,67  0,67  0,66  0,66  0,65  0,64  0,67  0,69  0,67  0,65  0,65  0,64  0,64  0,63  0,62  0,66  0,70  0,66  0,62 
2024-08-06 23:24:43 INFO   y 17:  0,61  0,61  0,61  0,61  0,61  0,61  0,61  0,60  0,60  0,62  0,64  0,63  0,62  0,62  0,62  0,62  0,61  0,61  0,64  0,68  0,66  0,63  0,63  0,63  0,63  0,63  0,62  0,67  0,71  0,67  0,62 
2024-08-06 23:24:43 INFO   y 18:  0,59  0,59  0,59  0,59  0,59  0,59  0,59  0,59  0,58  0,61  0,63  0,63  0,62  0,62  0,63  0,63  0,63  0,64  0,68  0,72  0,69  0,67  0,67  0,68  0,69  0,69  0,69  0,74  0,79  0,74  0,69 
2024-08-06 23:24:43 INFO   y 19:  0,57  0,57  0,57  0,57  0,57  0,57  0,57  0,57  0,57  0,60  0,63  0,62  0,61  0,62  0,63  0,64  0,65  0,67  0,71  0,75  0,73  0,71  0,72  0,72  0,73  0,74  0,75  0,81  0,86  0,81  0,75 
2024-08-06 23:24:43 INFO   y 20:  0,54  0,54  0,54  0,54  0,54  0,54  0,54  0,54  0,55  0,58  0,62  0,61  0,60  0,60  0,61  0,61  0,62  0,63  0,68  0,72  0,70  0,68  0,68  0,69  0,69  0,70  0,71  0,76  0,81  0,76  0,71 
2024-08-06 23:24:43 INFO   y 21:  0,51  0,51  0,51  0,51  0,51  0,51  0,51  0,52  0,53  0,57  0,61  0,59  0,58  0,58  0,58  0,58  0,59  0,60  0,65  0,69  0,67  0,64  0,64  0,65  0,65  0,65  0,66  0,72  0,77  0,72  0,66 
2024-08-06 23:24:43 INFO   y 22:  0,49  0,49  0,49  0,49  0,49  0,49  0,49  0,50  0,51  0,56  0,60  0,58  0,56  0,57  0,57  0,57  0,58  0,59  0,65  0,69  0,67  0,64  0,64  0,64  0,64  0,65  0,66  0,72  0,78  0,72  0,66 
2024-08-06 23:24:43 INFO   y 23:  0,46  0,46  0,46  0,46  0,46  0,46  0,46  0,48  0,50  0,54  0,59  0,57  0,55  0,55  0,55  0,55  0,57  0,58  0,64  0,70  0,66  0,63  0,63  0,63  0,63  0,65  0,66  0,73  0,79  0,73  0,66 
2024-08-06 23:24:43 INFO   y 24:  0,43  0,43  0,43  0,43  0,43  0,43  0,43  0,45  0,48  0,53  0,58  0,55  0,53  0,53  0,53  0,53  0,56  0,58  0,64  0,70  0,66  0,62  0,62  0,62  0,62  0,64  0,66  0,74  0,80  0,74  0,66 
2024-08-06 23:24:43 INFO   y 25:  0,30  0,30  0,30  0,30  0,30  0,30  0,30  0,34  0,38  0,45  0,52  0,49  0,47  0,47  0,48  0,49  0,51  0,54  0,62  0,69  0,65  0,60  0,61  0,61  0,62  0,64  0,66  0,74  0,82  0,74  0,66 
2024-08-06 23:24:43 INFO   y 26:  nan   nan   nan   nan   nan   nan   nan   0,18  0,26  0,37  0,45  0,42  0,40  0,41  0,42  0,43  0,47  0,50  0,59  0,67  0,63  0,59  0,59  0,60  0,61  0,64  0,66  0,75  0,83  0,75  0,66 
2024-08-06 23:24:43 INFO   y 27:  0,00  0,00  0,00  0,00  0,00  nan   nan   0,18  0,26  0,37  0,46  0,44  0,41  0,44  0,46  0,48  0,52  0,56  0,65  0,73  0,69  0,65  0,66  0,68  0,69  0,72  0,75  0,84  0,92  0,84  0,75 
2024-08-06 23:24:43 INFO   y 28:  0,00  0,00  0,00  nan    0    nan   -In   0,18  0,26  0,37  0,46  0,45  0,43  0,46  0,49  0,52  0,57  0,61  0,70  0,79  0,75  0,70  0,72  0,74  0,76  0,80  0,83  0,92  1,00  0,92  0,83 
2024-08-06 23:24:43 INFO   y 29:  nan   nan   nan   nan    0    nan   nan   0,18  0,26  0,37  0,46  0,44  0,41  0,44  0,46  0,48  0,52  0,56  0,65  0,73  0,69  0,65  0,66  0,68  0,69  0,72  0,75  0,84  0,92  0,84  0,75 
2024-08-06 23:24:43 INFO   y 30:  nan   +In   nan   nan   nan   nan   nan   0,18  0,26  0,37  0,45  0,42  0,40  0,41  0,42  0,43  0,47  0,50  0,59  0,67  0,63  0,59  0,59  0,60  0,61  0,64  0,66  0,75  0,83  0,75  0,66

I tried with blue iso white needle, which fails on both red and white. I will provide a github repo with working code and sample images tomorrow (github is acting up).

For now, here’s the full code (it’s hacky, but it’s just a POC): package org.whatever;import java.io.File;import java.util.ArrayList;impo - Pastebin.com
Here’s the full log: 2024-08-06 23:24:42 INFO 2024-08-06 23:24:42 INFO Match method: 32024-08-06 - Pastebin.com

Any hints?

crackwitz · August 7, 2024, 5:44am

I haven’t seen a way to lift the 1-picture restriction easily. for now, just add posts to the thread with the data you want to share.

kinram · August 7, 2024, 8:13pm

I need to add 14 images to the post, if I would add 14 replies to this thread with 1 image per reply (I tried, it has the same limitation: can’t add more than 1 image per reply), then the story becomes incomprehensible I will provide a github repo with a readme (with markdown and inline images)

berak · August 8, 2024, 7:45am

please, rather restrict it to a few images here, than adding multiple external links (which will expire, and make this post useless)

kinram · August 9, 2024, 8:09pm

I made a single image that contains all of the imagery with some numbers that I use in the story, it’s the best I can do. Ignore the text in my original post, the updated version is in this comment (I don’t have permissions to edit my original post )

A repo with all this info, images and code to reproduce is here: kinram42/opencv-matchtemplate-issue (github.com) (it’s easier to read that readme, it has imagery inline).

The image that contains the reference numbers:

Hi,

I’ve been breaking my head over this for hours, and I either don’t understand how the mask template works, or there is some kind of bug. I tried with the last 5 versions of OpenCV (4.9.0, 4.8.1, 4.7.0, 4.6.0) but they all have the same behaviour.

Let me first explain what I’m trying to do, which is pretty straightforward: I want to find a template using a mask, using TM_CCORR_NORMED (to get values between 0 and 1), which means that I want to ignore some parts of the template (the transparent pixels).

see reference-0 in image

It looks like a window, it has a few white pixels (the borders), and a few transparent pixels.

Trying to find it in a 50x50 image (haystack) and drawing a yellow border around it sometimes works fine.

see reference-1 in image

see reference-2 in image

see reference-3 in image

I tried to investigate the results, and it looks like there are some +Infinity, -Infinity (results of dividing by 0) and NaN (not sure why that occurs) entries in the normalized matrices.

Out of the 4 matches that seem to work fine:

2 are “really” OK: they have a normalized heatmap that makes sense, with values between 0 and 255. These are where I draw the top-left corner of the yellow box (± 1 pixel).
2 have a correct maxLoc, but the heatmaps for the other 2 matches are black (or at least they appear to be black because of wrong normalization)
The last 2 (red) don’t find the correct maxLoc.

see reference-4 in image

Here is some code and logs:

This is how I read images:

private static Mat readImage(String imageName, int codec) {
    Mat img = Imgcodecs.imread(new File(Main.class.getClassLoader().getResource("images/" + imageName + ".png").getFile()).getAbsolutePath(), codec);
    l.info("Read image '{}' using codec {}, result Mat: {}", imageName, codec, img);
    return img;
}

And this is how I write them to disk:

private static void writeImage(String prefix, String imageName, Mat img) {
    String imageFilename = prefix + imageName + ".png";
    Imgcodecs.imwrite(imageFilename, img);
    l.info("Wrote image '{}'", imageFilename);
}

This is the code to create a mask out of the needle image:

public static Mat createMask(String imageName) {
    Mat img = readImage(imageName, Imgcodecs.IMREAD_UNCHANGED);

    Mat mask = new Mat(img.size(), CvType.CV_8UC1);
    for (int y = 0; y < img.rows(); y++) {
        for (int x = 0; x < img.cols(); x++) {
            // OpenCV uses BGR order, so the alpha channel is the 4th channel
            double alphaValue = img.get(y, x)[3];
            if (alphaValue > 0) {
                mask.put(y, x, 255); // Consider this pixel
            } else {
                mask.put(y, x, 0); // Ignore this pixel
            }
        }
    }
    return mask;
}

Docs say that “any non-0 value is not ignored”, I used 0 and 255 but tried 0 and 1 too, result is the same.

First, I read both needle and haystack as IMREAD_COLOR, ignoring any alpha channel (transparency), these will be used for the template matching. I then re-read the needle INCLUDING the alpha channel, only to create a mask out of it. I write needle and heystack back to disk (to ensure I know what I’m working with), and I also write the mask to disk:

// note that we ignore transparency here (4th alpha channel), we read as 3 channels
Mat needleImg = readImage(needle, Imgcodecs.IMREAD_COLOR);
writeImage(prefix, "01_needle", needleImg);

Mat haystackImg = readImage(haystack, Imgcodecs.IMREAD_COLOR);
writeImage(prefix, "02_haystack", haystackImg);

// this uses the 4th alpha channel
Mat mask = createMask(needle);
writeImage(prefix, "03_mask", mask);

This is what is logged:

Log:

2024-08-06 23:24:43 INFO Read image 'needle_white' using codec 1, result Mat: Mat [ 20*20*CV_8UC3, isCont=true, isSubmat=false, nativeObj=0x222f65fcbc0, dataAddr=0x222f65abdc0 ]
2024-08-06 23:24:43 INFO Wrote image 'method_3_find_needle_white_in_haystack_white_needle_on_red_01_needle.png'
2024-08-06 23:24:43 INFO Read image 'haystack_white_needle_on_red' using codec 1, result Mat: Mat [ 50*50*CV_8UC3, isCont=true, isSubmat=false, nativeObj=0x222f65fc220, dataAddr=0x222f5e2ab40 ]
2024-08-06 23:24:43 INFO Wrote image 'method_3_find_needle_white_in_haystack_white_needle_on_red_02_haystack.png'
2024-08-06 23:24:43 INFO Read image 'needle_white' using codec -1, result Mat: Mat [ 20*20*CV_8UC4, isCont=true, isSubmat=false, nativeObj=0x222f65fc300, dataAddr=0x222f6a810c0 ]
2024-08-06 23:24:43 INFO Wrote image 'method_3_find_needle_white_in_haystack_white_needle_on_red_03_mask.png'

see reference-5 in image

I then use matchTemplate, and also try normalizing it (0-100) to understand what happens.

Code:

Mat matchResult = new Mat();
Imgproc.matchTemplate(haystackImg, needleImg, matchResult, matchMethod, mask);
l.info("matchResult:           {}", matchResult);

Mat normalizedMatchResult = new Mat();
Core.normalize(matchResult, normalizedMatchResult, 0, 100, Core.NORM_MINMAX, CvType.CV_32F);
l.info("normalizedMatchResult: {}", matchResult);

Logs:

2024-08-06 23:24:43 INFO matchResult:           Mat [ 31*31*CV_32FC1, isCont=true, isSubmat=false, nativeObj=0x222f65fd790, dataAddr=0x222f6c9cb00 ]
2024-08-06 23:24:43 INFO normalizedMatchResult: Mat [ 31*31*CV_32FC1, isCont=true, isSubmat=false, nativeObj=0x222f65fd790, dataAddr=0x222f6c9cb00 ]

Nothing special so far. The heatmap generation also happens here, the black heatmap suggests that we’re not staying withing the 0-255 boundaries, so the normalization fails. Code:

Mat heatmap = new Mat();
Core.normalize(matchResult, heatmap, 0, 255, Core.NORM_MINMAX, CvType.CV_8U);
writeImage(prefix, "04_heatmap", heatmap);

I then use minMaxLoc as documented, and I tried both on the original matchResult and my custom-normalized normalizedMatchResult.

Code:

Core.MinMaxLocResult mmr = Core.minMaxLoc(matchResult);
l.info("minMaxLoc matchResult:           maxVal {}, maxLoc {}, minVal {}, minLoc {}", mmr.maxVal, mmr.maxLoc, mmr.minVal, mmr.minLoc);
Core.MinMaxLocResult mmr2 = Core.minMaxLoc(normalizedMatchResult);
l.info("minMaxLoc normalizedMatchResult: maxVal {}, maxLoc {}, minVal {}, minLoc {}", mmr2.maxVal, mmr2.maxLoc, mmr2.minVal, mmr2.minLoc);

Logs:

2024-08-06 23:24:43 INFO minMaxLoc matchResult:           maxVal Infinity, maxLoc {2.0, 27.0}, minVal -Infinity, minLoc {5.0, 30.0}
2024-08-06 23:24:43 INFO minMaxLoc normalizedMatchResult: maxVal -Infinity, maxLoc {0.0, 0.0}, minVal Infinity, minLoc {0.0, 0.0}

This isn’t right, maxVal should be between 0.0 and 1.0 when using TM_CCORR_NORMED, but it’s Infinity.

Also, maxLoc {2.0, 27.0} is not correct, it’s close: it should be {3.0, 28.0}, but there is more.

I custom-printed the values in the result (the code is ugly, but it’s readable output). I replace +Infinity and -Infinity with “+In” and “-In”, I printed NaN as “nan”, and I rounded the values I expect to be between 0 and 1.
This is the result:

2024-08-06 23:24:43 INFO   y  0:  0,66  0,66  0,74  0,82  0,74  0,66  0,66  0,66  0,66  0,66  0,66  0,70  0,75  0,69  0,62  0,62  0,62  0,62  0,62  0,62  0,64  0,67  0,62  0,58  0,58  0,58  0,58  0,58  0,58  0,58  0,58 
2024-08-06 23:24:43 INFO   y  1:  0,66  0,66  0,75  0,83  0,75  0,66  0,66  0,66  0,66  0,66  0,66  0,71  0,75  0,69  0,62  0,62  0,62  0,62  0,62  0,62  0,65  0,67  0,63  0,58  0,58  0,58  0,58  0,58  0,58  0,58  0,58 
2024-08-06 23:24:43 INFO   y  2:  0,74  0,75  0,84  0,92  0,84  0,75  0,74  0,74  0,73  0,72  0,72  0,76  0,81  0,74  0,67  0,66  0,65  0,65  0,64  0,63  0,65  0,68  0,63  0,58  0,58  0,58  0,58  0,58  0,58  0,58  0,58 
2024-08-06 23:24:43 INFO   y  3:  0,82  0,83  0,92  1,00  0,92  0,83  0,82  0,80  0,79  0,78  0,77  0,81  0,86  0,79  0,71  0,70  0,69  0,67  0,66  0,64  0,66  0,68  0,63  0,58  0,58  0,58  0,58  0,58  0,58  0,58  0,58 
2024-08-06 23:24:43 INFO   y  4:  0,74  0,75  0,84  0,92  0,84  0,75  0,74  0,74  0,73  0,72  0,72  0,76  0,81  0,74  0,67  0,66  0,65  0,65  0,64  0,63  0,65  0,68  0,63  0,58  0,58  0,58  0,58  0,58  0,58  0,58  0,58 
2024-08-06 23:24:43 INFO   y  5:  0,66  0,66  0,75  0,83  0,75  0,66  0,66  0,66  0,66  0,66  0,66  0,71  0,75  0,69  0,62  0,62  0,62  0,62  0,62  0,62  0,65  0,67  0,63  0,58  0,58  0,58  0,58  0,58  0,58  0,58  0,58 
2024-08-06 23:24:43 INFO   y  6:  0,66  0,66  0,74  0,82  0,74  0,66  0,66  0,66  0,66  0,66  0,66  0,70  0,75  0,69  0,62  0,62  0,62  0,62  0,62  0,62  0,64  0,67  0,62  0,58  0,58  0,58  0,58  0,58  0,58  0,58  0,58 
2024-08-06 23:24:43 INFO   y  7:  0,64  0,64  0,71  0,78  0,71  0,64  0,64  0,64  0,64  0,65  0,65  0,69  0,73  0,68  0,62  0,63  0,63  0,63  0,64  0,64  0,66  0,69  0,65  0,61  0,62  0,62  0,62  0,62  0,62  0,62  0,62 
2024-08-06 23:24:43 INFO   y  8:  0,61  0,61  0,68  0,75  0,68  0,61  0,61  0,62  0,62  0,63  0,64  0,68  0,72  0,67  0,63  0,63  0,64  0,65  0,65  0,66  0,68  0,71  0,68  0,65  0,66  0,66  0,67  0,67  0,67  0,67  0,67 
2024-08-06 23:24:43 INFO   y  9:  0,60  0,60  0,67  0,73  0,67  0,60  0,60  0,61  0,62  0,63  0,64  0,67  0,71  0,67  0,63  0,63  0,64  0,65  0,66  0,67  0,69  0,71  0,68  0,65  0,66  0,67  0,68  0,68  0,68  0,68  0,68 
2024-08-06 23:24:43 INFO   y 10:  0,59  0,59  0,65  0,71  0,65  0,59  0,59  0,61  0,62  0,63  0,64  0,67  0,70  0,66  0,62  0,63  0,65  0,66  0,67  0,67  0,69  0,71  0,69  0,66  0,67  0,68  0,69  0,69  0,69  0,69  0,69 
2024-08-06 23:24:43 INFO   y 11:  0,63  0,64  0,70  0,75  0,70  0,64  0,63  0,65  0,66  0,66  0,66  0,69  0,72  0,69  0,65  0,65  0,66  0,68  0,68  0,68  0,70  0,71  0,69  0,67  0,67  0,69  0,71  0,71  0,71  0,71  0,71 
2024-08-06 23:24:43 INFO   y 12:  0,67  0,68  0,74  0,79  0,74  0,68  0,67  0,68  0,69  0,69  0,69  0,72  0,74  0,71  0,67  0,67  0,68  0,69  0,69  0,69  0,70  0,71  0,69  0,67  0,68  0,70  0,72  0,72  0,72  0,72  0,72 
2024-08-06 23:24:43 INFO   y 13:  0,60  0,60  0,66  0,71  0,66  0,60  0,60  0,62  0,64  0,64  0,64  0,67  0,70  0,67  0,64  0,64  0,66  0,68  0,68  0,68  0,70  0,71  0,70  0,68  0,69  0,71  0,73  0,73  0,73  0,73  0,73 
2024-08-06 23:24:43 INFO   y 14:  0,51  0,51  0,57  0,62  0,57  0,51  0,51  0,55  0,58  0,58  0,59  0,63  0,66  0,63  0,60  0,61  0,64  0,66  0,67  0,68  0,70  0,71  0,70  0,69  0,69  0,72  0,74  0,74  0,74  0,74  0,74 
2024-08-06 23:24:43 INFO   y 15:  0,50  0,50  0,55  0,60  0,55  0,50  0,50  0,54  0,58  0,58  0,59  0,62  0,65  0,62  0,60  0,61  0,64  0,67  0,68  0,68  0,70  0,72  0,70  0,69  0,70  0,73  0,75  0,75  0,75  0,75  0,75 
2024-08-06 23:24:43 INFO   y 16:  0,47  0,47  0,51  0,55  0,51  0,47  0,47  0,51  0,56  0,57  0,58  0,61  0,63  0,62  0,60  0,61  0,65  0,68  0,69  0,70  0,72  0,74  0,73  0,72  0,73  0,76  0,79  0,79  0,79  0,79  0,79 
2024-08-06 23:24:43 INFO   y 17:  0,43  0,43  0,47  0,50  0,47  0,43  0,43  0,49  0,53  0,55  0,57  0,59  0,62  0,61  0,61  0,62  0,66  0,70  0,71  0,72  0,74  0,76  0,76  0,75  0,77  0,80  0,83  0,83  0,83  0,83  0,83 
2024-08-06 23:24:43 INFO   y 18:  0,42  0,42  0,45  0,47  0,45  0,42  0,42  0,48  0,53  0,55  0,57  0,59  0,61  0,61  0,61  0,62  0,66  0,70  0,72  0,73  0,74  0,76  0,76  0,76  0,77  0,81  0,84  0,84  0,84  0,84  0,84 
2024-08-06 23:24:43 INFO   y 19:  0,41  0,41  0,42  0,44  0,42  0,41  0,41  0,47  0,53  0,55  0,57  0,58  0,60  0,60  0,61  0,62  0,67  0,71  0,72  0,73  0,75  0,76  0,76  0,77  0,78  0,81  0,85  0,85  0,85  0,85  0,85 
2024-08-06 23:24:43 INFO   y 20:  0,43  0,44  0,45  0,46  0,45  0,44  0,43  0,50  0,55  0,57  0,58  0,59  0,60  0,61  0,62  0,63  0,68  0,72  0,73  0,74  0,75  0,76  0,77  0,77  0,78  0,82  0,86  0,86  0,86  0,86  0,86 
2024-08-06 23:24:43 INFO   y 21:  0,45  0,46  0,47  0,47  0,47  0,46  0,45  0,52  0,57  0,58  0,59  0,60  0,61  0,62  0,63  0,64  0,69  0,73  0,74  0,75  0,75  0,76  0,77  0,78  0,79  0,83  0,87  0,87  0,87  0,87  0,87 
2024-08-06 23:24:43 INFO   y 22:  0,37  0,38  0,38  0,39  0,38  0,38  0,37  0,46  0,53  0,54  0,55  0,57  0,58  0,59  0,61  0,62  0,67  0,72  0,73  0,74  0,75  0,76  0,77  0,78  0,79  0,84  0,88  0,88  0,88  0,88  0,88 
2024-08-06 23:24:43 INFO   y 23:  0,27  0,27  0,27  0,27  0,27  0,27  0,27  0,39  0,48  0,50  0,52  0,53  0,55  0,57  0,58  0,60  0,66  0,72  0,73  0,74  0,75  0,76  0,78  0,79  0,80  0,85  0,89  0,89  0,89  0,89  0,89 
2024-08-06 23:24:43 INFO   y 24:  0,26  0,26  0,26  0,26  0,26  0,26  0,26  0,38  0,48  0,50  0,51  0,53  0,55  0,57  0,58  0,60  0,66  0,72  0,73  0,75  0,76  0,77  0,78  0,79  0,81  0,85  0,90  0,90  0,90  0,90  0,90 
2024-08-06 23:24:43 INFO   y 25:  0,18  0,18  0,18  0,18  0,18  0,18  0,18  0,34  0,45  0,48  0,50  0,52  0,54  0,57  0,59  0,60  0,67  0,73  0,75  0,76  0,78  0,79  0,81  0,82  0,84  0,89  0,93  0,93  0,93  0,93  0,93 
2024-08-06 23:24:43 INFO   y 26: -0,00 -0,00 -0,00  nan   nan   nan   nan   0,30  0,43  0,46  0,49  0,51  0,54  0,57  0,59  0,61  0,68  0,75  0,76  0,78  0,80  0,82  0,83  0,85  0,87  0,92  0,97  0,97  0,97  0,97  0,97 
2024-08-06 23:24:43 INFO   y 27:  nan   nan   +In   nan   nan   nan   nan   0,30  0,43  0,46  0,49  0,51  0,54  0,57  0,59  0,61  0,68  0,75  0,76  0,78  0,80  0,82  0,83  0,85  0,87  0,92  0,97  0,97  0,97  0,97  0,97 
2024-08-06 23:24:43 INFO   y 28:   0    0,00  0,00  nan   0,00  nan   nan   0,30  0,43  0,46  0,49  0,51  0,54  0,57  0,59  0,61  0,68  0,75  0,76  0,78  0,80  0,82  0,83  0,85  0,87  0,92  0,97  0,97  0,97  0,97  0,97 
2024-08-06 23:24:43 INFO   y 29:  nan   nan   nan   nan   nan   nan   nan   0,30  0,43  0,46  0,49  0,51  0,54  0,57  0,59  0,61  0,68  0,75  0,76  0,78  0,80  0,82  0,83  0,85  0,87  0,92  0,97  0,97  0,97  0,97  0,97 
2024-08-06 23:24:43 INFO   y 30:  0,00  0,00  0,00 -0,00   0    -In   -In   0,30  0,43  0,46  0,49  0,51  0,54  0,57  0,59  0,61  0,68  0,75  0,76  0,78  0,80  0,82  0,83  0,85  0,87  0,92  0,97  0,97  0,97  0,97  0,97

As you can see, the maxLoc of {2.0, 27.0} comes from the +Infinity from y 27 in the 3d column. I think the infinities/nan’s are throwing off the minMaxLoc.

see reference-6 in image

It has similar Infinity/NaN fields, but the best match location is even further off:

2024-08-06 23:24:43 INFO minMaxLoc matchResult:           maxVal Infinity, maxLoc {1.0, 30.0}, minVal -Infinity, minLoc {6.0, 28.0}
2024-08-06 23:24:43 INFO minMaxLoc normalizedMatchResult: maxVal -Infinity, maxLoc {0.0, 0.0}, minVal Infinity, minLoc {0.0, 0.0}

In the bottom right of the matrix, similar Infinity/NaN things are happening:

2024-08-06 23:24:43 INFO   y  0:  0,97  0,97  0,97  0,97  0,97  0,97  0,97  0,93  0,90  0,89  0,88  0,87  0,86  0,85  0,84  0,83  0,79  0,75  0,74  0,73  0,72  0,71  0,69  0,68  0,67  0,62  0,58  0,58  0,58  0,58  0,58 
2024-08-06 23:24:43 INFO   y  1:  0,97  0,97  0,97  0,97  0,97  0,97  0,97  0,93  0,90  0,89  0,88  0,87  0,86  0,85  0,84  0,83  0,79  0,75  0,74  0,73  0,72  0,71  0,69  0,68  0,67  0,62  0,58  0,58  0,58  0,58  0,58 
2024-08-06 23:24:43 INFO   y  2:  0,97  0,97  0,97  0,97  0,97  0,97  0,97  0,93  0,90  0,89  0,88  0,87  0,86  0,85  0,84  0,83  0,79  0,75  0,74  0,73  0,72  0,71  0,69  0,68  0,67  0,62  0,58  0,58  0,58  0,58  0,58 
2024-08-06 23:24:43 INFO   y  3:  0,97  0,97  0,97  0,97  0,97  0,97  0,97  0,93  0,90  0,89  0,88  0,87  0,86  0,85  0,84  0,83  0,79  0,75  0,74  0,73  0,72  0,71  0,69  0,68  0,67  0,62  0,58  0,58  0,58  0,58  0,58 
2024-08-06 23:24:43 INFO   y  4:  0,97  0,97  0,97  0,97  0,97  0,97  0,97  0,93  0,90  0,89  0,88  0,87  0,86  0,85  0,84  0,83  0,79  0,75  0,74  0,73  0,72  0,71  0,69  0,68  0,67  0,62  0,58  0,58  0,58  0,58  0,58 
2024-08-06 23:24:43 INFO   y  5:  0,97  0,97  0,97  0,97  0,97  0,97  0,97  0,93  0,90  0,89  0,88  0,87  0,86  0,85  0,84  0,83  0,79  0,75  0,74  0,73  0,72  0,71  0,69  0,68  0,67  0,62  0,58  0,58  0,58  0,58  0,58 
2024-08-06 23:24:43 INFO   y  6:  0,97  0,97  0,97  0,97  0,97  0,97  0,97  0,93  0,90  0,89  0,88  0,87  0,86  0,85  0,84  0,83  0,79  0,75  0,74  0,73  0,72  0,71  0,69  0,68  0,67  0,62  0,58  0,58  0,58  0,58  0,58 
2024-08-06 23:24:43 INFO   y  7:  0,92  0,92  0,92  0,92  0,92  0,92  0,92  0,89  0,85  0,85  0,84  0,83  0,82  0,81  0,81  0,80  0,76  0,73  0,72  0,71  0,70  0,69  0,68  0,67  0,66  0,62  0,58  0,58  0,58  0,58  0,58 
2024-08-06 23:24:43 INFO   y  8:  0,87  0,87  0,87  0,87  0,87  0,87  0,87  0,84  0,81  0,80  0,79  0,79  0,78  0,78  0,77  0,77  0,73  0,70  0,69  0,69  0,68  0,67  0,67  0,66  0,66  0,62  0,58  0,58  0,58  0,58  0,58 
2024-08-06 23:24:43 INFO   y  9:  0,85  0,85  0,85  0,85  0,85  0,85  0,85  0,82  0,79  0,79  0,79  0,78  0,78  0,78  0,77  0,77  0,74  0,71  0,71  0,71  0,70  0,70  0,69  0,69  0,69  0,66  0,63  0,63  0,63  0,63  0,63 
2024-08-06 23:24:43 INFO   y 10:  0,83  0,83  0,83  0,83  0,83  0,83  0,83  0,81  0,78  0,78  0,78  0,78  0,78  0,77  0,77  0,77  0,75  0,73  0,73  0,72  0,72  0,72  0,72  0,72  0,72  0,70  0,67  0,68  0,68  0,68  0,67 
2024-08-06 23:24:43 INFO   y 11:  0,82  0,82  0,82  0,82  0,82  0,82  0,82  0,79  0,77  0,77  0,77  0,77  0,76  0,76  0,76  0,75  0,73  0,71  0,71  0,71  0,71  0,70  0,70  0,70  0,69  0,67  0,65  0,65  0,66  0,65  0,65 
2024-08-06 23:24:43 INFO   y 12:  0,80  0,80  0,80  0,80  0,80  0,80  0,80  0,78  0,76  0,76  0,76  0,76  0,75  0,75  0,74  0,74  0,71  0,69  0,69  0,70  0,69  0,69  0,68  0,67  0,67  0,64  0,62  0,63  0,64  0,63  0,62 
2024-08-06 23:24:43 INFO   y 13:  0,78  0,78  0,78  0,78  0,78  0,78  0,78  0,76  0,75  0,75  0,76  0,75  0,74  0,73  0,73  0,72  0,70  0,68  0,69  0,70  0,69  0,68  0,67  0,67  0,66  0,64  0,62  0,64  0,66  0,64  0,62 
2024-08-06 23:24:43 INFO   y 14:  0,76  0,76  0,76  0,76  0,76  0,76  0,76  0,75  0,73  0,74  0,75  0,74  0,73  0,72  0,72  0,71  0,69  0,68  0,69  0,70  0,69  0,67  0,67  0,66  0,65  0,64  0,62  0,65  0,67  0,65  0,62 
2024-08-06 23:24:43 INFO   y 15:  0,75  0,75  0,75  0,75  0,75  0,75  0,75  0,73  0,72  0,73  0,74  0,73  0,72  0,71  0,70  0,70  0,68  0,67  0,69  0,71  0,69  0,67  0,66  0,65  0,65  0,63  0,62  0,65  0,69  0,65  0,62 
2024-08-06 23:24:43 INFO   y 16:  0,68  0,68  0,68  0,68  0,68  0,68  0,68  0,67  0,66  0,68  0,69  0,68  0,67  0,67  0,66  0,66  0,65  0,64  0,67  0,69  0,67  0,65  0,65  0,64  0,64  0,63  0,62  0,66  0,70  0,66  0,62 
2024-08-06 23:24:43 INFO   y 17:  0,61  0,61  0,61  0,61  0,61  0,61  0,61  0,60  0,60  0,62  0,64  0,63  0,62  0,62  0,62  0,62  0,61  0,61  0,64  0,68  0,66  0,63  0,63  0,63  0,63  0,63  0,62  0,67  0,71  0,67  0,62 
2024-08-06 23:24:43 INFO   y 18:  0,59  0,59  0,59  0,59  0,59  0,59  0,59  0,59  0,58  0,61  0,63  0,63  0,62  0,62  0,63  0,63  0,63  0,64  0,68  0,72  0,69  0,67  0,67  0,68  0,69  0,69  0,69  0,74  0,79  0,74  0,69 
2024-08-06 23:24:43 INFO   y 19:  0,57  0,57  0,57  0,57  0,57  0,57  0,57  0,57  0,57  0,60  0,63  0,62  0,61  0,62  0,63  0,64  0,65  0,67  0,71  0,75  0,73  0,71  0,72  0,72  0,73  0,74  0,75  0,81  0,86  0,81  0,75 
2024-08-06 23:24:43 INFO   y 20:  0,54  0,54  0,54  0,54  0,54  0,54  0,54  0,54  0,55  0,58  0,62  0,61  0,60  0,60  0,61  0,61  0,62  0,63  0,68  0,72  0,70  0,68  0,68  0,69  0,69  0,70  0,71  0,76  0,81  0,76  0,71 
2024-08-06 23:24:43 INFO   y 21:  0,51  0,51  0,51  0,51  0,51  0,51  0,51  0,52  0,53  0,57  0,61  0,59  0,58  0,58  0,58  0,58  0,59  0,60  0,65  0,69  0,67  0,64  0,64  0,65  0,65  0,65  0,66  0,72  0,77  0,72  0,66 
2024-08-06 23:24:43 INFO   y 22:  0,49  0,49  0,49  0,49  0,49  0,49  0,49  0,50  0,51  0,56  0,60  0,58  0,56  0,57  0,57  0,57  0,58  0,59  0,65  0,69  0,67  0,64  0,64  0,64  0,64  0,65  0,66  0,72  0,78  0,72  0,66 
2024-08-06 23:24:43 INFO   y 23:  0,46  0,46  0,46  0,46  0,46  0,46  0,46  0,48  0,50  0,54  0,59  0,57  0,55  0,55  0,55  0,55  0,57  0,58  0,64  0,70  0,66  0,63  0,63  0,63  0,63  0,65  0,66  0,73  0,79  0,73  0,66 
2024-08-06 23:24:43 INFO   y 24:  0,43  0,43  0,43  0,43  0,43  0,43  0,43  0,45  0,48  0,53  0,58  0,55  0,53  0,53  0,53  0,53  0,56  0,58  0,64  0,70  0,66  0,62  0,62  0,62  0,62  0,64  0,66  0,74  0,80  0,74  0,66 
2024-08-06 23:24:43 INFO   y 25:  0,30  0,30  0,30  0,30  0,30  0,30  0,30  0,34  0,38  0,45  0,52  0,49  0,47  0,47  0,48  0,49  0,51  0,54  0,62  0,69  0,65  0,60  0,61  0,61  0,62  0,64  0,66  0,74  0,82  0,74  0,66 
2024-08-06 23:24:43 INFO   y 26:  nan   nan   nan   nan   nan   nan   nan   0,18  0,26  0,37  0,45  0,42  0,40  0,41  0,42  0,43  0,47  0,50  0,59  0,67  0,63  0,59  0,59  0,60  0,61  0,64  0,66  0,75  0,83  0,75  0,66 
2024-08-06 23:24:43 INFO   y 27:  0,00  0,00  0,00  0,00  0,00  nan   nan   0,18  0,26  0,37  0,46  0,44  0,41  0,44  0,46  0,48  0,52  0,56  0,65  0,73  0,69  0,65  0,66  0,68  0,69  0,72  0,75  0,84  0,92  0,84  0,75 
2024-08-06 23:24:43 INFO   y 28:  0,00  0,00  0,00  nan    0    nan   -In   0,18  0,26  0,37  0,46  0,45  0,43  0,46  0,49  0,52  0,57  0,61  0,70  0,79  0,75  0,70  0,72  0,74  0,76  0,80  0,83  0,92  1,00  0,92  0,83 
2024-08-06 23:24:43 INFO   y 29:  nan   nan   nan   nan    0    nan   nan   0,18  0,26  0,37  0,46  0,44  0,41  0,44  0,46  0,48  0,52  0,56  0,65  0,73  0,69  0,65  0,66  0,68  0,69  0,72  0,75  0,84  0,92  0,84  0,75 
2024-08-06 23:24:43 INFO   y 30:  nan   +In   nan   nan   nan   nan   nan   0,18  0,26  0,37  0,45  0,42  0,40  0,41  0,42  0,43  0,47  0,50  0,59  0,67  0,63  0,59  0,59  0,60  0,61  0,64  0,66  0,75  0,83  0,75  0,66

Again, maxLoc {1.0, 30.0} (which is wrong) is caused by the +Infinity on line 30 in column 2.

I tried with blue iso white needle, which fails on both red and white.

kinram · August 9, 2024, 8:15pm

I also just noticed that if I use the matchTemplate() without the mask parameter, the Infitniy/NaN values are not there. We don’t find 100% matches (because the needle has “white” windows and the haystacks don’t) but we get “normal” results, and all the maxLoc values are correct. So there must be something that I either do wrong with the mask (or a bug, but I consider it unlikely that I would find that )

When not using the mask, these are the results:

needle needle_blue haystack haystack_blue_needle_on_all match percentage: 67.40497350692749%, top left location: {16.0, 16.0}
needle needle_blue haystack haystack_blue_needle_on_black match percentage: 99.99999403953552%, top left location: {3.0, 28.0}
needle needle_blue haystack haystack_blue_needle_on_green match percentage: 71.4142918586731%, top left location: {28.0, 3.0}
needle needle_blue haystack haystack_blue_needle_on_red match percentage: 71.4142918586731%, top left location: {3.0, 3.0}
needle needle_blue haystack haystack_blue_needle_on_white match percentage: 50.75191855430603%, top left location: {28.0, 28.0}
needle needle_white haystack haystack_white_needle_on_all match percentage: 87.15630173683167%, top left location: {16.0, 16.0}
needle needle_white haystack haystack_white_needle_on_black match percentage: 100.0%, top left location: {3.0, 28.0}
needle needle_white haystack haystack_white_needle_on_green match percentage: 87.03022003173828%, top left location: {28.0, 3.0}
needle needle_white haystack haystack_white_needle_on_purple match percentage: 78.4403145313263%, top left location: {28.0, 28.0}
needle needle_white haystack haystack_white_needle_on_red match percentage: 87.03022003173828%, top left location: {3.0, 3.0}
needle needle_white haystack haystack_white_needle_on_red_bottom_right match percentage: 87.03022003173828%, top left location: {28.0, 28.0}

The bottom-left region of the matrices are also very different from the rest, lots of 0 values, but no Infinity/NaN.

crackwitz · August 9, 2024, 8:50pm

there’s a lot written and it seems to me as if you’re overthinking this.

looks like you want one of the SQDIFF modes.

kinram · August 9, 2024, 9:23pm

I have a use case in which I need transparency to ensure with 99-ish percent certainty that the (non-transparent part of) a need is in a haystack, and it seems like things go wrong when I use it (because I use it wrong, or because of a bug). Even if I would use SQDIFF iso TM_CCORR, I think the transparency will behave similarly.

I’m pretty sure that Infinity/NaN is not a value that should be in the histogram matrix, it’s also clear that it’s throwing off the minMaxLoc (and the normalization). Pointing me to another mode is not very relevant.

I don’t understand your comment about overthinking this, the code is very simple, it’s just a lot of text to explain my tought process while troubleshooting, in the hopes that someone with more expertise can read it and spot my thought-error.

crackwitz · August 10, 2024, 10:32am

I think I spotted it… you needed help. I gave you solid advice. you dismiss my advice and you insult me. newbies often do that. it’s your loss.

I am testing you. will you again refuse to listen? will you require proof in pictures before you will swallow your ego? or will you refuse even then?

kinram · August 10, 2024, 11:10am

Wow, I’m not sure where that is coming from. I didn’t mean to insult you, I don’t know why you’re saying I’m dismissing your advice, or that I’m a newbie, or that I have an ego that I can’t swallow. I’m sorry my reply makes you feel that way, I really do, I appreciate the effort open source contributors spend on projects like this, and I appreciate you even just taking the time to reply to my question.

I’m trying to understand WHY what I do doesn’t work using the TM_CCORR, because the docs don’t mention anything about transparency needing SQDIFF, and my approach is very similar to the code samples I find in the codebase. There’s a clear “weird” behaviour with Infinity/NaN values and normalization that trips over it, I’m trying to understand why that is, to learn, and maybe to help someone else in the future to bump into the same problem by adapting some docs or finding this post. Your advice to use another method doesn’t help me understand what I do wrong or the Infinity/NaN values, I didn’t mean to dismiss it, but it felt like “you’re trying A, which doesn’t work, you should try B” while I’m trying to figure out why A doesn’t work

Another reason I maybe wasn’t receptive enough of you proposal to move from TM_CCORR to SQDIFF is because I’m working on a project where thousands of images/matches should keep working as today, not changing the “match percentage”, and I’m afraid that changing the match method will introduce a difference in behaviour in other use cases. We use TM_CCORR when not using transparency, and I now only want it to work using transparency, not sure if it’s smart to keep using TM_CCORR when there is no transparent match needed, and SQDIFF for transparency.

I will adapt my code to use SQDIFF as you suggest and report back soon, but if this magically works, I will still be stuck wondering why my original code does what it does and figure out how to introduce this without breaking existing functionality.

I hope I’ll pass your test

crackwitz · August 10, 2024, 1:04pm

I think it is very relevant. I think that because I have the experience, competence, and intelligence to make that determination. that is why I bothered to recommend it.

in my experience, SQDIFF is the only appropriate mode when the situation is characterized by exact color values and areas of zero variance. the other modes (CCORR, CCOEFF) are hard to predict because of their more complex calculations.

aside from that, all the “normed” modes involve a division (see docs). if that term happens to be zero for some reason, you get inf/nan. that happens when the needle or haystack are exactly black in a particular area. “normed” modes sound convenient but they really aren’t.

I have had to deal with modes other than SQDIFF often enough. they are always a headache. they are invalid options in many cases. invalid in the way that drinking beer with a fork is invalid. I say this because too often people have been of the opinion that if they just try hard enough, they’ll manage to drink that beer with a fork. any nicer way to put this has in the past frequently not gotten the point across.

kinram · August 11, 2024, 12:30pm

Thanks for this explanation, it clears a few things up, but it triggers other questions. I think the code in the codebase (that I inherited) doesn’t do what our users/developer think it does, and I’m afraid it goes even beyond the SQDIFF/CCORR/CCOEFF difference, there’s also a wrong assumption on the _NORM variant…

The idea is to check if a needle is found in a haystack with “high certainty”, so we can conclude yes/no. So although you always find a location of the best match, if the maxValis not high enough (ie. 0,97-ish), we consider the needle not to be present because it’s not a good enough match.

The TM_CCORR_NORMED method is used, of which (afaik) the resulting values in the matrix are between 0 and 1 (because of normalization), although because of your explanation (and my tests) it seems like that’s not true, and there are also Infinity/NaN values. This maxVal value is then interpreted as the “certainty” (as a percentage), but assuming python - Understanding and evaluating template matching methods - Stack Overflow is a good explanation, I now understand that that is not what it means at all. You can use this metric to compare matches with eachother, but you cannot interpret is a the certainty of a single match.

It’s a miracle our code is able to detect anything based on this fundamental flaw.

I’m considering moving towards SQDIFF, but I read (assuming explanation on that article is correct, I can’t interpret the OpenCV docs since they’re limited to matrix functions and I’m not sure how to understand them):

As for picking the right threshold, the value from ZNCC or SSD is not a confidence or probability number at all. If you want to pick the right threshold, you can measure the parameter in any number of typical ways. You can calculate ROC curves or PR curves for different thresholds. You can use regression to find the optimal parameter. You’ll need to label some data, but then at least you’ll have measurements of how you’re doing against some test set so that your choice is not arbitrary. As usual with a data-filled field, you’ll need to make sure your data is as close to real world examples as possible, and that your test data covers your edge cases as well as your typical images.

So I’m not sure how to model the “certainty”, and hence to conclude if the needle is in the haystack or not.

Also, my first and biggest mistake seems to be that I’m trying to prove how well my implementation works with hand-crafted images (with very simple pixel values and very little variation), which is probably irrelevant for the types of images that the algorithm will work on. The real data is very different.

I think I’m going to try to ONLY use a separate transparency-implementation (to not break existing functionality), use SQDIFF (not SQDIFF_NORMED), throw some real world example against it and see what an acceptable minVal threshold is.

crackwitz · August 11, 2024, 1:54pm

the score for SQDIFF can be interpreted, and you can calculate your own relative score. you can figure out a theoretical maximum for such a score, for the worst possible match. that would be the worst case difference in each pixel, squared and summed. that depends on the template’s values (black/white template: difference of 255, gray template: difference of half that, and so on).

due to the squaring involved, you could take the square root of the score to obtain something somewhat linear. that would be a Root Mean Square, which is specifically sensitive to outliers, i.e. large differences. another measure, not available in matchTemplate, would be the maximum of all pixel differences, rather than the sum of differences. that’d be most sensitive to outliers. SQDIFF is very sensitive to differences in average, i.e. illumination. if you don’t want that, you’ll either have to normalize illumination/brightness yourself or consider one of the other matching modes.

there is no easy or obvious way to condense the matching/difference between two image patches into a simple percentage. correlation (CCORR, CCOEFF) is theoretically valid but you have to be aware of the situations where it is meaningless. “confidence” is generally a nebulous term. image patch differences could be quantified as a histogram or cumulative distribution. then you could say, “what fraction of pixels differs by at least some threshold”.

yes, real world data would definitely help in understanding what you need.

Topic		Replies	Views
TemplateMatch -- Applying masks to templates Python	3	312	March 13, 2024
Inf values in template matching Python matchtemplate	1	679	November 12, 2021
Missing the cost function of TM_CCOEFF_NORMED with mask for matchTemplate() in API doc C++ matchtemplate	4	716	December 3, 2021
Working with mathTemplate C++ matchtemplate	1	381	April 6, 2022
Matching and Code improvement questions Python matchtemplate	8	1548	August 17, 2022

Java matchTemplate with mask and TM_CCORR_NORMED fails to find match (Infinity/NaN)

Related Topics