matchTemplate and old games graphics

[veilleur]

hi all,

#elementachercher = path to template png file
#screenascanner = path to global picture where to find template


import pyautogui
import time
import cv2
import numpy as np
from matplotlib import pyplot as plt
import os
import imutils

def tryfoundobject(elementachercher, screenascanner):
    img_rgb = cv2.imread(screenascanner)
    img_gray = cv2.cvtColor(img_rgb, cv2.COLOR_BGR2GRAY)
    template = cv2.imread(elementachercher,0)
    w, h = template.shape[::-1]
    res = cv2.matchTemplate(img_gray,template,cv2.TM_CCOEFF_NORMED)
    threshold = 0.8
    loc = np.where( res >= threshold)
    for pt in zip(*loc[::-1]):
        cv2.rectangle(img_rgb, pt, (pt[0] + w, pt[1] + h), (0,0,255), 2)
        if pt != None:
            print(elementachercher+" matched")
            break
        else:
            continue
    cv2.imwrite("NEW-"+elementachercher,img_rgb)
    
  tryfoundobject("/home/noway/template.png", "/home/noway/mypicture.png")  

But no matches are found.

What am I doing wrong? 1 screen is what result i want, the second screen is what i got after script

what im trying yo do

[veilleur]

what i got:

[veilleur]

normalise picture with

res_normed = (res - np.min(res)) / (np.max(res) - np.min(res))
res_uint = np.array(res_normed * 255, dtype=np.uint8)
cv2.imshow("res_uint ", res_uint)
cv2.waitKey(0)
cv2.destroyAllWindows()

[crackwitz]

@veilleur

    cv2.imwrite(elementachercher,img_rgb)
    
  tryfoundobject("/home/noway/template.png", "/home/noway/mypicture.png")  

you overwrite template.png . check that it contains what you expect.

[veilleur]

i fail my copy/past, i edited my first post

[crackwitz]

please provide template.png

it would also be a good idea to strip your source code down to a “minimal reproducible example”

[veilleur]

this is my template :

[veilleur]

sry i can post only one picture per post

[veilleur]

and my picture for seach :

[veilleur]

i suspect my template is to big (144px144px) not match, because when i try manual rezize (53px48px) it match

[crackwitz]

your template is not the same size as its instances in the search picture. it must have the same size.

I’ve resized it to be “pixel-perfect”:

I’ve then resized the template to 54x54 pixels, which approximately matches the size of the instances in the search picture.

your search picture appears to be resampled. results will not be perfect.

expect “detections” for adjacent positions. you need to perform “non-maximum suppression”. for just a threshold, this is the type of result you get:

pt [435 477]
pt [435 478] # neighbor of previous one
pt [508 637]
pt [508 638] # neighbor of previous one
pt [522 909]
pt [523 909] # neighbor of previous one
pt [631 867]

some red rectangles are drawn over each other. you “see” 4 but all 7 were drawn, and some of them almost coincide with each other.

I’ve changed the code, included non-maximum suppression, used a different matchTemplate mode (TM_SQDIFF), and calculated the theoretical maximum difference given the template.

one could use TM_SQDIFF_NORMED but I don’t like how it’s normed.

#!/usr/bin/env python3

import cv2 as cv
import numpy as np

def non_maximum_suppression(values, radius=1):
	local_area = cv.dilate(values, kernel=None, iterations=radius)
	mask = (values == local_area)
	return mask


template = "40eb56591bd9383a6d5c1f75a90c8bfeb28f877c.png"
picture = "61a178bd038544749c64310a9812b5e35217fdd1.jpeg"

picture = cv.imread(picture)
template = cv.imread(template, cv.IMREAD_UNCHANGED) # read alpha channel too!
assert template.shape[:2] == (24, 24), "template ought to be pixel perfect"
assert template.shape[2] == 4, "template contains no alpha channel!"

# 54x54 empirically
template = cv.resize(template, dsize=(54, 54), interpolation=cv.INTER_CUBIC)
(th, tw) = template.shape[:2]

# separate alpha channel, construct binary mask for matchTemplate
template_color = template[:,:,:3]
template_alpha = template[:,:,3]
template_mask = (template_alpha >= 128).astype(np.uint8)

# theoretical maximum difference for this template
# (largest possible difference in every pixel and every color channel)
#maxdiff = template_mask.sum() * 255**2
maxdiff = np.maximum(template_color, 255-template_color) # worst case difference
maxdiff *= np.uint8(template_mask)[:,:,np.newaxis] # apply mask
maxdiff = (maxdiff.astype(np.uint32)**2).sum() # sum of squared differences
print("maximal difference:", maxdiff)

#cv.imshow("picture", picture)
cv.imshow("template",
	np.hstack([template_color, cv.cvtColor(template_alpha, cv.COLOR_GRAY2BGR)])
)

sqdiff_values = cv.matchTemplate(image=picture, templ=template_color, method=cv.TM_SQDIFF, mask=template_mask)

# only supposed to kill slopes of peaks, not anything that's below threshold
# radius 1 already effective if data is smooth; removes anything on a slope
# have to pass `-sqdiff_values` because sqdiff_values contains minima, this looks for maxima
peakmask = non_maximum_suppression(-sqdiff_values, radius=1)

# kill everything below threshold too
threshold = 0.10 # applies to sum of SQUARED differences
peakmask[sqdiff_values > maxdiff * threshold] = False

# (i,j)
loc = np.array(np.where(peakmask)).T

for pt in loc:
	(i,j) = pt
	print("pt", pt, "diff", (sqdiff_values[i,j] / maxdiff))
	cv.rectangle(picture, (j,i), (j + tw, i + th), (0,0,255), 2)

cv.imwrite("61a178bd038544749c64310a9812b5e35217fdd1-out.png", picture)
cv.imshow("picture", picture)

cv.waitKey(-1)
cv.destroyAllWindows()

pt [435 478] diff 0.016204495277050632
pt [508 637] diff 0.02022078829591532
pt [523 909] diff 0.025676983107246407
pt [631 867] diff 0.014345963889969395

[veilleur]

thank you for your reply, great job ! im gonna try this =)