When comparing pictures, SIFT and ORB methods don't work when one picture has been mirrored

Python
1

Hi all,

I’m trying to find all pictures that are “almost” identical in a folder. By “almost identical”, I mean for example you have an original picture, and you have modifications of this picture. The modifications can be change of resolution, picture changed to greyscale, picture was cropped, picture rotated, a frame or some text was added, picture was mirrored…

I’m using OpenCV with SIFT and ORB (I choose each what method I want to use, I don’t use them at the same time). For all the picture variations, both SIFT and ORB work quite well. But not for the mirror picture. Even if I only make a mirror image of the first picture (meaning I don’t change anything else), the score is about 10%.

I don’t understand, as I thought that SIFT and ORB were calculating distances of keypoints. But when taking a mirror image, the distances don’t change. Only the direction.

What am I missing ?

Here is the extract from my code :



    if method   == 'ORB':
        finder = cv2.ORB_create()
    elif method == 'SIFT':
        finder = cv2.xfeatures2d.SIFT_create()

    lowe_ratio = 0.86

    # find the keypoints and descriptors with SIFT or ORB
    for i in range(0,count-1):
        for j in range(i+1,count):
            kp1, des1 = finder.detectAndCompute(all_new_images_to_compare[i],None)
            kp2, des2 = finder.detectAndCompute(all_new_images_to_compare[j],None)
            bf = cv2.BFMatcher()
            matches = bf.knnMatch(des1,des2, k=2)        
            good_points = []
            for m,n in matches:
                if m.distance < lowe_ratio * n.distance:
           good_points.append(m)
           number_keypoints = 0
           if len(kp1) >= len(kp2):
               number_keypoints = len(kp1)
           else:
               number_keypoints = len(kp2)
           percentage_similarity = len(good_points) / (number_keypoints) * 100
           
            if  (percentage_similarity)>=10:
                myfile1=open("C:/Users/ABC/Documents/Find-Similar-Pictures/results_" + method + ".txt","a")
                myfile1.write(str(titles[i]) + "\t" + str(titles[j]) + "\t" + method + " (" + str(lowe_ratio) + ") \t" + str(int(percentage_similarity) ) + "\t\n")
                myfile1.close()
                print(titles[i], titles[j],"== Similarity: " + str(int(percentage_similarity)), method + " (" + str(lowe_ratio) + ")")
        print("___done with file", titles[i])
    print("=====done=====")


Thanks a lot for your help
2

please make sure the indentation of your python code is correct. indentation has meaning in python, it’s part of the syntax.

further… feature descriptors such as SIFT weren’t designed to be invariant/robust to mirroring, only rotation, scale, and slight affine transformations. I would only expect features to match that have symmetry (i.e. look the same when mirrored). that should be the case for very few features. most corner features don’t satisfy that condition. line ends and simple points would satisfy it.

you can see why that is when you look into the construction of SIFT/ORB. SIFT contains a histogram of local gradients, aligned by a coarse orientation.

3

Hi Crackwitz,
Thanks for your answer.
So I guess I’ve 2 options:

  • change the code of the SIFT and ORB feature descriptors
  • when I compare 2 pictures, if it gives nothing, then also try comparing one with the mirror copy of the second one.

Maybe I’ll start with option 2 :grin:

4

related: When comparing pictures, SIFT and ORB methods in OPenCV don't work when one picture has been mirrored - Stack Overflow