[OpenCV.js] Getting SIFT and FLANN to work

, , ,
1

Hi, all! I’m looking for help on how to use SIFT and FLANN with OpenCV.js.

I basically want to translate this code from Python:

needle = cv.imread('needle.png', 0)
haystack = cv.imread('haystack.png', 0)

sift = cv.SIFT_create()

FLANN_INDEX_KDTREE = 1
index_params = dict(algorithm = FLANN_INDEX_KDTREE, trees = 5)
search_params = dict(checks = 50)
flann = cv.FlannBasedMatcher(index_params, search_params)

kp_needle, desc_needle = sift.detectAndCompute(needle, None)
kp_haystack, desc_haystack = sift.detectAndCompute(haystack, None)
matches = flann.knnMatch(desc_needle, desc_haystack, k=2)

This is what I understand so far would be the “translation” (and please correct me if I’m wrong!):

const needle = await loadImage('needle.png'),
    haystack = await loadImage('haystack.png'),
    needle_src = cv.imread(needle, 0),
    haystack_src = cv.imread(haystack, 0),

    sift = new cv.SIFT(),

    FLANN_INDEX_KDTREE = 1
    // ??? for flann variable of FlannBasedMatcher? (A)

let kp_needle = new cv.KeyPointVector(),
    kp_haystack = new cv.KeyPointVector(),
    desc_needle =  new cv.Mat(),
    desc_haystack =  new cv.Mat()

sift.detectAndCompute(needle_src, new cv.Mat(), kp_needle, desc_needle)
sift.detectAndCompute(haystack_src, new cv.Mat(), kp_haystack, desc_haystack)
// ??? for flann's knnMatch (B)

.

I tried setting up this for issue area A, mimicking the Python version as closely as I could, but the error I received indicates FlannBasedMatcher() takes 0 parameters:

const index_params = {algorithm: FLANN_INDEX_KDTREE, trees: 5},
    search_params = {checks: 50},
    flann = new cv.FlannBasedMatcher(index_params, search_params)

RuntimeError: abort(BindingError: Tried to invoke ctor of FlannBasedMatcher with invalid number of parameters (2) - expected (0) parameters instead!). Build with -s ASSERTIONS=1 for more info.

.

I’m not sure what I need to do to get that param data passed over if it wants no parameters. Any ideas?

I haven’t tried figuring out issue area B yet, since I imagine I need A sorted out first.

It was thanks to a code someone wrote using ORB that I was able to figure it out for SIFT. I can’t seem to find anything like that for FLANN.

I had to build OpenCV.js myself to add both SIFT and FLANN. It’s been rough, but I’ve been able to figure it out up until now. If I can get all these pieces figured out, I’ll share what I had to do so others can more easily use this, too.

I appreciate any help!

2

this is weird, there is no FlannBasedMatcher in the bindings at all, you should get something like a

Uncaught TypeError: cv.FlannBasedMatcher is not a constructor

seems, all you can use from js is

https://docs.opencv.org/4.x/db/d39/classcv_1_1DescriptorMatcher.html#ab5dc5036569ecc8d47565007fa518257

as in:

flann = new cv.DescriptorMatcher("FlannBased");
3

Thanks for the input! FlannBasedMatcher is not included with OpenCV.js by default. But I followed what was suggested here on GitHub and built OpenCV.js myself to add it.

That said, while I was waiting for my post to be accepted, I kept tinkering around and was able to get this working:

const needle = await loadImage('needle.png'),
    haystack = await loadImage('haystack.png'),
    needle_src = cv.imread(needle, 0),
    haystack_src = cv.imread(haystack, 0),
    sift = new cv.SIFT(),
    flann = new cv.FlannBasedMatcher()

let kp_needle = new cv.KeyPointVector(),
    kp_haystack = new cv.KeyPointVector(),
    desc_needle = new cv.Mat(),
    desc_haystack = new cv.Mat(),
    matches = new cv.DMatchVectorVector()

sift.detectAndCompute(needle_src, new cv.Mat(), kp_needle, desc_needle)
sift.detectAndCompute(haystack_src, new cv.Mat(), kp_haystack, desc_haystack)
flann.knnMatch(desc_needle, desc_haystack, matches, 2)

I found the Python version also works if you don’t pass anything to FlannBasedMatcher(), but I’m not sure if I’m losing anything in the process. The results for Python seem to be the same either way, though different from what I’m getting with OpenCV.js (though I need to test if the images are being converted to grayscale or not there yet, or maybe loadImage() is adding some other kind of factors).

Do you know if those index_params and search_params are the universal default and I don’t have to worry about them? Or do you know if there’s any methods that exist to add the index_params and search_params outside of creating a new FlannBasedMatcher object?

I’ve tried reading the documentation for FlannBasedMatcher and its listed inheritance tree, but I’m having difficulty making sense of it. It looks like the only methods that might be involved are read() and write(), but it’s rather vague.

1 Like
4

yes, they are, dont worry:

(just note, that you will need a BFMatcher(HAMMING) if you decide to switch back to ORB or AKAZE)

i’d think so

1 Like
5

Awesome, thank you so much for the help!

Edit: Yup, I had to use cv.cvtColor() with cv.COLOR_RGBA2GRAY and got the same results as the Python code. It looks like cv.imread() won’t do the color conversion for you like it does in Python.

6

right. it does not even accept any flag
(and js is quite “liberal” with additional unused args)

1 Like
7

Can you provide me an opencv.js code that contains SIFT algorithm and FlannBasedMatcher? I can’t compile opencv.js code that can be used through Emscripten, thank you very much