Why are images not warped before feature detection and matching in the stitching pipeline?

Python
,
1

Hi, so I am new to image stitching. Ultimately, I would like to implement a pipeline for CUDA (which I am also unfamiliar with), but for now, I am trying to implement a pipeline in Python to understand the processes conceptionally.

I already have a working pipeline, but I naively implemented the cypindrical projection of the images before the feature matching and detection. This made intuitive sense to me, because the appearance of features is affected by the projection, but in the warped images I would expect the same objects/features to look more similar between images than they would in the original ones. So I thought that doing the feature detection and matching on the warped images should yield better results. But it seems that is not so? If I understand correctly, In openCV’s detailed stitching example, the warping is done essentially as the final step together with the homography application.

Can somebody explain the logic behind this?

2

for the projection (cylindrical) you need a pose for every picture. you can only estimate a pose once you have the features.

lens distortion has so little effect on the features you’d extract that it’s wasted effort to calculate an undistorted image. undistorted images are useless. the lens distortion is folded into the calculation of the final composite map.

3

Hi, thank you for your answer! But what do you mean by pose? For the cylindrical projection I only need to know the focal length and image dimensions, no?

So far, my stitching pipeline looks something like this:

# cylindrical transformation
# ...

# feature detection and matching
# ...

homography = cv2.findHomography(src_pts, dst_pts, cv2.RANSAC, 5.0)
warped_image = cv2.warpPerspective(image, homography, output_size)

# ...
# image blending

If I don’t do the cylindrical transformation manually, the homography does not take care of it automatically, which makes sense to me, since I’m not supplying the focal length to the functions. What am I missing?

Also: What do you mean by “the lens distortion is folded into the calculation of the final composite map”? It’s interesting that you mention lens distortion, because along with shadowing effects on the lens and color correction it is also something I was considering on doing before feature detection.

4

of what? what do your images look like? or is it one image? I need to see. I can’t guess what you have there.

you either use the stitching module or you don’t. any involvement of explicit feature matching, homography calculation, or perspective warping is incompatible with use of the stitching module. the module does all that it needs, on its own. if you claim to involve any cylindrical map, then that is impossible to do without the stitching module. homography and perspective warp cannot do cylindrical projections.

this is the wrong place to give you an entire semester’s worth of teaching. nor are the OpenCV docs any teaching material. they are documentation for the library, with tutorials on how to use the library, sometimes teaching the very basics of the theory. the more advanced theory is too vast to shoehorn into library documentation; it is assumed to be known already. the docs do not convey the theory required to understand what’s going on and why things are done the way they are.

you’d better go find books and/or lecture recordings on the topic. this does not mean “blogs” or youtube videos made for any “online academy” or other junk sites, made for the advertisement revenue. those are too shallow and won’t teach you anything substantial. I mean proper text books, proper lectures. you can confidently ignore any books by “tech publishers”. that’s barely better than printed out blogs.

5

Hi, thank you for your feedback and sorry for my late reply!

of what? what do your images look like? or is it one image? I need to see. I can’t guess what you have there.

I was thinking in general. As far as I understood, the focal length (and image dimensions) are all you need to perform a cylindrical transformation. So I was confused about what you meant by pose.

you either use the stitching module or you don’t…nor are the OpenCV docs any teaching material.

I am trying to build my own pipeline so that I can adapt it to my specific needs. But to get a general idea about what such a pipeline might look like, and because I might use some of the lower-lewel functions of OpenCV, I was looking at the detailed stitching example. I get your sentiment however and I would love to patiently build my understanding from the ground up. So I appreciate your general recommendation. But I’m afraid I don’t have the time to do that. I was therefore hoping there might be an explanation for why the cylindrical projection is applied together with the homography that I can understand without first having read semester’s worth of books. If this is not the place to get that answer that’s a shame but understandable. I didn’t mean to ask the unreasonable.