Hi there, I’d like to solve a problem I have with my 2D image panorama stitcher. I have a working code that creates a panorama from multiple images based on salient point matching. All looks well, except, I’d like to change how the images overlap. Currently, the left image is on top of the right image in any given pair of images of the panorama. I’d like to reverse that and have the right image on top of the left image. Or in other words, I’d like to use all pixels of the right-hand image instead of using all pixels of the left-hand image.
I’ve attempted doing so by just assembling the images in a different z-order. However, I get undesirable results this way. Example:
I’m guessing that this is because my images are not corrected for lens distortion. If they were, I’d probably be able to do it. Instead, maybe I have to tell the homography finding function to use a different seam?
that’s the wrong function. it’s only good for orthogonal projections… and I’m sure you don’t have such a lens because it’d be as large as that plane.
you stand still while panning the camera. you are taking pictures that map a spherical view from your position.
you could estimate a perspective transformation (homography) and apply perspective warps. your result would be a wide angle composite shot, with the usual distortions.
in fact, you should be using the stitching module because it handles projections properly.
if you took a 360 degree panorama, you’d REQUIRE a cylindrical or other projection (equirectangular? common in 360° cameras). a simple flat projection would be impossible in that case.
I wish I could use the stitching module in my application. However, this is not an option as the application has an existing render pipeline using OpenGL that I have to use. OpenCV would also be way to slow in this case, as there are hundreds of cameras needed to be stitched and warped in real time at HD or 4K resolution each.
The render pipeline allows for 9 dof, however the 4dof from estimateAffinePartial2D already gives very good results with only minimal error. I could be using the 9dof homography, however I don’t see a way to preselect a seam there either. This is important for me because the render pipeline requires the right image to be on top of the left image. Are you suggesting that the function cv::findHomography can handle custom seams?
Cyclindrical or other projections than the orthogonal projections are really not of much importance here, the scene is always far away from the cameras, which are a cluster of cameras very close together, each with a slightly different lens (same model, but intrinsic parameters vary). With 4dof + lens undistortion I’m getting the desired results. Only in some instances where I’m missing the intrinsic camera calibration to do the lens undistortion do I get undesirable results if I’m not using OpenCV’s mandated seam.
So my initial question remains: is there a a way to select the seam that OpenCV uses to create the homography/affineTransform?
no. your cameras aren’t moving, are they? stitching in general implies that camera positions are unknown and to be estimated.
once you know your camera positions, compositing the picture is trivial.
your first post should have included these details. you know it’s going to be helpful to understand your problem. don’t wait for people to prompt you.
you talk about “seams”. please explain what you mean by that, and why you talk about “selecting seams”. explain that action. the stitching module blends individual pictures to reduce artefacts…
Correct, they aren’t moving. A few pixels per year at most. The stitching is typically done once and only adjusted every now and then. Ideally this should be automated - which is what I’m working on currently. The position is unknown to me at first. I can run the OpenCV methods to find the affine image transformations and never have to run them again for the remainder of the program runtime.
The camera images have overlap (between 5% and 20% of the width of an image). Therefore, when stitching the images one could choose any line in the overlap area. OpenCV’s pipeline (specifically the parts to get the homography & then warping/affine transformation) seems to always want to use as much of the left-hand image as possible → right most possible seam. I’d like to use as much of the right-hand image though, to better match by render pipeline → left most seam.
Due to imperfections with these images and the process used by me (no lens correction and just an affine transform, not a true perspective correction, also some parallax) I have to use different image transformations depending on the picked seam. OpenCV only gives me the correct transformation values for the image for the rightmost seam.
I’m attaching an example to illustrate what I mean by the seam and how the transformations are different. Note that the seam could be anywhere between A and B in theory and would still provide a good stitching.
EDIT: My apologies, the percentages in the image should be multiplied by 100.
I understand that if the images were warped correctly or at least lens corrected, then the seam would not matter. As then the pixels from left and right image would only differ due to parallax and be pretty much identical otherwise. However, I can move the images by hand and get a very good seam, even with these restrictions. So it must be possible to achieve this programmatically as well.
I guess my use case can be stated as such: I’m only interested in a correct stitching in one line (the seam) between two neighboring images. The images should only be adjusted in terms of translation, scale and z-rotation. Which is what I already have, with the exception that the seam is not the one I need.
you haven’t presented how you composite these pictures. it’s perfectly possible to pick which is on top of what. how to do that requires you to show some code.
as for the distortions… you need enough feature points in the overlap for any algorithm to find a good match.
you’ll want to investigate
how many feature descriptors you get for each picture
how many matches are good (Lowe’s ratio test after FLANN k-NN match)
how many remain after estimateAffinePartial2D
in absolute numbers. and share those so we can have a look too.
