Undoing a perspective transform results in the initial image but warped

[WJ_Lee]

Building a front-end only image processing application and I am struggling to perform an un-transform logic for my image.
FYI, I am using https://docs.opencv.org/4.7.0/opencv.js

I have a canvas that shows an image from a user upload.
The user then can select four points of the image and see a birds-eye-view perspective. After the image has transformed, the user can further add shapes onto it.

When the user is done adding things on to the transformed image, I want to be able to untransform the image back to the original with the added shapes.

here’s my transform logic:

export function transformPerspective({
  originalImage,
  padding,
  srcPoints,
}: Omit<
  TransformPerspective,
  "inverseMatrix" | "transformedImage"
>): Promise<Transform> {
  return new Promise((resolve) => {
    let img = new Image();
    img.id = "konvaImage";
    img.src = originalImage;
    img.style.display = "none";
    document.body.appendChild(img);

    img.onload = () => {
      let srcTri = cv.matFromArray(4, 1, cv.CV_32FC2, srcPoints);
      const dstWidth = 900;
      const dstHeight = 1000;

      let dstTri = cv.matFromArray(4, 1, cv.CV_32FC2, [
        padding,
        padding,
        dstWidth - padding,
        padding,
        dstWidth - padding,
        dstHeight - padding,
        padding,
        dstHeight - padding,
      ]);

      let matrix = cv.getPerspectiveTransform(srcTri, dstTri);
      const inverseMatrix = matrix.inv(cv.DECOMP_LU);
      let dsize = new cv.Size(dstWidth, dstHeight);
      let myimg = cv.imread(
        document.querySelector("#konvaImage") as HTMLImageElement
      );
      let warpedImage = new cv.Mat();
      cv.warpPerspective(myimg, warpedImage, matrix, dsize);

      const outputCanvas = document.createElement("canvas");
      outputCanvas.id = "outputCanvas";
      outputCanvas.width = 900;
      outputCanvas.height = 1000;
      outputCanvas.style.display = "none";
      document.body.append(outputCanvas);

      cv.imshow("outputCanvas", warpedImage);
      const imageFromCanvas = new window.Image();
      imageFromCanvas.src = outputCanvas.toDataURL();

      // cleanup
      myimg.delete();
      warpedImage.delete();
      matrix.delete();
      srcTri.delete();
      dstTri.delete();
      outputCanvas.remove();

      resolve({ imageFromCanvas, inverseMatrix });
    };
  });
}

and here’s my un-transform logic:

export function reverseTransformPerspective({
  inverseMatrix,
  padding,
  srcPoints,
  transformedImage,
}: Omit<TransformPerspective, "originalImage">): Promise<HTMLImageElement> {
  return new Promise((resolve) => {
    let img = new Image();
    img.id = "konvaImage";
    img.src = transformedImage;
    img.style.display = "none";
    document.body.appendChild(img);

    img.onload = () => {
      let dstTri = cv.matFromArray(4, 1, cv.CV_32FC2, srcPoints);

      const srcWidth = img.width;
      const srcHeight = img.height;

      let srcTri = cv.matFromArray(4, 1, cv.CV_32FC2, [
        padding,
        padding,
        srcWidth - padding,
        padding,
        srcWidth - padding,
        srcHeight - padding,
        padding,
        srcHeight - padding,
      ]);

      let matrix = cv.getPerspectiveTransform(srcTri, dstTri);
      let dsize = new cv.Size(srcWidth, srcHeight);

      let myimg = cv.imread(img);
      let unTransformedImage = new cv.Mat();
      cv.warpPerspective(myimg, unTransformedImage, inverseMatrix, dsize);

      const outputCanvas = document.createElement("canvas");
      outputCanvas.id = "outputCanvas";
      outputCanvas.width = srcWidth;
      outputCanvas.height = srcHeight;
      outputCanvas.style.display = "none";
      document.body.append(outputCanvas);

      cv.imshow("outputCanvas", unTransformedImage);
      const imageFromCanvas = new window.Image();
      imageFromCanvas.src = outputCanvas.toDataURL();

      // cleanup
      myimg.delete();
      unTransformedImage.delete();
      matrix.delete();
      srcTri.delete();
      dstTri.delete();
      outputCanvas.remove();

      resolve(imageFromCanvas);
    };
  });
}

again the un-transform works, slightly, however the end result shows the transformed image (with the added shapes) and it is warped.

I’m trying to get the original image with the shapes
any ideas?

[WJ_Lee]

initial image:

[WJ_Lee]

selecting points / area to transform

[WJ_Lee]

transformed image

[crackwitz]

don’t take badly cropped screenshots. save the images themselves.

save them such that the output is the same size as the input. that way I can trivially compare them.

why do you not invert the homography? why do you recalculate it (or its alleged inverse) using getperspectivetransform and whatever points?

have you compared the two point lists you are using? are they the same? if not, why not? they must be. if they are not, why do you expect the transform to be the same?

[WJ_Lee]

un-transformed image

[WJ_Lee]

@crackwitz

thank you for the questions. I re-uploaded the images above (and one below) for your reference!

have you compared the two point lists you are using? are they the same?

Yes they are the same. Currently, I’m saving them as a url query param, so that I can 1) to make sure they are the same 2) get them easily when I’m doing the inverse.

below is kinda what I’m trying to get: the shape inside the new canvas, but of course it needs to be ontop of the original image

[WJ_Lee]

I kept it as is and just overlayed it on top of the original image. it works just fine. thanks for your comment.