Finding the Cut From HERE Part with OpenCV

Stranger · March 25, 2021, 12:03pm

Hello there,

I am using OpenCV in NodeJS.

I want to crop a certain part of image and for that i need to find a landmark in the image.

This is the code i wrote:

function installDOM() {
    const dom = new JSDOM();
    global.document = dom.window.document;
    global.Image = Image;
    global.HTMLCanvasElement = Canvas;
    global.ImageData = ImageData;
    global.HTMLImageElement = Image;
}

function loadOpenCV() {
    return new Promise(resolve => {
        installDOM();
        global.Module = {
            onRuntimeInitialized: resolve
        };
        global.cv = require("./opencv");
    });
} 

function cropImage(PDF) {
    PDFToImage(PDF).then((buffer) => {
        /**
         * Converted PDF to JPEG Image
         * Now Use the Image Buffer as input for sharp and trim to content
         */
        sharp(buffer).trim().toBuffer()
            .then(async (buffer) => {
                /**
                 * Removed White Space
                 * Now find the location of dashed line part with openCV
                 */
                let trimImage = sharp(buffer);

                let img1 = await loadImage(buffer);
                let img2 = await loadImage("./find.jpeg");

                img1 = cv.imread(img1);
                img2 = cv.imread(img2);

                let canIn = createCanvas();
                cv.imshow(canIn, img1);

                let canFi = createCanvas();
                cv.imshow(canFi, img2);

                const src = cv.imread(canIn);
                const temp = cv.imread(canFi);

                let dest = new cv.Mat();
                let mask = new cv.Mat();

                cv.matchTemplate(src, temp, dest, cv.TM_CCOEFF_NORMED, mask);

                let result = cv.minMaxLoc(dest, mask);
                let maxPoint = result.maxLoc;

                src.delete(); dest.delete(); mask.delete();

                // Position from top where the dashed line was found
                let cvTop = maxPoint.y + 8;

                trimImage.metadata().then((meta) => {
                    /**
                     * Get width and height of image using the metadata 
                     */
                    let { width, height } = meta;

                    trimImage
                        .extract({ top: cvTop, left: 0, width: width, height: (height - cvTop) })
                        .toFile("./croppedFinal.jpeg");
                });
            })
    })
};

loadOpenCV().then(() => {
    let name = "SHAS2002_OCT2019.pdf"
    cropImage("./pdfs/" + name);
})

The code is created while following the tutorial in the docs.

The Problem i am facing is that on some images the code is working fine but on some images it is throwing errors:

abort(6626128). Build with -s ASSERTIONS=1 for more info.
Thrown at:
    at C:\Users\mss2015\Desktop\debugPDFs\opencv.js:30:1693
    at emit (events.js:315:20)
    at internal/process/execution.js:156:25

please see this stackoveflow question for refferance about the question.

This is the template image

I removed the scissorss from the template as the scissors can be at left or right side.

This is throwing errors mostly. And always when there is a slight change in the background color. Most of the time the background color is white but in some cases the background color is a bit greenish.

I haven’t built the openCV i am using the javascript file from here
https://docs.opencv.org/4.5.1/opencv.js

How can i make it work ? and speed it up if possible as i will be using it in cloud functions. Currently i am testing it on my pc.

Alejandro_Silvestri · March 25, 2021, 12:29pm

@Stranger

Please tell us in what point or you code the error arise. I believe the error is thrown in an internal OpenCV event, so the message won’t point out the line in you code that initiate it.

You can reproduce the error, and the try to run you code until some random point to see if the error is thrown, and then decide if the problem is before or after that point.

I’m adding you image to better explain you are trying to detect those cutting lines at the bottom.

Stranger · March 25, 2021, 6:00pm

Hello, Alejandro. I’ve find out that the width of the template image is bigger then the base image. so in that case it’s making the mess. I tried to make the width of template image same as width of the base image while creating the canvas. But i didn’t find success there. I can’t reduce the width of the template image as if the base image’s width is bigger it’s going to create the mess again.

Overall, How to keep the template images width same as the width of base image ?
I tried this:

   let img1 = await loadImage(buffer); 
   let img2 = await loadImage("./find.jpeg");
   img1 = cv.imread(img1); 
   img2 = cv.imread(img2);
   let canIn = createCanvas();
   cv.imshow(canIn, img1);
   let { width, height } = await trimImage.metadata();
   let canFi = createCanvas(width, height); 
   cv.imshow(canFi, img2)

Update:

I Managed to handle the width of template by resizing it but i don’t think so that it’s efficient in any way. The Current code looks like this:

sharp(imagePath).trim().toBuffer()
        .then(async (buffer) => {
            /**
             * Removed White Space
             * Now find the location of dashed line part with openCV
             */
            let trimImage = sharp(buffer);

            let { width, height } = await trimImage.metadata();

            let findBuffer = await sharp("./find.jpeg").resize(width).toBuffer();

            let img1 = await loadImage(buffer);
            let img2 = await loadImage(findBuffer);


            img1 = cv.imread(img1);
            img2 = cv.imread(img2);

            let canIn = createCanvas();
            cv.imshow(canIn, img1);

            let canFi = createCanvas();
            cv.imshow(canFi, img2);

            const src = cv.imread(canIn);
            const temp = cv.imread(canFi);

            let dest = new cv.Mat();
            let mask = new cv.Mat();

            cv.matchTemplate(src, temp, dest, cv.TM_CCOEFF_NORMED, mask);

            let result = cv.minMaxLoc(dest, mask);
            let maxPoint = result.maxLoc;
            src.delete(); dest.delete(); mask.delete();

            // Position from top where the dashed line was found
            let cvTop = maxPoint.y + 8;


            trimImage
                .extract({ top: cvTop, left: 0, width: width, height: (height - cvTop) })
                .toFile("./croppedFinal.jpeg");
        })

Other thing is it won’t work if the background color is not white. Sometimes the background color is bit greenish so how can i handle that.

Alejandro_Silvestri · March 25, 2021, 9:58pm

There are so many ways to try and see. Random thoughts:

search twice, with two patterns, one for white paper, another for green paper
convert image to grayscale
convert to grayscale and binarize with threshold
look for a single segment pattern, not the entire line. Then, in the result, look for a dotted horizontal line
use morphological operation hit-or-miss:

https://docs.opencv.org/master/db/d06/tutorial_hitOrMiss.html

Stranger · March 26, 2021, 11:14am

Good afternoon Ale,

I tried to work out your random thoughts on this problem but didn’t found any solutions yet

For Thought 1, I can’t do that. The code needs to be executed within 4 seconds and the work, other than the OpenCV work, is taking 2500ms so I can’t do multiple searches.

For Thought 2, I converted the base and template image to grayscale using the Sharp Library but it didn’t work.

For Thought 3. I am not sure that I had done it in the right way but it didn’t work either.

This was the code i used for the 3rd thought:

sharp(imagePath).trim().toBuffer()
        .then(async (buffer) => {
            /**
             * Removed White Space
             * Now find the location of dashed line part with openCV
             */
            let trimImage = sharp(buffer);

            let { width, height } = await trimImage.metadata();

            let findBuffer = await sharp("./findAA.jpeg").resize(width).toBuffer();

            let img1 = await loadImage(buffer);
            let img2 = await loadImage(findBuffer);


            img1 = cv.imread(img1);
            img2 = cv.imread(img2);

            let canIn = createCanvas();
            cv.imshow(canIn, img1);

            let canFi = createCanvas();
            cv.imshow(canFi, img2);

            let src = cv.imread(canIn);
            const temp = cv.imread(canFi);

            let dest = new cv.Mat();
            let mask = new cv.Mat();

            cv.threshold(src, dest, 177, 200, cv.THRESH_BINARY);
            cv.threshold(temp, mask, 177, 200, cv.THRESH_BINARY);

            cv.matchTemplate(src, temp, dest, cv.TM_CCOEFF_NORMED, mask);

            let result = cv.minMaxLoc(dest, mask);
            let maxPoint = result.maxLoc;
            src.delete(); dest.delete(); mask.delete();

            // Position from top where the dashed line was found
            let cvTop = maxPoint.y + 8;


            trimImage
                .extract({ top: cvTop, left: 0, width: width, height: (height - cvTop) })
                .toFile("./croppedFinal.jpeg");
        })

For Thought 4, I don’t know what it is. Just started with OpenCV. So i don’t have much knowledge of its patterns.

For Thought 5, I Followed the link but I didn’t get it. sry.

Below is that greenish background image on which the code is not working quite well.

Alejandro_Silvestri · March 26, 2021, 10:50pm

@Stranger

It seems you have a lot to learn. But it’s fun!

About threshold:
https://docs.opencv.org/master/d7/dd0/tutorial_js_thresholding.html
Try cv.THRESH_OTSU | cv.BINARY

About morphological operations:
https://docs.opencv.org/master/d4/d76/tutorial_js_morphological_ops.html

Topic		Replies	Views
[opencv.js] guidance on most appropriate use of cv.findContours	10	1768	March 14, 2021
(JS) grabCut to findContours understanding masking opencvjs , imgproc	2	827	March 25, 2021
OpenCV JS - Rectangle and edges opencvjs , imgproc , core	0	720	March 30, 2021
Driveway detection using color masking Python	0	309	August 20, 2021
How do I create the best object finding mask? imgproc	4	337	February 7, 2021

Finding the Cut From HERE Part with OpenCV

Related Topics