Signature recognition and extraction

Android/Java
1

I must extract a signature from a blank scanned image containing only the signature, which can be placed everywhere. The image may be also a smartphone picture.
I realized that OpenCV can fit well but I’m facing some problems.

I’m using the HSV analysis approach:

  • I defined the color threshold for the lower and upper bounds.
  • Exctacted a mask corresponding to the bounds.
  • Called the find contours method to get all the contours corresponding to the mask.
  • Merged all the contours matching the mask.
  • Extracted the signature.

Here is the problem: the bbox of the signature is smaller than what it is expected, the signature is recognized but the bbox borders do not cover the full signature.

Here is the code:

	public static synchronized void extractSignature(File signature) {

		if (signature == null)
			return;

		String fileExtension = FilenameUtils.getExtension(signature.getName());

		// check if the file extension is correct!
		if (!fileExtension.toLowerCase().equals("jpg") && !fileExtension.toLowerCase().equals("jpeg") && !fileExtension.toLowerCase().equals("png")) {
			return;
		}

		try {

			if (!isLoaded) {
				// System.loadLibrary(Core.NATIVE_LIBRARY_NAME); // not working
				// OpenCV.loadShared(); // not working
				OpenCV.loadLocally();
				isLoaded = true;
			}

		} catch (Exception e) {
			System.out.println(e.toString());
		}

		// Load image
		Mat image = Imgcodecs.imread(signature.getPath());
		Mat originalImage = image.clone();

		// Convert image to HSV color space
		Mat hsv = new Mat();
		Imgproc.cvtColor(image, hsv, Imgproc.COLOR_BGR2HSV);

		// Define lower and upper bounds for color threshold
		Scalar lower = new Scalar(90, 38, 22); 		// original 90 38 0
		Scalar upper = new Scalar(255, 255, 255);   // original 145 255 255
		
		// Threshold the HSV image to get only desired colors
		Mat mask = new Mat();
		Core.inRange(hsv, lower, upper, mask);
        
		// Find contours
		List<MatOfPoint> contours = new ArrayList<>();
		Mat hierarchy = new Mat();
		Imgproc.findContours(mask, contours, hierarchy, Imgproc.RETR_LIST , Imgproc.CHAIN_APPROX_SIMPLE);

		// Combine all contours into one and get bounding box
		MatOfPoint allContours = new MatOfPoint();
		int i = 0;
		for (MatOfPoint contour : contours) {
			List<Point> pts = contour.toList();
			System.out.println(i + " -> " + pts.toString());
			allContours.push_back(new MatOfPoint(Converters.vector_Point_to_Mat(pts)));
			i++;
		}

		Rect boundingBox = Imgproc.boundingRect(allContours);

		// Apply Green box to the ROI from the original image using the bounding box coordinates
		Imgproc.rectangle(image, boundingBox.tl(), boundingBox.br(), new Scalar(36, 255, 12), 2);

		// Add some pixels to each dimension of the bounding box for the final extraction
		int padding = 150;
		int x = Math.max(boundingBox.x - 150, 0); // width
		int y = Math.max(boundingBox.y - 55, 0); // height
		int width = Math.min(boundingBox.width + 2 * 150, image.cols() - x);
		int height = Math.min(boundingBox.height + 2 * 55, image.rows() - y);
		Rect paddedBoundingBox = new Rect(x, y, width, height);

		// Extract ROI
		// Mat ROI = new Mat(imageOriginal, boundingBox); // bbox without padding
		Mat ROI = new Mat(originalImage, paddedBoundingBox); // bbox with padding

		// Convert ROI to grayscale
		Mat grayROI = new Mat();
		Imgproc.cvtColor(ROI, grayROI, Imgproc.COLOR_BGR2GRAY);
		
        // Convert to B/W
		Imgproc.threshold(ROI, ROI, 150, 255, Imgproc.THRESH_BINARY);


		// Save and display images
		//Imgcodecs.imwrite(FileUtils.getSignaturesFolder().getAbsolutePath() + File.separator+ "result.jpg", result);
		Imgcodecs.imwrite(FileUtils.getSignaturesFolder().getAbsolutePath() + File.separator + "image_with_bbox.jpg", image);
		Imgcodecs.imwrite(FileUtils.getSignaturesFolder().getAbsolutePath() + File.separator + signature.getName(), ROI);
		Imgcodecs.imwrite(FileUtils.getSignaturesFolder().getAbsolutePath() + File.separator + "grayROI.jpg", grayROI);


		return;
	}

To avoid a bit of the problem I manually increased the BBOX, but it is not a solution. I’m new to OpenCV so whatever suggestion would be highly appreciated even a completely new approach.

Original Image + Green BBOX extracted

image_with_bbox
image_with_bbox2480×3508 76.8 KB

2

that’s the actual weakness here. once the bg changes, it’s all broken.

what if you try ‘higher level’ things, like text detection cnn’s ?

1 Like
3

Hi berak! Thank you, yes you are right this is “working” only till the bg is white.

Could you please share a link to an example? or to some useful documentation to start using the text detection CNN?

4

east, also mser tut

last (and quite serious) – scratch your head, if implementing signature verification, being a total noob, sound like a good idea at all (not to me …)

5

Hi Berak, just to give you a full picture. In reality, I need only extract the signature to be applied later to a contract. I don’t need to verify it. Unfortunately with the technology I have in my web app I cannot implement an upload and crop tool for my user’s, so I thought it could be a good idea to allow users to upload a scanned version of their signature so I would later crop the signature using opencv. But it seems that I underestimated the problem.

Update …

Thank you for your suggestion!
I leave here my new final solution that seems to work with good accuracy for all my test cases.

These are the main steps implemented:

  • Convert the image to gray
  • Detected regions
  • Calculated the region’s centroid
  • Added two thresholds one vertical and one horizontal
  • Created a bounding box with all the regions respecting the threshold distances with the centroid
  • Extracted the ROI using the bbox

It is working with all my test cases. So, it is not a super elegant solution but it works with scanned documents where there is only a signature to be cropped and stored.

	public static synchronized void extractSignature(File signature) {

		if (signature == null)
			return;

		String fileExtension = FilenameUtils.getExtension(signature.getName());

		// check if the file extension is correct!
		if (!fileExtension.toLowerCase().equals("jpg") && !fileExtension.toLowerCase().equals("jpeg") && !fileExtension.toLowerCase().equals("png")) {
			return;
		}

		// Load OpenCV library
		try {

			if (!isLoaded) {
				// System.loadLibrary(Core.NATIVE_LIBRARY_NAME); // not working
				// OpenCV.loadShared(); // not working
				OpenCV.loadLocally();
				isLoaded = true;
			}

		} catch (Exception e) {
			System.out.println(e.toString());
		}

		// Load image from file
		Mat image = Imgcodecs.imread(signature.getPath());
		// Make a copy of the original image
		Mat originalImage = image.clone();

		// Convert the image to grayscale
		Mat gray = new Mat();
		Imgproc.cvtColor(image, gray, Imgproc.COLOR_BGR2GRAY);

		// Create MSER and Detect MSER regions
		MSER mser = MSER.create();
		List<MatOfPoint> regions = new ArrayList<>();
		MatOfRect bboxes = new MatOfRect();
		mser.detectRegions(gray, regions, bboxes);

		// Calculate the Centroid of all MSER regions
		int totalX = 0;
		int totalY = 0;
		int totalPoints = 0;
		for (MatOfPoint region : regions) {
			Point[] points = region.toArray();
			for (Point point : points) {
				totalX += point.x;
				totalY += point.y;
				totalPoints++;
			}
		}

		Point centroid = new Point(totalX / totalPoints, totalY / totalPoints);


		// Draw the Centroid
		Mat centroidM = image.clone();
		Imgproc.circle(centroidM, centroid, 5, new Scalar(255, 0, 0), -1); // Draw the Centroid as a blue dot

		
		// Define vertical and horizontal distance thresholds (adjust as needed)
		double verticalThreshold = 200.0; // Vertical distance threshold
		double horizontalThreshold = 700.0; // Horizontal distance threshold

		// Find the bounding box that covers filtered MSER regions
		int minX = Integer.MAX_VALUE;
		int minY = Integer.MAX_VALUE;
		int maxX = Integer.MIN_VALUE;
		int maxY = Integer.MIN_VALUE;

		for (MatOfPoint region : regions) {
		    Point[] points = region.toArray();
		    for (Point point : points) {
		        double verticalDistance = Math.abs(point.y - centroid.y);
		        double horizontalDistance = Math.abs(point.x - centroid.x);

		        if (verticalDistance <= verticalThreshold && horizontalDistance <= horizontalThreshold) {
		            if (point.x < minX) minX = (int) point.x;
		            if (point.y < minY) minY = (int) point.y;
		            if (point.x > maxX) maxX = (int) point.x;
		            if (point.y > maxY) maxY = (int) point.y;
		        }
		    }
		}

		// Draw the bounding box if valid points were found
		if (minX < Integer.MAX_VALUE && minY < Integer.MAX_VALUE && maxX > Integer.MIN_VALUE && maxY > Integer.MIN_VALUE) {
			Imgproc.rectangle(image, new Point(minX, minY), new Point(maxX, maxY), new Scalar(0, 255, 0), 2);
		}

		// Draw convex hulls
		Mat grayImg = image.clone();
		for (MatOfPoint region : regions) {
			MatOfInt hullIndices = new MatOfInt();
			Imgproc.convexHull(region, hullIndices);

			// Convert hull indices to points
			Point[] regionArray = region.toArray();
			List<Point> hullPoints = new ArrayList<>();
			for (int index : hullIndices.toArray()) {
				hullPoints.add(regionArray[index]);
			}
			MatOfPoint hull = new MatOfPoint();
			hull.fromList(hullPoints);

			// Draw the hull on the image
			List<MatOfPoint> hulls = new ArrayList<>();
			hulls.add(hull);
			Imgproc.polylines(grayImg, hulls, true, new Scalar(0, 0, 255), 2);
		}

		// Extract ROI, crop the original image using the bounding box
		Rect ROI = new Rect(minX, minY, maxX - minX, maxY - minY);
		Mat croppedImage = new Mat(originalImage, ROI);

		// Convert ROI to grayscale
		Mat grayROI = new Mat();
		Imgproc.cvtColor(croppedImage, grayROI, Imgproc.COLOR_BGR2GRAY);

		Imgproc.threshold(croppedImage, croppedImage, 150, 255, Imgproc.THRESH_BINARY);

		// Save and display images
//		Imgcodecs.imwrite(FileUtils.getSignaturesFolder().getAbsolutePath() + File.separator + "image_with_bbox.jpg", image);
		Imgcodecs.imwrite(FileUtils.getSignaturesFolder().getAbsolutePath() + File.separator + signature.getName(), croppedImage);
//		Imgcodecs.imwrite(FileUtils.getSignaturesFolder().getAbsolutePath() + File.separator + "grayROI.jpg", grayROI);
//		Imgcodecs.imwrite(FileUtils.getSignaturesFolder().getAbsolutePath() + File.separator + "last.jpg", grayImg);
//		Imgcodecs.imwrite(FileUtils.getSignaturesFolder().getAbsolutePath() + File.separator + "centroid.jpg", centroidM);

		return;
	}
1 Like