Help figure how to count shapes in noisy picture

[Mathieu_Parent]

Hello everyone, here’s the deal. I’m trying to count the number of shapes visible in an image, the shapes can only be triangles or circles (I think this point will help on the counting at the end).

Here is the current state of my program in image, the steps go in order from left to right, top to bottom.

1. Original image
2. Blurred image to take out the noise the best I can. The blur used is a GaussianBlur with a [7,11] kernel as it seemed to provide better results when the longer side was following the background lines.
3. I recreate a background “guess” by taking the trimmed average (removing 10% lowest and 10% highest values)
4. I create a diff between picture 2 and 3
5. Apply a threshold to remove most of the noise artifacts
6. Turn the image into grayscale and threshold again to have a binary
7. Get all contours
8. Remove contours of shapes with small area
9. Reduce the amount of edges using approxPolyDP

Please note that after step 6, I compare the number of white pixels vs black ones. Whites should account for less than 40% of the image, if not this means the isolation did not work correctly because the background lines are vertical instead of horizontal so I redo steps 1 to 6 with everything rotated 90degrees.

This is the result of a long trial and error process, as I do not really know what I am doing nor what tools are at my disposal here…

The next challenge I have concerns cut shapes like on the image above, and overlapping shapes. I need to figure a way to join this triangle so I can count the number of contours that I have to know the amount of shapes.

As for the overlapping shapes. I thought about incrementing count by 1 for each convex shape and by 2 for concave ones, but 3 shapes can overlap, and the current contours I have are creating concave shapes even for a simple triangle here.

Would you have any suggestions on how I could better create the binary image for shape detections? Or how would you go from here to counting the shapes?

Thank you in advance.

[Mathieu_Parent]

Here are two more examples to give you an idea of the challenges I’m facing.

[Mathieu_Parent]

[crackwitz]

smells like a type of captcha.

as such, use AI.

any “traditional” (hand crafted) CV is bound to require “easy” data. as you see in the very first example, some areas of some shapes aren’t colored distinctly from the background. take the green rows through the green triangle. the shapes aren’t fully defined. it’s like an optical “illusion”. some parts of it suggest the shape, but other parts don’t suggest it but also don’t refute it.

overcoming that requires enough machine cognition to look for incomplete shapes. nothing you’ve done in your 9-step plan counts as cognitive.

[Mathieu_Parent]

Thanks for your reply. I was hoping to not receive this answer I am not skilled enough to train an AI, and I’m pretty sure I do not have a big enough dataset to do it any ways.

I was hoping that, with maybe some other steps, I’d be able to close the gap, as I’m already getting 90% of the results… I know, the last 10% is always 90% of the work but yeh.

You certainly know way more than me, but I am sure we could get better results with simple CV even if not perfect ones. This is more of a project to learn the CV field and algorithms than providing a full blown paid captcha service…