Photonic Chip Image Recognition

[R.W.V]

I have photos of these photonic chips (top left) for which I need to create and overlay highlighting its shape. I dont get the full chip in the image, only a corner. I have tried using Canny and HoughlinesP to recognize the outside edges but these images arent of the highest quality and the recognized lines are several and very short so i have tried to write my own clustering algorithm that combines them but these lines are hard to work with: Image of houghlines image is bottom left

So I thought ill try to use findcontours to outline the whole square and heres that image: (bottom right)

And i tried filtering by the longest contour to get just the outline of the two sides of the square but that doesnt really work i just get some gibberish dots inside the chip

Im just not sure where to go from here because these sides of the chip arent imaged very well so its hard to get a good one line outline, instead its a bunch of small discontinuous lines. what canny sees is top right by the way

If theres another approach you know I would appreciate any advice.

[crackwitz]

Yeah, Canny and Hough are newbie traps. Let’s put those aside and back up.

@R.W.V

create and overlay highlighting its shape

That is a means to some end. What’s the end?

[R.W.V]

The probe, or pick you see facing the chip will align with one of the waveguides and the goal is to use computer vision to recognize both shapes and with precise stage motors be able to align them. The goal is to develop automated photonic integrated circuit characterization setups. And a neural network will be used to do this automatically I think. Im just an intern so this is as far as my knowledge goes about the project.

[crackwitz]

the resolution you present (a small screenshot) should not be worked with. you need better resolution than that.

I see moire patterns even in the top left quarter of the screenshot.

the rainbow colors across its surface also seem odd to me.

that makes it hard to figure out what’s actually there and what is just some effect of a ruined picture.

if anything can be done with the lighting, physically, that should be addressed first.

I know nothing about photonic waveguides, why they would show up in this image, or how they show up, physically. I can’t tell if they themselves are dark, or if the dark features are just shadows.

all these lines, at the edge, appear perpendicular to the edge. that’s good.

assuming the sample is oriented to be straight, so that edge of it aligns with the pixel grid.

with decent source data, you could just locate the minima along a line across the sample.

since that needle hopefully never changes appearance, manually take a crop of it, store it, manually annotate the tip of it, and then use the stored template of this needle to locate it in the image.

---

if you’re the intern, they are expected to have the experience and guide you, rather than sending you off to solve their problem for them. they should perhaps consult with whatever local EE/CS department you guys have for appropriate machine vision solutions to this. and even before that, they should consult with friendly departments that do similar things (building and imaging such waveguides), for tips on how they solved this. no need to start from scratch.

[R.W.V]

Thanks you. Here is the original chip photo, the top left one was one I applied a high contrast filter on to try to make it sharper. All the dark lines on the chip are waveguides that are part of the chip, and I’ll inquire about the lighting.

“with decent source data, you could just locate the minima along a line across the sample.”
How would I go about doing this? what do you mean by source data? Should i completely drop the canny and houghlines method?

“and then use the stored template of this needle to locate it in the image.”
How do I locate a template in an image?
Can this all be done in OpenCV?

I’d just like to know the best library or framework to use because i dont want to waste more time on something that wont work. Thanks for your help.

[crackwitz]

IDK if OpenCV is “the best” but it’s certainly useful.

Work in Python, in a Jupyter notebook, with numpy arrays. You can mix and match libraries, such as OpenCV, diplib, dlib, scipy, scikit-image, imageio, tifflib, …

Yes, ditch Canny and Hough. They are useless here.

assuming the latest picture, being 800 by 600, has not been processed in any way, I would consider it source data.

take a scanline from there (green line, picked manually). maybe take a few adjacent rows and average them.

the profile looks like that (plot). there is some noise, probably edge sharpening done by the camera’s driver, which could be suppressed with a lowpass (another plot).

then, your task is simply to find extrema/peaks. scipy already has a function for that or you can devise your own.

as for the probe, template matching should do well.

[R.W.V]

ok thanks a lot I’ll look into it