Identify Potatoes

[jogl]

I am trying to identify the potatoes in the following image and place boxes around them.

Ultimately, the box location would be a target location for a pick and place to pick up the potatoes.

After a lot of experimenting, cutting and pasting and generally learning a little about OpenCV I have this which I think is looks promising, but needs a lot more work.

import numpy as np
import cv2

image = cv2.imread("DSCN0901.JPG")
blurred_image = cv2.GaussianBlur(image, (55,55), 0)
gray = cv2.cvtColor(blurred_image, cv2.COLOR_BGR2GRAY)
_, threshold = cv2.threshold(gray, 150, 255, cv2.THRESH_BINARY)

contours, hierarchy = cv2.findContours(threshold, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
height, width  = threshold.shape
min_x, min_y = width, height
max_x = max_y = 0;

for cnt in contours: 
    approx = cv2.approxPolyDP(cnt, 0.005*cv2.arcLength(cnt, True), True)
    (x,y,w,h) = cv2.boundingRect(approx)
    min_x, max_x = min(x, min_x), max(x+w, max_x)
    min_y, max_y = min(y, min_y), max(y+h, max_y)
    rect = cv2.minAreaRect(approx)
    box = cv2.boxPoints(rect)
    box = np.int0(box)
    cv2.drawContours(image, [box],0,(0,0,255),2)

cv2.imshow("threshold", threshold)
cv2.imshow("blurred", blurred_image)
cv2.imshow("image",image)

cv2.waitKey(0)
cv2.destroyAllWindows()

Resulting in:

As you can see the boxes are catching some of the potatoes ok but get fooled by the brushes in some instances.

Any ideas on how I could more reliably identify the potatoes only?

[crackwitz]

cvtColor to HSV or comparable space, then investigate what you see.

basically, your scene is hard to work with. situation would benefit from changes to lighting and maybe even the hardware (those brushes).

do you know of any type of light that gives strong contrast between the taters and the brushes?

this is what I get from selecting suitable ranges in hue, saturation, and value, and and-ing them together. the noise is due to your red rectangles and jpeg compression.

[darenw]

I see a big difference in texture between potato and brush. Enhance high-frequency detail. Maybe some established texture measure, one to pick up on “fast and busy” elongated fine detail. Parts of the image rating high in that can be marked as Brush. Dilate those regions as far as sameness of color allows. Anything rating low in that texture measure, and having potatoesque color, is Potato. Then all pixels not in one type of blob or the other need to be categorized, which my first inclination is to do by color. Tweak the details to ignore creases, dents, discolorations on the potatoes.

A texture measure that comes to mind is to compute the ratio of eigenvalues in a matrix formed as the outer product of gradient-x and gradient-y. But that’s just what comes off the top of my mind, not from actually working on a sample image.

[crackwitz]

some improved results with median filter and morphological opening. it’s just a heavy median operator to remove noise, conversion to HSV, some inRange and intersection, and morphological opening to remove noise

[jogl]

Thank you guys for your responses. I have been fooling around with HSV and inRange.

Programming aside for a bit, I have a few questions about “lighting and hardware” as mentioned by Crackwitz. I am going to experiment with some lighting. The photo that I posted was taken with just the ambient light from mercury HID high bays. I intend to try some different light sources (ie.: LED’s) and try different lighting angles.

So, just using off the shelf lights do you think that lower colour temperature lights (ie.: 3500 *K) or higher ones, 5000 or 6000 *K would give better results?

And on other hardware, what would be the best colour for the brushes? I am pretty sure that I can get that style in black, and may be able to get them in other colours.

Thanks again,
JP.

[crackwitz]

black brushes would give great contrast, better than these off-white ones.

color temperature: try everything, even monochromatic light (red/green/blue). none of that would have nearly as much effect as the strongly contrasting brushes have, so you can afford to not experiment with color temperature.

right now the lights appear to be top-down (high angle, ~90 degrees). I would place lights at a lower angle (~45 degrees), perhaps as low as almost from the sides. that will give you less reflection off flat areas (also the potatoes), and illuminate the sides of your objects better, which gives their edges greater contrast to the background.

all this would give you more robust segmentation. what I achieved there was using minute tweaks to thresholds that highly depend on color temperature of your lights, what white balance the camera is set to, and probably also what breed (?) of potato you’re looking at.

[tmanthey]

The way you do it the system will depend on stable lighting conditions very much (the threshold variable in findContours) which is unlikely to happen in agriculture. This is a classic use case for Deep Learning. Haven’t tried yet the inference of opencv but we could you provide a professional training set or neural network.

[crackwitz]

automated optical inspection for industrial use always depends on stable lighting. that’s the most robust way to ensure reliable results.

deep learning for this application is a lot of effort to fix a problem (lighting) you could just avoid even happening by simply modifying lights and some of the machinery.

[kpachinger]

I just saw a potato brusher on YouTube. They tend to spin and bounce, bump back off others, etc. You may want to juggle video recognition and tracking.

[crackwitz]

good point! or make the brushes stop for a second until they’ve stopped dancing, pick the area clean, and resume brushie

[jogl]

@crackwitz

make the brushes stop

That would be the plan, since that is the way the line works now. The line goes faster than we can (manually) pick and grade the taters, so I have it programmed to start and stop.

I will do some experimenting this weekend with lighting, brushes (I have some old black brushes in the scrap pile, good thing I did not throw them out), and with morphological and smoothing operations.

The pack shed is a little busy this time of year so my progress is slow.

JP.

[jogl]

Finally have done some experimenting with black brushes and lighting and results are very encouraging.

The following are screen grabs of the first thing that fell out of my attempts using blurring and thresholding.

Thanks for the advice, I’ll surely be back for more,
JP.

[jogl]

Making a little more progress. Next I would like to be able to isolate each potato and find the defects.

In the attached image I would like to isolate the top left hand potato and then identify the hole. I am pretty sure that using using thresholding it would be possible to identify the hole. But how do I limit the search to just the masked off area of the potato?

Thanks,
JP.

[crackwitz]

since “potato” is still a sizable fraction of the picture, restricting to those regions will probably gain you just that fraction, and the extra work specifying/obeying the regions may take up the time saved.

depends on what does “the search”. you can take a bounding box for each potato and work on the submatrix of that bounding box. that’s probably the least overhead. some functions also take mask arguments.

if you’re only going to threshold for dark spots, you could pick that threshold relative to the average color/brightness of the visible potato(es) and run that on the entire picture, then take intersection (for masks, bitwise and equals logical and) as needed.