You’re converting the color images to gray-scale in your code, so it’s no doubt keying on something else.
From your initial post, I thought that there were only two categories. You’ve got five. So I’m guessing that color range alone won’t be sufficient. But you should look at the RGB range (actually BGR in OpenCV) of the pixels that you’re interested in. You should be able to figure a range, or multiple ranges, that represent the leaves. For example, you don’t need any of the black, white/gray, or spectral blue pixels from your cell phone image.
Also, I think that scaling will be important. IOW, distance from your camera to the objects. Again, I thought that the camera was going to be mounted higher above. Your test images should be representative of the camera distance, lighting, etc.