surface inspection.
companies make lots of money with knowing how to do that. and they keep their trade secrets. there might be scientific publications but those are scientific, i.e. not industrially proven.
you have correctly identified that reflection/lighting plays a crucial role in this. failure to do this right cannot be corrected after image acquisition. you might need to move light/workpiece/camera to acquire multiple pictures. you might have to do that with mechatronics (“robotics” but not smart).
your first task should be to figure out how to take pictures that show the surface condition well.
“well” means “machine-readable”, i.e. good contrast.