I’d like to find new objects (new foreground blobs) and track them in color cameras with IR capabilities. Shadows, lighting changes, switching from color to IR should not identify lighting as foreground objects.
Very many of the evaluation sets ignore this (extremely) common scenario and pretty much all of the stock background subtraction algorithms can’t tell the difference between FG and BG when there are large lighting changes (e.g. someone turns on a light), and in particular when cameras switch to IR mode.
I’m thinking that looking at feature similarities may help identify what is the background vs foreground.