“I want to track color” could mean so many different things - more detail (and example images / image sequences) would help.
I would suggest you google “color constancy problem” and familiarize yourself with that. “Tracking color” is a great example of a vision task that seems easy, but is in fact challenging.