I went down the same path, even to the point of using red and green squares. The only difference is I had a single red square in the upper right and a green one in the lower left.
My advice is not to do this at all, unless you will always have the exact same lighting, the exact same shade of white for a background, and never have any other objects/backgrounds visible. Basically the color approach works with a fixed image in a file, but not if you are using a camera. Because you will go crazy trying to distinguish “red” or “green” under varying circumstances.