Mahjong Solver Script Issues

My script is 150 lines of code. GitHub - derkangkino/mahjongsolver
I am trying to make a Mahjong solver as an exercise to learn some opencv.
This is the game it plays: Mahjong Quest
There is a demo video on the github (however I am not familiar with the site).

I started out with feeding my script any static image. (
It identified everything correctly, and whether a tile was active or not.
If you run, the escape key refreshes every tile identified and whether it is active.

However, as soon as I have it going from repeated screen captures (, it starts misidentifying whether a tile is active or not, and additionally does not identify tiles such as the seven tile which it successfully identified in the above static image.

I am using tile matching.

I’m having difficulty diagnosing the issue now that it is live and I don’t have feedback on what it’s identifying, and identifying as active, like I did with the static image.