How to preprocess video for better OpenCV tracking?

I am trying to improve my webcam based OpenCV mouse controller for disabled people (MFC C++ application): https://preability.com/face-controlled-mouse/ The cursor moves, when a person moves her/his head, clicks when smile, etc.

Controller finds face area then use goodFeaturesToTrack, cornerSubPix and calcOpticalFlowPyrLK. In general, I managed to stabilize cursor. What I use now:

  1. Evaluating and filtering the direction of the each corner point movement.
  2. Spreading the corner points all over the face area for cv::goodFeaturesToTrack() helped a little too.
  3. EWMA (or Kalman) filter for the cursor position.
  4. I’ve included equalizeHist() for the face ROI. The detector performed much better in low light conditions.
  5. In addition, I tried morphology operations of OpenCV without improvement.

However, the corner points still dance in uneven lighting. I can see that similar old program eViacam has preprocessing module for webcam Creavision (old too) and the corner points are more stable.

Please advise what can be done with the input Mat? Or how can the video be processed with reasonable CPU loading?

1 Like

try samples/python/mosse.py, which does correlation tracking, and if you dig into it, you can extract subpixel positions too

related:

1 Like

(post deleted by author)

(post deleted by author)

Thank you! I tried various trackers (MIL, KCF, etc) for moving objects. But when I turn my head, the bounding rectangle changes position slightly. Sure, I can divide the face area into segments and track, for example, nose, eyes, etc. separately. But in this case, I think the CPU load will greatly increase. Many people use weak laptops.
So, wouldn’t it be better for facial corner points to use optical flow? (how I doing now).
Or did I misunderstand your point?

don’t track the whole head. track each feature. and don’t use those trackers, they’re too complex. use MOSSE. it’s dumb but very precise, as long as the object (which should be a tiny feature on the face) doesn’t change much.

MOSSE approaches optical flow. methods to calculate optical flow work like MOSSE, except they use simpler math and smaller regions, hence the result is noisier. MOSSE uses a larger area (for a single track/point of course) and more sophisticated math, for a more accurate result.

1 Like

Thanks again for the useful advice! I changed the algorithm. When MOSSE function tracks “corner points”, cursor moves much more smoothly. There was a slight issue with discrete movement as the object rectangles moved the same number of pixels at the same time. Cursor moved in leaps. So, I had to use filter on each tracked point. Anyway, as you can see in the video, the CPU load did not increase comparing to Lukas-Kanade optical flow algorithm + filtration only cursor position. In good light, the difference is also very noticeable.

https://www.youtube.com/watch?v=WKwuas0GVkA

  1. Lucas-Kanade optical flow:

goodFeaturesToTrack(),
cornerSubPix(),
calcOpticalFlowPyrLK(),
cursor EWMA filter

  1. MOSSE object tracking:

goodFeaturesToTrack(),
cornerSubPix(),
cv::legacy::TrackerMOSSE,
all points EWMA filtration

And of course I had to remember to include the tracking453.lib to Linker when add cv::legacy::Tracker. I spent half a day googling the “unresolved external symbol LNK2001 error”. For some reason including a tracker from the core library (cv::Tracker) does not result such compilation error, so it is confusing.