Odd and difficult problem with cv::waitKey()

OpenCV 4.5.3; Visual Studio 2017 MFC; C++; Windows 10

Difficult to explain and too much code involved to put here but I will try to explain.

First of all this is a mult-thread app. In one mode, without the use of a camera, the main thread loops through a video file and “shows” the frames in the named window videoWindow.

In another mode, using a camera, the main thread calls a dialog box and from there the user selects the camera to use at which time the camera is initiated, a named window, cameraWindow, is created (actually a memory variable is set so that the main thread knows to create the cameraWindow) and then a thread is started and detached to stream the camera frames. Call this thread streamCamera.

Because there is a lot of processing that has to be done with the camera frames, before going into it’s loop, the streamCamera thread will start another thread and detach it (call this one the frameGrab thread) that will do the actual getting of the frames from the camera (3rd party camera software that will put the pixel data into a buffer which can then be read into Mat.data ) and then do some preliminary processing such as flip, thresholding, erode, dialate, etc and then put the original and thresholded frames in queues to be read by the streamCamera thread.

OK…so here is the fist question: The frameGrab thread that is detached has no highgui procedures in it whatsoever that I can see and yet it requires a waitKey(1) at the top of each loop or it won’t run. Why is a waitKey needed in a thread that has no highgui in it?

And here is the problem that started this post. If the user runs a video only in mode one with no camera, ie the cameraWindow is never created, the streamCamera and frameGrab threads are never started, etc. and then later tries to run in the camera mode, the waitKey(1) at the top of the frameGrab thread will fail and all processing the the frameGrab thread will cease and never get past the waitKey line…

I have recently ported over from OCV 2.4 to 4.5 and in that process made sure that all highgui (named windows, imshow, waitKey, setMouseCallback, etc) are all done in the main thread. So streamCamera at the most will set memory variables to have the main thread call a highgui procedure.

I tried to have the frameGrab thread also pass a memory variable to have the main thread call the waitKey but, at times when the video is getting set up, the main thread is not in a loop and therefor not able to catch the change in value of various memory variables, and inevitably I hit an area where, because of the lack of a waitKey the frameGrab thread will stop. Or, if I put a waitKey in frameGrab to account for the exceptions when the main thread is not in a loop, then that waitKey will crash and everything stops.

There has to be something that I am not taking into consideration. I have tried every possible workaround that I can think of and it still ends up crashing or just not functioning at some point.

Any ideas, no matter how out there, will be appreciated.

Ed

I doubt that some of these things actually happen as described.

this sounds like a software engineering issue in the application code.

without being allowed to see the code, I would recommend starting over from scratch, and building this up piece by piece. and don’t do it all at once and then ask for advice. get someone with experience in multithreading to either do this for you, or peek over your shoulder while you do it.

Real quick, to better clarify when this is happening: I can start the app and run in mode 1 without the camera all day long, no problem. I can also start the app and run in mode 2 with the camera all day long, no problem. The problem happens when ever I run in mode one and then try to run in mode 2.

That said. the fact that I even need a waitKey in the frameGrab stream is a mystery to me but the fact is, if I take it away the thread will not work. This has been part of the code for years. However, previously, we were using OCV 2.4 and there were imshows and waitKeys in the streamCamera thread. This new problem began when I switched from 2,4 to 4.5 and had to modify the code to keep all highgui functions in the main thread. So, as of now, the main thread creates the named windows and does all imshow, waitKey, etc.

But even then, the mode 2 with the camera and the 2 additional threads works fine. It is only when I run the mode 1 first without the camera that I get the crash in waitKey.

Following the code in debug mode the code will go into the waitKeyEx() call that is in waitKey to the line

lock(getwindowMutex)

and finally just stops on the line in VS2017 → VCtools → xthread →

_Mtx_lock(_Mtx_t, _Mtx)
return (_Check_C_return(_Mtx_lock(Mtx)

So it seems that something in the mode 1 without the camera as caused a lock that mode 2 cannot get past. Odd since I can run in mode 1 all day long. Would a lingering waitKey that had no code to process leave a lock in place?

Or is it that, since the named window cameraWindow is now created in the main thread that this lone waitKey (that apparently has to be there) is no longer able to lock onto the named window? But no, since I can run in mode 2 all day long. So that is not it. It has something to do with something that happened in mode 1.

I was just hoping, for one thing. that someone would know why I would need a waitKey in the frameGrab thread that has no highgui code or have a suggestion for an alternative.

Ed

For what it is worth, I figured out a work around.

Passing memory variables (memvars) from the streamCamera thread to the main thread to do any highgui stuff worked because, by definition, when streamCamera needed to imshow or whatever, the main thread was in a looping mode and could catch and process them… However, that was not working for the frameGrab thread since that was running constantly as long at the camera was on and, if the main thread happened to be out of it’s looping mode, the passed memvars would not have a means to be processed.

Basically what I had to do was to start a separate thread from the main thread at the time the camera was started so that it is looping all of the time the camera is on. I use that thread to catch and process the memvar from the frameGrab thread.

It works but I still can’t figure out why the frameGrab thread needs a waitKey to begin with. Oh well…all’s well that ends well.

Ed