Imshow crashing when from camera and not from video file

C++
, ,
1

I’m making the transition from OpenCV 3.4 to 4.5; C++; VS2017; Windows10

I can get frames from a video avi file and show them. No problem. But when I go to a separate thread to get frames from a camera the imshow crashes with cv_terminate_handler()

The code worked when I was in OCV 2.4 and it worked when I was in OCV 3.4 but it is crashing now.

To simplify my debugging I put all the code in one spot. Basically it is this

// iHeight and iWidth are taken from the size of the camera frame.
Mat mTmpImage(cvSize(iWidth, iHeight), CV_8UC3);
// VI is videoinput 3rd party library
bool bSuccess = VI.getPixels(iCamera1, chrCameraBuffer1, false, false);
mTmpImage = imread((const char*)chrCameraBuffer1);
namedWindow(“testWindow”, CV_WINDOW_KEEPRATIO);
cv::imshow(“testWindow”, mTmpImage);
// Never makes it to waitKey
// Crashes with cv_terminate_handler()
cv::waitKey(10);

Any ideas. Other than this is a separate thread from the successful imshow of the video file I can’t think of anything else that might be different. No code has changed from OCV 3.4 to 4.5

Any help would be greatly appreciated.

2

careful with multitheading !
all the gui code (imshow(), waitkey()) has to stay on the main thread

also please explain:

bool bSuccess = VI.getPixels(iCamera1, chrCameraBuffer1, false, false);
mTmpImage = imread((const char*)chrCameraBuffer1);

what is chrCameraBuffer1 ? what does the imread() do here ? check what it returns !

3

But, like I said, this all worked with OCV 2.4 and 3.4. My app depends on multi-threading. I’m sure there is something that changed that, once I find it, I can work around it.

The chrCameraBuffer1 is an unsigned char

unsigned char * chrCameraBuffer1;

Later today, when I get a chance, I will just to a loadImage into mTmpImage and see what happens. That might at least point to where the problem is. In the main thread I am doing a VideoCapture.read() so I guess that is at least one difference between what is happening here on this thread and the main frame.

Edit…the imread() of a jpg worked. So, the problem is in the chrCameraBuffer1. I’ll have to play with that and see which format works. Instead of the imread() I had also tried

mTmpImage.data = chrCameraBuffer1;

but that didn’t work either. The problem has to be in the chrCameraBuffer1. I’ll let you know what I find.

I swear that I originally had the line

mTmpImage.data = chrCameraBuffer1

and that didn’t work so that is why I changed it to imread() just to test that. I went to the docs for data and it said that it needed a pointer to a uchar and that is exactly what I had. So I put back the .data = line and now it works.

As I mentioned, I bunched up this code in one location to make debugging easier. I need now to go back to the original code to test again why it wasn’t working.

I should never trust myself at the end of a long day. Jeeeeze!

OK I am back to the original code that caused problems

cv::imshow(“cameraWindow”, mCameraImage);
cv::waitKey(10);|

If I put up some test code right before that

namedWindow(“testWindow”, CV_WINDOW_KEEPRATIO);
cv::imshow(“testWindow”, mCameraImage);
cv::waitKey(10);

that works with testWindow. So I am back to my original question someplace else. I can get a handle to “cameraWindow” with

HWND hwCameraWindow = (HWND)FindWindow(NULL, _T(“cameraWindow”));

so the window is there plus I see the blank window on my screen as I debug. I can even do a new

namedWindow(“cameraWindow”, CV_WINDOW_KEEPRATIO);

right before the imshow(“cameraWindow” but that does not help.

So I guess the question is. Are there ways to check if a named window is good to show in before the imshow? Obviously something is wrong with this named window at this point ( a point at when OCV 2.4 and 3.4 had no problem with) but I have no way that I know of to check or fix the problem.

Any thoughts?

Ed

4

imread takes a path

you try to use it as if you were looking for imdecode but you probably don’t need either. the problem here is simply that you misuse imread.

please explain precisely what data you have there. is it bitmap, or is it compressed? if compressed, what type? jpeg?

5

That imread() was only an attempt to try different things in order to try and track down where the problem was. The original code was

mTmpImage.data = chrCameraBuffer1;

and I have since moved it back to that and it is working there. Now I am back at the original location in the code where the problem was happening and it is still not working but the problem seems to be with the named window. The lines that are bombing are

cv::imshow(“cameraWindow”, mCameraImage);
cv::waitKey(10);

The program freezes on imshow and never makes it to waitKey. If immediately before this I put

namedWindow(“testWindow”, CV_WINDOW_KEEPRATIO);
cv::imshow(“testWindow”, mCameraImage);
cv::waitKey(10);

That will work. The same mCameraImage but a different named window. So the imshow and the mCameraImage are good it is a problem with the named cameraWindow. I know that the cameraWindow is there because I can get an HWND for that window and I can see it while debugging.

Out of curiosity I tried to destroyWindow and then recreate the named window with this.

destroyWindow(“cameraWindow”);
namedWindow(“cameraWindow”, CV_WINDOW_KEEPRATIO);

but the program will freeze with destroyWindow and never make it to the new namedWindow. There is something about that cameraWindow that I cannot imshow to it nor even destroy it.

Is there any function that you can think of that I can use to test this namedWindow that might give me a hint of what is happening.

The cameraWindow is created in a function called CreateCameraWindow with

namedWindow(“cameraWindow”, CV_WINDOW_KEEPRATIO);
resizeWindow(“cameraWindow”, iCameraImageWidth, iCameraImageHeight);
moveWindow(“cameraWindow”, 0, lClientTop);
// Get a pointer to the CWnd
pCamera = CWnd::FindWindow(NULL, _T(“cameraWindow”));
// Set the cameraWindow in TopMost position.
pCamera->SetWindowPos(&CWnd::wndTopMost, 0, 0, 0, 0, > SWP_NOMOVE | SWP_NOSIZE | SWP_SHOWWINDOW);
// Get a HWND to the cameraWindow
HWND hwCamera = (HWND)FindWindow(NULL, _T(“cameraWindow”));
// Set the cursor that the cameraWindow
DWORD dwresult = ::SetClassLong(hwCamera,
GCL_HCURSOR,
(LONG)LoadCursor(AfxGetInstanceHandle(),
MAKEINTRESOURCE(IDC_CWAIT2)));

After that, in effect, we grab a frame from the camera which eventually ends up as mCameraImage, do some things to it and then show it with an imshow into cameraWindow but somewhere along the line cameraWindow has gone rogue and cannot be shown on nor destroyed. As I have mentioned earlier, this code all worked in OCV 2.4 and 3.4. Whatever has caused cameraWindow to go rogue is happening because of OCV 4.5. But I don’t know how to determine what is wrong with the named window, I can’t even destroy it in order to create a new one.

Any ideas?

6

I’ve come to believe that the fact that the namedWindow cameraWindow was started outside the thread that is using it may be the problem. The confusing part is that this worked in OCV 2.4 and 3.4 and then all of a sudden stopped working with 4.5.

Later, when I get a change I will change the code to create the namedWindow inside the thread to test the theory. It is funny that Windows can see the cameraWindow but OCV refuses to work with it. I’ll find out latter. At least it is a direction to check which I was out of at this point.

Thanks.

Ed

7

The fact that a separate function CreateCameraWindow created the namedWindow cameraWindow and then started the thread in which the cameraWindow was used seems to be the problem.

I had not paid attention to your suggestion at first because this is the way it was working in OpenCV 2.4 and 3.4 so I really didn’t think that was going to be it. But when I saw that I could create a new namedWindow testWindow and imshow the camera frame but not be able to do that with the already existing cameraWindow I started to believe that this was the problem.

My question is, how did you know this? I don’t think that I have ever noticed this precaution and, like I said, it always worked in OCV 2.4 and recently when I converted to 3.4. Is this new to 4.x? Where is it mentioned?

Thanks for pointing this out.

Ed

8

you keep talking about mCameraImage but you haven’t shown what it is, where it comes from, etc… that might be a good idea

9

Basically I have one thread that reads the frames from a camera and does a little bit of processing on it. That would basically come from the code mentioned above like this:

bool bSuccess = VI.getPixels(iCamera1, chrCameraBuffer1, false, false);
mTmpImage = imread((const char*)chrCameraBuffer1);

After the processing, that mTmpImage becomes mTmpCombined. The original mTmpImage gets put into one Mat Queue and the partially processed mTmpCombined gets put into another Mat Queue for later use.

// Push Original BGR Image mat
ImageQueue.push(Mat(mTmpImage.size(), mTmpImage.type()));
mTmpImage.copyTo(ImageQueue.back());

While that thread is running reading the camera stream, another thread is running doing sampling of those frames with SimpleBlobDetector among many other things. This thread gets its frames from the Queues like this

if (ImageQueue.size() > 0){
ImageQueue.front().copyTo(mCameraImage);
ImageQueue.pop();
}

Obviously there is more to it than that but basically the mCameraImage comes from a camera stream via a Mat Queue

10

berak, I must apologize in that, like so many of my own users, I did not appreciate every word of your answer. You said “all the gui code (imshow(), waitkey()) has to stay on the main thread” It took me awhile to realize that the key word is “main”. I was beating my head against a wall putting all of the gui code mentioned into the secondary camera streaming thread and wondering why I was still having problems. When I finally realized that it is the “main” thread in which all of the code had to be I wrote a function called CheckCameraWindowStuff() that is called by the main thread to check for memory variables periodically. So, among others, it would check for the memvar bImShowCameraImage with

if (bImShowCameraImage == true) {
cv::imshow(“cameraWindow”, mCameraImage);
cv::waitKey(10);
bImShowCameraImage = false;
}

In the camera stream thread where it used to have the imshow it now just sets the memvar to be read by the main thread. This goes for all of the other gui functions dealing with this thread including setMouseCallback and destroyWindow. A separate function was made for cv::namedWindow to create the window when the time comes.

Now, with literally everything dealing with this camera stream, in the MAIN thread, everything is working fine in OCV 4.5

One other thing I had to do. The main thread called a Modal Dialog in which the user selected the camera and then started the camera which started the camera stream thread etc. From that dialog the user was also able to select the video and run that by programmatically clicking on the main threads “Select Video” menu item. Using OCV 2.4 and 3.4 that worked but with 4.5 the camera stream thread stopped. It took awhile to realize that, like the gui items mentioned above that “Select Video” menu item had to be clicked from within the main thread and the dialog button function to select a video had to return back to the modal dialog. Again I set a memvar to be read by the main thread and that solved the problem. Baby Steps… Thanks again for your help. Sorry I didn’t pay attention to every word the first time.