Hi,
I’m observing a behavior that is rather strange in my opinion (but maybe it is normal)
My program is basically a loop like :
while(true)
{
//read new input image
//do some processing (about 0.05 seconds)
cv::imshow("windows1", img1);
cv::imshow("windows2", img2);
cv::imshow("windows3", img3);
cv::waitKey(0);
}
The images are displayed for debug purposes, the “real” output is just some data.
When I press a key (I usually use enter) once, I do one more iteration of the loop, and my images update as expected.
When I want to go fast (for example because I’m interested in the image #300 of my dataset), I keep the key (enter) pressed for some time. This doesn’t work as I would expect:
multiple frames are processed (as expected), but the images aren’t updated until I release the key
when I release the key, the program keep processing new new images for some time (as if waitkey had buffered some key presses)
What I would expect/like would be that each imshow updates the image, and that waitkey detects a key if still pressed, but not once released (1 or 2 mor detecions would already be a big improvement)
Is the behaviour I’m observing the “normal” behaviour?
Is there a way to improve it (even if just solving one of the 2 issues, it would already be a big improvement)
OS : Ubuntu 22.04
Opencv version : 4.5.4, compiled with CUDA support
it doesnt. it just copies an image pointer.
to update a window, it needs sending messages to the os, which happens in the waitKey function
again, yes.
maybe rather change the ‘sleep time’ dynamically, instead of ‘holding keys down’, like:
int sleepTime = 1; // 'fast forward' mode by default
while(true)
{
//read new input image
//do some processing (about 0.05 seconds)
cv::imshow("windows1", img1);
cv::imshow("windows2", img2);
cv::imshow("windows3", img3);
int k = cv::waitKey(sleepTime);
if (k==' ') // press space (once !) to toggle between
sleepTime = ! sleepTime; // 0(wait forever) and 1 milli (fast)
}
Thanks. At least I know it’s not me doing soething wrong
There might still be some way to empty the buffer and discarding its content, but I suppose it will give bad results in other situations (for exemple if typing text)
There, I don’t get it : I call waitKey in each loop, so why don’t the image update once per loop?
Seems like a good solution to avoid the problems. Thanks a lot, I will give it a try.
the observed behavior is “known”. it’s undesirable but nobody has bothered to fix it.
this has nothing to do with BIOS. that part of the computer is not at all involved.
the behavior stems from any of OpenCV’s various GUI backends. it should be possible to update all windows regardless but nobody seems to be bothered enough to fix this.
it’s not the key presses that are buffered. the event loop simply doesn’t make it to the window update messages because it returns with the key event. I may be wrong there but something like this is probably the issue. this requires backend-specific debugging.
unless you want to dig around in the specific backend code (and you’re familiar with GTK, I suppose), there’s little you can do. see timeout suggestion in previous posts.
Then why would I keep processing new frames once the key is release (I got the impression that the longer I pressed the key, the longer I keep processing frames after releasing the key). So I suspect there is a buffer somewhere (maybe in GTK, it would make sense that GTK has an internal event queue).
I had a quick look at the implementation of waitkey for the GTK backend : it seems that both windows update and keypress end up handled through the GTK functions without beeing separated clearly in the opencv code. So it seems not easy to fix without an understanding of how GTK works. So i think I will just use berak’s work arround for now
oh, you do experience some continued action after releasing the key? curious. I have to confess, I haven’t used OpenCV’s highgui with GTK so far.
on the win32 backend, if you hold a key down, or call imshow() from a mouse handler while moving the mouse, the windows don’t seem to update but all the computation does happen. my brain isn’t working right today and I thought you described something like that.
Yes, I do experience some continued action after releasing the key.
I got the impression (but without solid data) that the duration of continued action after releasing increases (proportionnal?) with the duration during which the key was pressed. I think the continued action some times exceeded 1 minute (ie several hundreds, maybe even more than 1000 key presses detected once the key was already released)
A huge (or illimited) event buffer, with processing being at a slower rate than the rate at which events are added to the queue, might explain this behaviour
if waitKey() returns with a value (other than -1), process it, but then call waitKey() again, until it returns -1. then you know there’s no more key presses left in the queue.
while True:
key = cv.waitKey()
if key == -1: break
... # processing
for exactly those situations, I wish python had a “double break” or something like that, so you could break out of this inner loop, and whatever frame reading or drawing loop is around this. depending how your code is structured, you could have this as a method and return from it. or use a flag to check right after this loop so you can break the second time.
Thanks for the suggestion. I think it could even be improved by emptying the queue before processing the frame, so that a key pressed during processing can be taken into account.
Either process the first key :
while(true)
{
int key=cv::waitkey(0);
while(cv::waitkey(1) != -1)
{
//do nothing
}
//process frame (based on value of key)
}
or the last key :
while(true)
{
int key=cv::waitkey(0);
int new_key=key;
while(new_key != -1)
{
key=new_key;
new_key=cv::waitkey(1);
}
//process frame (based on value of key)
}
By the way : is it possible to do a waitkey with delay less than 1ms (or no delay at all)?
some time ago I introduced pollKey() but only on Windows it actually does what it says. on other implementations, it falls back to waitKey(1). anyone is free to dig into their backend of choice and improve things.
it’s an accident of history that 0 means infinity, instead of 0. that’s now an API and can never be fixed.
be aware that the code in your latest post will discard key events. don’t be surprised if key presses stop doing anything.
Thanks. It was indeed what I’ve seen in the gtk backend code.
I don’t think so : waitkey(0) is garanteed to return a key press if I’m not mistaken. And then, the variable key is only update with new_key if new_key is not -1, so key can never take value -1.
But I haven’t tested it for now, as I already implemented sucessfully Barack’s solution, and it works quite nicely (now it is a “feature” to swich between continuous mode and frame_by frame) : I think it is even better than what I initially wanted, as I can now just launch continuous mode and do something else while processing is going on
that first piece of code. that’s just discarding every other queued key event (until queue is empty) after you’ve taken the first one. the goal should be to handle every key event.
the second piece of code looks convoluted to me. you copy a value back and forth. the inner loop doesn’t say to process anything received in the loop. it only stops once -1 is returned. the last actual event manages to make it out alive, meaning all events before it are discarded.
I think now I understand our misunderstanding : your proposition was a way to address the issue of frames not refreshing while the key is pressed, while mine addresses the issue of many new key presses detected once the key is released (if it was pressed for a long time)
i am not familiar with opencv key events so dont know anything about its behaviour but if its a headache to work around the issue you can think about using SDL2 ant its key event feature ( once i have used both libraries in a simple project together)
@cendum : thanks for the suggestion.
For now, I’m quite happy with berack’s work around. But if it happens to become a limiting factor, it might be a solution (I used SDL quite some time back, while learning C, and as far as I remember the key event features where indeed easier to use).