# Opencv on tbb creating one more thread than expected **URL:** \ **Category:** C++\ **Tags:** multithreading\ **Created:** [October 13, 2024, 4:20pm UTC](https://forum.opencv.org/t/opencv-on-tbb-creating-one-more-thread-than-expected/18953 "2024-10-13T16:20:31Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![AdamWill](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.opencv.org/adamwill/32/11286_2.png) [@AdamWill](https://forum.opencv.org/u/AdamWill)\ **Post date:** [October 13, 2024, 4:20pm UTC](https://forum.opencv.org/t/opencv-on-tbb-creating-one-more-thread-than-expected/18953/1 "2024-10-13T16:20:31Z")
Hi, folks! I’m grappling with a tricky issue with opencv threading. I’m Fedora’s QA team lead, I maintain Fedora’s deployment of openQA, a screenshot match-based automated testing tool. openQA’s test runner backend, os-autoinst, uses OpenCV for low-level image stuff. It’s written in perl, but uses a small internal C++ library called “tinycv” to wrap OpenCV. There have been issues in the past with os-autoinst processes blowing up due to a signal handling issue. The parent perl process can wind up sending SIGTERM or SIGCHLD signals to the OpenCV threads, which they don’t know how to handle. So in os-autoinst we try to block these signals with a signal processing mask before spawning OpenCV threads, then unblock them again afterwards (so the parent process _can_ receive these signals, which we need later). The current way we implement this is to set a sigprocmask, use `cv::setNumThreads` to set a cap on the number of threads, and then pre-create all those threads using a `parallel_for_` loop, so they all have the sigprocmask set. Then we unset the sigprocmask. The code for pre-creating the threads is at [os-autoinst/ppmclibs/tinycv\_impl.cc at 6c67d2feff2af0c4d85e325eb2bee6dbdffa6562 · os-autoinst/os-autoinst · GitHub](https://github.com/os-autoinst/os-autoinst/blob/6c67d2feff2af0c4d85e325eb2bee6dbdffa6562/ppmclibs/tinycv_impl.cc#L351) ; it’s called right after the sigprocmask is set, and the sigprocmask is unset right after it’s called. In Fedora’s deployment, we use the system opencv, which is a build of 4.9.0. It is configured to use tbb and built against the system tbb, which is 2021.11.0. On that deployment, we are frequently hitting a case where this doesn’t quite work right. The thread pre-creation loop works and the threads it creates have the mask set, but then, later (it seems to be about 30 seconds later, in the case I examined anyway), after we unset the sigprocmask, one _more_ thread gets created by tbb. We assume this is an OpenCV thread, as we don’t think anything else is using tbb, but I can’t prove this yet as I have not managed to get the issue to reproduce one time when running under `strace` to trace the thread creation (which is another mystery, as it otherwise occurs fairly often). It doesn’t have the sigprocmask set, so it causes the crash if it receives a SIGTERM or SIGCHLD. We cannot figure out what’s going on here. I’ve been poking at it in [isotovideo frequently crashing in the signal handler stuff (especially on aarch64) · Issue #2549 · os-autoinst/os-autoinst · GitHub](https://github.com/os-autoinst/os-autoinst/issues/2549) , but not getting very far. When I started digging into exactly what _happens_ when we do `cv::setNumThreads` it seems to get pretty fuzzy; it seems like opencv and tbb both have caps, they may have separate hard and soft caps, and it’s not clear to me whether the count includes the parent process or not. os-autoinst sets the cap to whichever of `cv::getNumThreads()` or `cv::getNumberOfCPUs() - 1)` is _lower_. (This is explained as being “To avoid running into TBB’s soft limit which seems to be one thread less than the number of physical CPU threads (see TBB function `calc_workers_soft_limit`)”). On the box I’ve been testing it on, that number is 63, which is `cv::getNumberOfCPUs() - 1)`. One thing I found is that the pre-creation loop seems to create only _62_ new threads, because the first iteration of the `parallel_for_` loop appears to run on the parent. So I thought that was the issue. But if I have the loop attempt to create one _more_ thread, it seems to deadlock (the os-autoinst test suite times out, presumably because the process never completes startup). But since that’s the case, I’m baffled about how an extra thread seems to get created later. Does anyone have any idea about what’s going on here and how to fix it? Or, failing that, any alternative ideas for dealing with the signal handling issue which might sidestep this problem? There’s a lot more detail in the issue linked above. Thanks a lot! ---
**Author:** ![AdamWill](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.opencv.org/adamwill/32/11286_2.png) [@AdamWill](https://forum.opencv.org/u/AdamWill)\ **Post date:** [October 15, 2024, 11:50pm UTC](https://forum.opencv.org/t/opencv-on-tbb-creating-one-more-thread-than-expected/18953/2 "2024-10-15T23:50:31Z")
Small update on this today: I tried two things. I patched os-autoinst to arbitrarily cap the value it passes to `cv::setNumThreads` at 4, then watched what happened. New processes started out with 4 threads - the main thread, and three subthreads created by the `parallel_for_` loop, with a 14 sigprocmask - but rapidly acquired more subthreads, with 00 sigprocmasks. The highest count I’ve seen is 10. So it really seems like opencv/tbb are creating more threads than we are telling them to. I also tried patching opencv to set tbb’s global parallelism limit: ```auto diff --git a/modules/core/src/parallel.cpp b/modules/core/src/parallel.cpp index 5799d73599..841be3b3ac 100644 --- a/modules/core/src/parallel.cpp +++ b/modules/core/src/parallel.cpp @@ -108,6 +108,9 @@ #endif #include "tbb/tbb.h" #include "tbb/task.h" + #if TBB_INTERFACE_VERSION >= 8005 + #include "tbb/global_control.h" + #endif #if TBB_INTERFACE_VERSION >= 8000 #include "tbb/task_arena.h" #endif @@ -727,7 +730,9 @@ void setNumThreads( int threads_ ) } #ifdef HAVE_TBB - +#if TBB_INTERFACE_VERSION >= 8005 + tbb::global_control tbb_cont(tbb::global_control::max_allowed_parallelism, threads); +#endif #if TBB_INTERFACE_VERSION >= 8000 if(tbbArena.is_active()) tbbArena.terminate(); if(threads > 0) tbbArena.initialize(threads); ``` But it doesn’t seem like that helps at all either ☹ We still get more threads than the limit we set.