Dnn::net::forward aberrant outputs when using CUDA backend

Hello,

I’m a currently using a custom trained yolov5 to run inference in C++ on images.
Here is the git code i’m using: https://github.com/doleron/yolov5-opencv-cpp-python
The implementation is quite basic.

Versions : OPENCV 4.6, CUDA 11.6

While using DNN_BACKEND_CUDA and DNN_TARGET_CUDA to run inference on GPU i’m having detected box with aberrant coordinate (out of image bounding boxs) after dnn::net::forward.

This does not happens while using CPU

After debugging, this is not an initialization problem, I have tried declaring outputs filled with zeros but i still have this issue.

Does anyone came across this problem or has a hypothesis on what the problem is ?