Not really, I experianced a similar issue with previous versions of OpenCV so if it was me I would just confirm that I have matching versions of OpenCV from both the main and contrib repositories and if that didn’t work I would disable rgbd and to check if the build is successful without it.
Where did you install it? If you placed it in
/usr/local/cuda-11.8/lib
/usr/local/cuda-11.8/include
then it should have been picked up.