I am not sure how to use vcpkg but when calling CMake directly you can just disable all the extra CUDA modules
-DBUILD_opencv_cudaarithm=OFF -DBUILD_opencv_cudabgsegm=OFF -DBUILD_opencv_cudafeatures2d=OFF -DBUILD_opencv_cudafilters=OFF -DBUILD_opencv_cudaimgproc=OFF -DBUILD_opencv_cudalegacy=OFF -DBUILD_opencv_cudaobjdetect=OFF -DBUILD_opencv_cudaoptflow=OFF -DBUILD_opencv_cudastereo=OFF -DBUILD_opencv_cudawarping=OFF -DBUILD_opencv_cudacodec=OFF
and only build for the compute capability of your device.
e.g. If you had an RTX 3070 it would be
-DCUDA_ARCH_BIN=8.6
This will reduce the size of the world module significantly.