cv::cuda::VideoReader uses the FFMPEG backend so if you can’t use cv::VideoCapture(src, CAP_FFMPEG) then it won’t work with cv::cuda::VideoReader.
That said I did manage to get the FFMPEG backend to work with direct show
you may be able to follow a similar procedure with V4L2 devices on linux but I coudn’t guarantee it. An alternative would be to write your own parser to read the raw input from your stereo cams by inheriting from cv::cudacodec::RawVideoSource.