Given a Direct3D Texture2D I would like to process it using OpenCV cv::GpuMat. I am currently building 4.7 with CUDA 11.7 and CUDNN 8.9.
I am having some difficulties getting around CUDA’s interop with Direct3D 11 hence I would like to take a look at cv::GpuMat, which, from what I’ve read, is for managing OpenCV-related data on the GPU (e.g. using CUDA).
Is there an established workflow for “converting” a Direct3D Texture2D, which I use for my framebuffer, to an OpenCV cv::GpuMat and vice versa? It is also important that operations take place on the GPU without copying data back and forth between CPU and GPU. If that is not possible, then my hope is that I can at least use pinned memory, which I have already experimented with for CUDA and Torch C++ (libtorch).
Yes, and I am struggling with it. I need to draw to the texture, convert it to a CUDA compatible resource (as you have noted, the cudaGraphicsResource is the most promising way), process the contents (in my case with Torch) in that same buffer and then render the result to the screen.
When working with Torch, I found out that torch::from_blob() (with the option to create the resulting tensor on the CUDA-enabled GPU) basically (or so I think) wraps the contents of the CUDA memory block in that tensor. I can access it with both raw CUDA operations as well as Torch tensor operations. My hope is that I can e.g. wrap the OpenCV cv::Mat to a cudaArray in a similar way. Since that didn’t work out (don’t know why, but I can’t seem to be able to alter the data in wrapped array) I decided to look into anything else that OpenCV might offer.