Can OpenCV process DirectX Texture2D directly on GPU?

Given a Direct3D Texture2D I would like to process it using OpenCV cv::GpuMat. I am currently building 4.7 with CUDA 11.7 and CUDNN 8.9.

I am having some difficulties getting around CUDA’s interop with Direct3D 11 hence I would like to take a look at cv::GpuMat, which, from what I’ve read, is for managing OpenCV-related data on the GPU (e.g. using CUDA).

Is there an established workflow for “converting” a Direct3D Texture2D, which I use for my framebuffer, to an OpenCV cv::GpuMat and vice versa? It is also important that operations take place on the GPU without copying data back and forth between CPU and GPU. If that is not possible, then my hope is that I can at least use pinned memory, which I have already experimented with for CUDA and Torch C++ (libtorch).

I am allmost 100% sure there is no interop between Direct3D11 and GpuMat. I would think what you have already tried is the best thing to use

Yes, and I am struggling with it. I need to draw to the texture, convert it to a CUDA compatible resource (as you have noted, the cudaGraphicsResource is the most promising way), process the contents (in my case with Torch) in that same buffer and then render the result to the screen.

When working with Torch, I found out that torch::from_blob() (with the option to create the resulting tensor on the CUDA-enabled GPU) basically (or so I think) wraps the contents of the CUDA memory block in that tensor. I can access it with both raw CUDA operations as well as Torch tensor operations. My hope is that I can e.g. wrap the OpenCV cv::Mat to a cudaArray in a similar way. Since that didn’t work out (don’t know why, but I can’t seem to be able to alter the data in wrapped array) I decided to look into anything else that OpenCV might offer.

Also OpenCV: opencv2/core/directx.hpp File Reference is something I just found. I am not sure what happens underneath though.

that’s for ocl, not cuda ;(