Its working by passing through CUDA EGL Interop
as seen in How to create opencv gpumat from nvstream? - #18 by DaneLLL - DeepStream SDK - NVIDIA Developer Forums