I am trying to pass an OpenGL texture to CUDA , right now I am doing it via glReadPixels to save it as a Python byte object, which needs to be converted to an image with PIL, then to an array using Numpy, before finally using it with OpenCV…
but I saw in the OpenCV docs that there is this:
cv::ogl::Buffer::mapDevice
Maps OpenGL buffer to CUDA device memory.
how do I access this function from Python?
unfortunately, it’s not wrapped into the python api
(and even if, you’d have to build from src with both opengl and cuda support)
(and even if, you’d have to build from src with both opengl and cuda support)
I am using Cudawarped’s build of OpenCV, so it is with CUDA enabled
is there a workaround for my problem? at the very least I want to avoid transferring to system memory, maybe there is some way to make a copy of the OpenGL data on the GPU and put it into CUDA that way?
If you can do this in python (maybe with OpenGL - pycuda 2024.1.2 documentation) then you can pass the raw pointer to the CUDA memory you have mapped from OpenGL using createGpuMatFromCudaMemory().
I am going to need a bit of hand-holding here with regards to OpenGL … I hope this isn’t straying too far from the purpose of this forum, but all I know is that part of the OpenGL code involves this:
glBindTexture(GL_TEXTURE_2D, TextureID)
the first argument is the type of texture , and the second argument is the name of the texture… Does the name have anything to do with the pointer?
no, it’s the id (an integer) of the currently activated opengl gpu texture slot
https://registry.khronos.org/OpenGL-Refpages/gl4/html/glBindTexture.xhtml
OpenGL is strange and different from most other APIs for anything.
instead of calling methods on some texture, instead you bind the texture (it is noted in OpenGL’s state that you can’t see directly), and then you call methods/functions on whatever texture is bound, without specifying it in the method/function.
That is my superficial understanding anyway.
How do I get the address of the GPU memory out of OpenGL so that I can pass it onto createGpuMatFromCudaMemory() ?
do I do this
glBindTexture(GL_TEXTURE_BUFFER, TextureID)
glGetBufferPointerv(GL_TEXTURE_BUFFER,GL_BUFFER_MAP_POINTER)
or something like that?
I think there’s interop from OpenGL to OpenCL, and from OpenCL to CUDA… so that may involve a step via OpenCL. there might be interop from OpenGL to CUDA though I am in no position to have a clue about that.
I think I have a plan, to use GL_PIXEL_PACK_BUFFER , like so:
import numpy
import ctypes
from OpenGL.GL import *
pbo = glGenBuffers(1) # create a pixel buffer object
openglframe =np.zeros((imageheight,imagewidth,4),np.uint8) # array with the resolution of the image I want to use, in 4 channels for RGBA, used for the byte size of glBufferData
glBindBuffer(GL_PIXEL_PACK_BUFFER, pbo)
glBufferData(GL_PIXEL_PACK_BUFFER,openglframe , GL_DYNAMIC_READ) # configures the pixel buffer
firstloop = True
tempcudamat = cv2.cuda.GpuMat(height,width,cv2.CV_8UC3) # temporary texture used for mapping the address from OpenGL as a CUDA GpuMat
outputcudamat = cv2.cuda.GpuMat(height,width,cv2.CV_8UC3) # the OpenGL texture should end up as this CUDA GpuMat
while True:
// There is a function which writes to the texture with TextureID, which runs in this line in my program
glActiveTexture(GL_TEXTURE0) # activate the OpenGL texture slot
glBindTexture(GL_TEXTURE_2D, TextureID) # selects the texture
if firstloop is True:
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, imagewidth,imageheight, 0, GL_RGBA, GL_UNSIGNED_BYTE, None) # configures the texture , runs only once
firstloop = False
glBindBuffer(GL_PIXEL_PACK_BUFFER, pbo) #selects the pixel buffer
glGetTexImage(GL_TEXTURE_2D, 0, GL_RGBA, GL_UNSIGNED_BYTE,openglframe) # gets the texture into the buffer
// After this is the Python part
mapdata = glMapBuffer(GL_PIXEL_PACK_BUFFER,GL_READ_ONLY) # returns a ctype pointer of the pixel buffer
pointer = ctypes.from_address(mapdata) # convert ctypes pointer to address
tempcudamat=cv2.cuda.createGpuMatFromCudaMemory(spoutreceiveheight,spoutreceivewidth,pointer) # give address to createGpuMatFromCudaMemory , and assign the memory to the temporary CUDA texture
tempcudamat.copyTo(outputcudamat) # copy the OpenGL texture from temporary CUDA GpuMat
// And finally, unmap the pixel buffer in OpenGl
glUnmapBuffer(GL_PIXEL_PACK_BUFFER)
glGetTexImage(GL_TEXTURE_2D, 0, GL_RGBA, GL_UNSIGNED_BYTE)
I am getting hung up on this part, no matter what I try I am getting an ‘invalid operation’ here
I have had some help and turns out this works:
glGetTexImage(GL_TEXTURE_2D, 0, GL_RGBA, GL_UNSIGNED_BYTE,0)
all I had to do was add a 0 in the fifth argument which is normally the array argument, because PyOpengl was automatically inserting an array or something in the argument and passing it to OpenGL
Now I have a new problem, and it is specifically to do with OpenCV:
openglcudamat = cv2.cuda.GpuMat(inputheight,inputwidth,cv2.CV_8UC4)
cudafirst = cv2.cuda.GpuMat(inputheight, inputwidth,cv2.CV_8UC4)
mapdata = glMapBuffer(GL_PIXEL_PACK_BUFFER,GL_READ_ONLY)
openglcudamat=cv2.cuda.createGpuMatFromCudaMemory(rows=height,cols=width,type=cv2.CV_8UC4,cudaMemoryAddress=mapdata)
The above lines of code run without errors, but when I try to do this:
openglcudamat.copyTo(cudafirst)
I get this error:
cv2.error: OpenCV(4.10.0) D:\repos\opencv\opencv-python\opencv\modules\core\src\cuda\gpu_mat.cu:270: error: (-217:Gpu API call) invalid argument in function ‘cv::cuda::GpuMat::copyTo’
First I don’t know anything about OpenGL however I know OpenCV fairly well so the below may or may not be of any help.
This error can be a result of getting the dimensions (rows, cols) or step wrong or passing a ptr to a memory address which the CUDA runtime is unaware of. I would double check your height and width first to be sure but I suspect the mapdata location is the issue.
Is mapdata an int containing the address of your cuda memory? e.g.
npMat_8UC4 = (np.random.random((128, 129, 4)) * 255).astype(np.uint8)
cuMat_8UC4 = cv.cuda_GpuMat(npMat_8UC4)
cuMat_8UC4_ptr_wrapped = cv.cuda.createGpuMatFromCudaMemory(cuMat_8UC4.size(),cuMat_8UC4.type(),cuMat_8UC4.cudaPtr(), cuMat_8UC4.step)
cuMat_8UC4_ptr_wrapped_cpy = cuMat_8UC4_ptr_wrapped.copyTo()
type(cuMat_8UC4.cudaPtr()), cuMat_8UC4.cudaPtr()
(int, 47398060032)
Another possibility is that OpenGL is using a different CUDA context to the one in OpenCV meaning the virtual memory spaces are different and the address from OpenGL is invalid in OpenCV.
e.g. If you replace cuMat_8UC4.cudaPtr() with an integer in the above
cv.cuda.createGpuMatFromCudaMemory(cuMat_8UC4.size(),cuMat_8UC4.type(),10, cuMat_8UC4.step)
you will generate the same error because 10 is an invalid address.
My goal is to avoid transferring the image data to system memory, as it is the biggest bottleneck in my Python script
It doesn’t have to involve remapping the memory from OpenGL to CUDA…If there was a way which involved copying/converting the data from OpenGL to CUDA , it should also avoid the problems you mentioned… But is it possible to do it entirely on the GPU?
Is
mapdataan int containing the address of your cuda memory?
mapdata = glMapBuffer(GL_PIXEL_PACK_BUFFER,GL_READ_ONLY) print(type(mapdata), mapdata)
the code above results in the following output for me:
<class ‘int’> 2617721748480
If you can pass me an MRE then I will try and take a look. In your current example TextureID is not defined.
What is your current issue with the code?
Sorry, my previous post had a mistake with the variable names of the GpuMats
I have reduced the code down to less than 50 lines of code, I hope this constitutes a minimum reproducible example
my issue with this code is the same error again
File "C:\Users\fdasfasdfasd\Documents\test.py, line 45, in <module>
openglcudamat.copyTo(outputcudamat) # copy the OpenGL texture from temporary CUDA GpuMat
cv2.error: OpenCV(4.10.0) D:\repos\opencv\opencv-python\opencv\modules\core\src\cuda\gpu_mat.cu:270: error: (-217:Gpu API call) invalid argument in function 'cv::cuda::GpuMat::copyTo'
import numpy as np
from OpenGL.GL import *
import cv2
import pygame
height = 480
width = 640
#pygame
#-----------------------
pygame.init()
pygame.display.set_caption('Texture Receiver Example')
pygame.display.set_mode((width, height),
pygame.OPENGL | pygame.DOUBLEBUF)
TextureID = glGenTextures(1)
glBindTexture(GL_TEXTURE_2D, TextureID)
#--------------------------------
pbo = glGenBuffers(1) # create a pixel buffer object
openglframe =np.random.randint(low=0,high=255,size=(height,width,4)) # array with the resolution of the image I want to use, in 4 channels for RGBA, used for the byte size of glBufferData
glBindBuffer(GL_PIXEL_PACK_BUFFER, pbo)
glBufferData(GL_PIXEL_PACK_BUFFER,openglframe , GL_DYNAMIC_READ) # configures the pixel buffer
openglcudamatcudamat = cv2.cuda.GpuMat(height,width,cv2.CV_8UC3) # temporary texture used for mapping the address from OpenGL as a CUDA GpuMat
outputcudamat = cv2.cuda.GpuMat(height,width,cv2.CV_8UC3) # the OpenGL texture should end up as this CUDA GpuMat
glActiveTexture(GL_TEXTURE0) # activate the OpenGL texture slot
glBindTexture(GL_TEXTURE_2D, TextureID) # selects the texture
glTexImage2D(GL_TEXTURE_2D, 0, 3, width, height, 0, GL_RGBA, GL_UNSIGNED_BYTE, openglframe) # Load the image into OpenGL
glBindBuffer(GL_PIXEL_PACK_BUFFER, pbo) #selects the pixel buffer
glGetTexImage(GL_TEXTURE_2D, 0, GL_RGBA, GL_UNSIGNED_BYTE,0)
# After this is the Python part
mapdata = glMapBuffer(GL_PIXEL_PACK_BUFFER,GL_READ_ONLY) # returns a ctype pointer of the pixel buffer
openglcudamat=cv2.cuda.createGpuMatFromCudaMemory(rows=height,cols=width,type=cv2.CV_8UC4,cudaMemoryAddress=mapdata)
openglcudamat.copyTo(outputcudamat) # copy the OpenGL texture from temporary CUDA GpuMat
# And finally, unmap the pixel buffer in OpenGl
glUnmapBuffer(GL_PIXEL_PACK_BUFFER)
outputimage = outputcudamat.download()
cv2.imshow("test",outputimage)
print (outputimage)
This error can be a result of getting the dimensions (rows, cols) or step wrong
What is a ‘step’ in CUDA ?
Also, I find that if I comment out
pygame.init()
pygame.display.set_caption('Texture Receiver Example')
pygame.display.set_mode((width, height),
pygame.OPENGL | pygame.DOUBLEBUF)
this is the result:
Traceback (most recent call last):
File "C:\Users\fdasfasdfasd\AppData\Local\Programs\Python\Python310\lib\site-packages\OpenGL\latebind.py", line 43, in __call__
return self._finalCall( *args, **named )
TypeError: 'NoneType' object is not callable
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\fdasfasdfasd\Documents\test.py", line 18, in <module>
TextureID = glGenTextures(1)
File "C:\Users\fdasfasdfasd\AppData\Local\Programs\Python\Python310\lib\site-packages\OpenGL\latebind.py", line 47, in __call__
return self._finalCall( *args, **named )
File "C:\Users\fdasfasdfasd\AppData\Local\Programs\Python\Python310\lib\site-packages\OpenGL\wrapper.py", line 678, in wrapperCall
raise err
File "C:\Users\fdasfasdfasd\AppData\Local\Programs\Python\Python310\lib\site-packages\OpenGL\wrapper.py", line 671, in wrapperCall
result = wrappedOperation( *cArguments )
File "C:\Users\fdasfasdfasd\AppData\Local\Programs\Python\Python310\lib\site-packages\OpenGL\platform\baseplatform.py", line 415, in __call__
return self( *args, **named )
File "C:\Users\fdasfasdfasd\AppData\Local\Programs\Python\Python310\lib\site-packages\OpenGL\error.py", line 230, in glCheckError
raise self._errorClass(
OpenGL.error.GLError: GLError(
err = 1282,
description = b'invalid operation',
baseOperation = glGenTextures,
pyArgs = (
1,
<object object at 0x000001A7A79EB0C0>,
),
cArgs = (1, array([0], dtype=uint32)),
cArguments = (1, array([0], dtype=uint32))
)
so it looks like PyGame is handling the OpenGL , could it be that it is a different OpenGL context from OpenCV?
This is a pointer to CPU memory. As I said before I have not idea about OpenGL but if you can get a pointer to the GPU memory in the same context everything should work.