I have had a reply from the Khronos OpenGL forums, which says that glMapBuffer is a pointer to GPU memory, but in the CPU address space… does this sound correct?
Another reply mentioned CUDA OpenGL interop , but I am not sure if this is beyond the scope of OpenCV
It sounds correct but
The result from my understanding is that you don’t have a pointer to an address which CUDA can use, you have a pointer to an address you can access from the host which is what you are trying to avoid as there will be some overhead from using managed memory in this way.
If you were using C++ you could use OpenGL interop as described however to do the same in python I am pretty sure you need to use pycuda compiled for use with OpenGL as I mentioned in my original post.
I am having a lot of trouble trying to compile PyCuda with OpenGL functionality turned on, I did everything that I could find, like build Boost, add CUDA_ENABLE_GL = True in siteconf.py , etc etc etc but I cannot even import the PyCuda install I compiled into my Python file without it throwing one error message or another… In fact, I ran into the same error message as No module named 'pycuda.compyte' · inducer/pycuda · Discussion #339 · GitHub and the author’s solution is to install the version of PyCuda from PiPy … Which does not have OpenGL turned on…
Is there a reason why cv::ogl::Buffer::mapDevice cannot be implemented into OpenCV-Python?
Not that I know of however I suspect there will be a reason why this class has no python bindings.
Anyway its worth a try. To enable the bindings you need to build with
-DWITH_OPENGL=ON
and then wrap the Buffer class by changing CV_EXPORTS top CV_EXPORTS_W. i.e.
class CV_EXPORTS_W Buffer
and wrapping the methods you want to be available in python. e.g.
CV_WRAP cuda::GpuMat mapDevice();
CV_WRAP void unmapDevice();
Then make sure you follow the interop procedure:
This section describes OpenGL interoperability.
To enable OpenGL support, configure OpenCV using CMake with WITH_OPENGL=ON . Currently OpenGL is
supported only with WIN32, GTK and Qt backends on Windows and Linux (MacOS and Android are not
supported). For GTK-2.0 backend gtkglext-1.0 library is required.To use OpenGL functionality you should first create OpenGL context (window or frame buffer). You can
do this with namedWindow function or with other OpenGL toolkit (GLUT, for example).
I have no experiance with this so I can’t even test whether this still works in C++. All I can tell you is that the bindings will be generated with the above modifications whether they work or work from python is up to you to discover.
This function should be explicitly called after OpenGL context creation and before any CUDA calls.
@param device System index of a CUDA device starting with 0.
@ingroup core_opengl
*/
CV_EXPORTS void setGlDevice(int device = 0);
I found this in opengl.hpp , how do I expose this in Python?
Change to CV_EXPORTS_W
CV_EXPORTS_W void setGlDevice(int device = 0);
I have been reading this OpenCV: How OpenCV-Python Bindings Works?
and I think I understand the following:
Within the Buffer class of opengl.hpp ,
CV_WRAP is for extending the methods such as void release() and void create()
CV_PROP is used for extending the fields such as int rows() const; and int cols() const;
CV_WRAP_AS is for overloaded functions such as void copyTo(OutputArray arr) const; and void copyTo(OutputArray arr, cuda::Stream& stream) const;
but how do I extend the overloaded constructors:
Buffer();
/** @overload
@param arows Number of rows in a 2D array.
@param acols Number of columns in a 2D array.
@param atype Array type ( CV_8UC1, ..., CV_64FC4 ). See Mat for details.
@param abufId Buffer object name.
@param autoRelease Auto release mode (if true, release will be called in object's destructor).
*/
Buffer(int arows, int acols, int atype, unsigned int abufId, bool autoRelease = false);
/** @overload
@param asize 2D array size.
@param atype Array type ( CV_8UC1, ..., CV_64FC4 ). See Mat for details.
@param abufId Buffer object name.
@param autoRelease Auto release mode (if true, release will be called in object's destructor).
*/
Buffer(Size asize, int atype, unsigned int abufId, bool autoRelease = false);
/** @overload
@param arows Number of rows in a 2D array.
@param acols Number of columns in a 2D array.
@param atype Array type ( CV_8UC1, ..., CV_64FC4 ). See Mat for details.
@param target Buffer usage. See cv::ogl::Buffer::Target .
@param autoRelease Auto release mode (if true, release will be called in object's destructor).
*/
Buffer(int arows, int acols, int atype, Target target = ARRAY_BUFFER, bool autoRelease = false);
/** @overload
@param asize 2D array size.
@param atype Array type ( CV_8UC1, ..., CV_64FC4 ). See Mat for details.
@param target Buffer usage. See cv::ogl::Buffer::Target .
@param autoRelease Auto release mode (if true, release will be called in object's destructor).
*/
Buffer(Size asize, int atype, Target target = ARRAY_BUFFER, bool autoRelease = false);
/** @overload
@param arr Input array (host or device memory, it can be Mat , cuda::GpuMat or std::vector ).
@param target Buffer usage. See cv::ogl::Buffer::Target .
@param autoRelease Auto release mode (if true, release will be called in object's destructor).
*/
explicit Buffer(InputArray arr, Target target = ARRAY_BUFFER, bool autoRelease = false);
because according to this Is it possible to bind a OpenCV GpuMat as an OpenGL texture? - Stack Overflow , the first step is to create a cv::ogl::Buffer , and this is what I have attempted with
oglbuf = cv2.ogl.Buffer(height,width,cv2.CV_8UC4,pbo)
but when I call openglcudamat = oglbuf.mapDevice() , this happens:
cv2.error: Unknown C++ exception from OpenCV code
if line 104 of opengl.hpp reads
CV_WRAP Buffer();
the error instead becomes
SystemError: <class ‘cv2.ogl.Buffer’> returned NULL without setting an exception
trying to do this in opengl.hpp , line 113
CV_WRAP Buffer(int arows, int acols, int atype, unsigned int abufId, bool autoRelease = false);
in order to match my line of Python code in my previous post, it causes this error in Visual Studio
| Severity | Code | Description | Project | File | Line | Suppression State | Details |
|---|---|---|---|---|---|---|---|
| Error | C2146 | syntax error: missing ‘;’ before identifier ‘abufId’ | opencv_python3 | C:\opencv-4.10.0\build\modules\python_bindings_generator\pyopencv_generated_types_content.h | 88472 |
the relevant line of code in pyopencv_generated_types_content.h is
PyObject* pyobj_int abufId = NULL;
unsigned int abufId;
I don’t know too much about C++ programming, but at a cursory glance, comparing to the lines of code immediately preceeding it:
PyObject* pyobj_arows = NULL;
int arows=0;
PyObject* pyobj_acols = NULL;
int acols=0;
PyObject* pyobj_atype = NULL;
int atype=0;
it does not seem to follow the type - name - equals - value syntax
is this fixable, or am I making a mistake somewhere?
Try uint. e.g.
CV_WRAP Buffer(int arows, int acols, int atype, uint abufId, bool autoRelease = false);
Try
uint. e.g.CV_WRAP Buffer(int arows, int acols, int atype, uint abufId, bool autoRelease = false);
I just noticed that, when I do the build process in Visual Studio, early on there was this error message
6>Traceback (most recent call last):
6> File "C:\opencv-4.10.0\modules\python\src2\typing_stubs_generator.py", line 49, in wrapped_func
6> ret_type = func(*args, **kwargs)
6> File "C:\opencv-4.10.0\modules\python\src2\typing_stubs_generator.py", line 148, in _generate
6> generate_typing_stubs(self.cv_root, output_path)
6> File "C:\opencv-4.10.0\modules\python\src2\typing_stubs_generation\generation.py", line 90, in generate_typing_stubs
6> root.resolve_type_nodes()
6> File "C:\opencv-4.10.0\modules\python\src2\typing_stubs_generation\nodes\namespace_node.py", line 106, in resolve_type_nodes
6> raise TypeResolutionError(
6>typing_stubs_generation.nodes.type_node.TypeResolutionError: Failed to resolve "cv2" namespace against "None". Errors: ['Failed to resolve "cv2.ogl" namespace against "cv2". Errors: [\'Failed to resolve "cv2.ogl.Buffer" class against "cv2". Errors: [\\\'Failed to resolve "cv2.ogl.Buffer.__init__" function against "cv2". Errors: [0]: Failed to resolve "abufId" argument: Failed to resolve "uint" exposed as "uint"\\\']\']']
6>
and yet the build process continues and is declared a success in the end
PyObject* pyobj_int abufId = NULL;
My biggest question now is, is this even valid C++ code?
I’ve taken it upon myself to begin editing the pyopencv_generated_types_content.h file
There are several instances of these lines of code:
PyObject* pyobj_asize = NULL;
Size asize;
PyObject* pyobj_atype = NULL;
int atype=0;
PyObject* pyobj_int abufId = NULL;
unsigned int abufId;
PyObject* pyobj_autoRelease = NULL;
bool autoRelease=false;
const char* keywords[] = { "asize", "atype", "int abufId", "autoRelease", NULL };
if( PyArg_ParseTupleAndKeywords(py_args, kw, "OOO|O:Buffer", (char**)keywords, &pyobj_asize, &pyobj_atype, &pyobj_int abufId, &pyobj_autoRelease) &&
pyopencv_to_safe(pyobj_asize, asize, ArgInfo("asize", 0)) &&
pyopencv_to_safe(pyobj_atype, atype, ArgInfo("atype", 0)) &&
pyopencv_to_safe(pyobj_int abufId, int abufId, ArgInfo("int abufId", 0)) &&
pyopencv_to_safe(pyobj_autoRelease, autoRelease, ArgInfo("autoRelease", 0)) )
{
new (&(self->v)) Ptr<cv::ogl::Buffer>(); // init Ptr with placement new
if(self) ERRWRAP2(self->v.reset(new cv::ogl::Buffer(asize, atype, int abufId, autoRelease)));
return 0;
}
I have edited them so that int abufId becomes either int_abufId or abufId, like so:
PyObject* pyobj_asize = NULL;
Size asize;
PyObject* pyobj_atype = NULL;
int atype=0;
PyObject* pyobj_int_abufId = NULL;
unsigned int abufId;
PyObject* pyobj_autoRelease = NULL;
bool autoRelease=false;
const char* keywords[] = { "asize", "atype", "int abufId", "autoRelease", NULL };
if( PyArg_ParseTupleAndKeywords(py_args, kw, "OOO|O:Buffer", (char**)keywords, &pyobj_asize, &pyobj_atype, &pyobj_int_abufId, &pyobj_autoRelease) &&
pyopencv_to_safe(pyobj_asize, asize, ArgInfo("asize", 0)) &&
pyopencv_to_safe(pyobj_atype, atype, ArgInfo("atype", 0)) &&
pyopencv_to_safe(pyobj_int_abufId, abufId, ArgInfo("int abufId", 0)) &&
pyopencv_to_safe(pyobj_autoRelease, autoRelease, ArgInfo("autoRelease", 0)) )
{
new (&(self->v)) Ptr<cv::ogl::Buffer>(); // init Ptr with placement new
if(self) ERRWRAP2(self->v.reset(new cv::ogl::Buffer(asize, atype, abufId, autoRelease)));
return 0;
}
for this error
| Severity | Code | Description | Project | File | Line | Suppression State | Details |
|---|---|---|---|---|---|---|---|
| Error | C2065 | ‘Target’: undeclared identifier | opencv_python3 | C:\opencv-4.10.0\build\modules\python_bindings_generator\pyopencv_generated_types_content.h | 88530 |
caused by this line of code:
PyObject* pyobj_target = NULL;
Target target=ARRAY_BUFFER;
I got it to build by copying and pasting lines 84-90 of opengl.hpp (the enum Target definition) into the line above every instance of Target in pyopencv_generated_types_content.h , like so:
PyObject* pyobj_target = NULL;
enum Target
{
ARRAY_BUFFER = 0x8892, //!< The buffer will be used as a source for vertex data
ELEMENT_ARRAY_BUFFER = 0x8893, //!< The buffer will be used for indices (in glDrawElements, for example)
PIXEL_PACK_BUFFER = 0x88EB, //!< The buffer will be used for reading from OpenGL textures
PIXEL_UNPACK_BUFFER = 0x88EC //!< The buffer will be used for writing to OpenGL textures
};
Target target=ARRAY_BUFFER;
But now I am stuck with this error
| Severity | Code | Description | Project | File | Line | Suppression State | Details |
|---|---|---|---|---|---|---|---|
| Error | C2665 | ‘cv::ogl::Buffer::Buffer’: no overloaded function could convert all the argument types | opencv_python3 | C:\opencv-4.10.0\build\modules\python_bindings_generator\pyopencv_generated_types_content.h | 88618 |
if(self) ERRWRAP2(self->v.reset(new cv::ogl::Buffer(arr, target, autoRelease)));
This seems to be generated in response to line 145 of opengl.hpp:
CV_WRAP explicit Buffer(InputArray arr, Target target = ARRAY_BUFFER, bool autoRelease = false);
because of the ‘arr’ variable name
In pyopencv_generated_types_content.h , it was represented as
PyObject* pyobj_arr = NULL;
Mat arr;
It seems to have been automatically assigned a ‘Mat’ data type
in the ‘problem details’ in Visual Studio, it says that it was trying to match the argument list (cv::Mat , pyopencv_cv_ogl_ogl_Buffer_Buffer::Target, bool)
but line 145 in opengl.hpp has defined it as
(cv::InputArray, pyopencv_cv_ogl_ogl_Buffer_Buffer::Target, bool)
No but the bindings I get when using uint are
PyObject* pyobj_abufId = NULL;
uint abufId;
which are valid. To avoid the bindings error try using unsigned instead.
Its better to move this outside of the buffer class (cut and paste above class CV_EXPORTS_W Buffer) so the bindings generate this for you. You will also need to do a find and replace
FROM -> TO
Buffer::ARRAY_BUFFER -> ARRAY_BUFFER
Buffer::ELEMENT_ARRAY_BUFFER -> ELEMENT_ARRAY_BUFFER
Buffer::PIXEL_PACK_BUFFER -> PIXEL_PACK_BUFFER
Buffer::PIXEL_UNPACK_BUFFER -> PIXEL_UNPACK_BUFFER
Yes as you can see from your experiance so far.
Note: cudaGLSetGLDevice is depreciated
https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__OPENGL__DEPRECATED.html
So I am not sure if adding the bindings here is the right path to persue.
so this is an OpenCV problem, not simply a Python bindings problem?
I wonder in which CUDA version was cudaGLSetGLDevice depreciated in , I am using 12.1 as required by one of my Python libraries
Its depreciated but should still work so this will be your problem if you put a lot of effort into getting the bindings to work only to have the functionality removed in the next version of CUDA.
My scope of interest is in Python, so core functionality of OpenCV such as the WITH_OPENGL switch is not really something that should be left to me
Right now my concern is to get the Python bindings for cv::ogl::Buffer working so that my project can transfer images between OpenGL and CUDA without going through the PCIe twice
The situation for me right now is that if I leave out CV_WRAP for any of the constructors in the Buffer class inside opengl.hpp (lines 104-145) , calling cv2.ogl.Bufferfrom Python will result in an overload resolution error…So it seems that I must solve the binding at line 145
CV_WRAP explicit Buffer(InputArray arr, Target target = ARRAY_BUFFER, bool autoRelease = false);
I assume that cv::InputArray is supposed to be mapped to a Numpy array in Python? It has to be one of the most basic functions in OpenCV Python, but I don’t even know where to begin
| Severity | Code | Description | Project | File | Line | Suppression State | Details |
|---|---|---|---|---|---|---|---|
| Error | C2039 | ‘ELEMENT_ARRAY_BUFFER’: is not a member of ‘cv::ogl::Buffer’ | opencv_core | C:\opencv-4.10.0\modules\core\src\opengl.cpp | 1565 | ||
| Error | C2039 | ‘PIXEL_UNPACK_BUFFER’: is not a member of ‘cv::ogl::Buffer’ | opencv_core | C:\opencv-4.10.0\modules\core\src\opengl.cpp | 1009 | ||
| Error | C2039 | ‘PIXEL_UNPACK_BUFFER’: is not a member of ‘cv::ogl::Buffer’ | opencv_core | C:\opencv-4.10.0\modules\core\src\opengl.cpp | 1011 | ||
| Error | C2039 | ‘PIXEL_UNPACK_BUFFER’: is not a member of ‘cv::ogl::Buffer’ | opencv_core | C:\opencv-4.10.0\modules\core\src\opengl.cpp | 1021 | ||
| Error | C2039 | ‘PIXEL_UNPACK_BUFFER’: is not a member of ‘cv::ogl::Buffer’ | opencv_core | C:\opencv-4.10.0\modules\core\src\opengl.cpp | 1023 | ||
| Error | C2039 | ‘PIXEL_UNPACK_BUFFER’: is not a member of ‘cv::ogl::Buffer’ | opencv_core | C:\opencv-4.10.0\modules\core\src\opengl.cpp | 1025 | ||
| Error | C2039 | ‘PIXEL_UNPACK_BUFFER’: is not a member of ‘cv::ogl::Buffer’ | opencv_core | C:\opencv-4.10.0\modules\core\src\opengl.cpp | 1035 | ||
| Error | C2039 | ‘PIXEL_UNPACK_BUFFER’: is not a member of ‘cv::ogl::Buffer’ | opencv_core | C:\opencv-4.10.0\modules\core\src\opengl.cpp | 1058 | ||
| Error | C2039 | ‘PIXEL_UNPACK_BUFFER’: is not a member of ‘cv::ogl::Buffer’ | opencv_core | C:\opencv-4.10.0\modules\core\src\opengl.cpp | 1123 | ||
| Error | C2039 | ‘PIXEL_UNPACK_BUFFER’: is not a member of ‘cv::ogl::Buffer’ | opencv_core | C:\opencv-4.10.0\modules\core\src\opengl.cpp | 1125 | ||
| Error | C2039 | ‘PIXEL_UNPACK_BUFFER’: is not a member of ‘cv::ogl::Buffer’ | opencv_core | C:\opencv-4.10.0\modules\core\src\opengl.cpp | 1135 | ||
| Error | C2039 | ‘PIXEL_UNPACK_BUFFER’: is not a member of ‘cv::ogl::Buffer’ | opencv_core | C:\opencv-4.10.0\modules\core\src\opengl.cpp | 1137 | ||
| Error | C2039 | ‘PIXEL_UNPACK_BUFFER’: is not a member of ‘cv::ogl::Buffer’ | opencv_core | C:\opencv-4.10.0\modules\core\src\opengl.cpp | 1139 | ||
| Error | C2039 | ‘PIXEL_UNPACK_BUFFER’: is not a member of ‘cv::ogl::Buffer’ | opencv_core | C:\opencv-4.10.0\modules\core\src\opengl.cpp | 1149 | ||
| Error | C2039 | ‘PIXEL_PACK_BUFFER’: is not a member of ‘cv::ogl::Buffer’ | opencv_core | C:\opencv-4.10.0\modules\core\src\opengl.cpp | 1174 | ||
| Error | C2039 | ‘PIXEL_PACK_BUFFER’: is not a member of ‘cv::ogl::Buffer’ | opencv_core | C:\opencv-4.10.0\modules\core\src\opengl.cpp | 1175 | ||
| Error | C2039 | ‘PIXEL_PACK_BUFFER’: is not a member of ‘cv::ogl::Buffer’ | opencv_core | C:\opencv-4.10.0\modules\core\src\opengl.cpp | 1177 | ||
| Error | C2039 | ‘PIXEL_PACK_BUFFER’: is not a member of ‘cv::ogl::Buffer’ | opencv_core | C:\opencv-4.10.0\modules\core\src\opengl.cpp | 1186 | ||
| Error | C2039 | ‘PIXEL_PACK_BUFFER’: is not a member of ‘cv::ogl::Buffer’ | opencv_core | C:\opencv-4.10.0\modules\core\src\opengl.cpp | 1188 | ||
| Error | C2039 | ‘PIXEL_PACK_BUFFER’: is not a member of ‘cv::ogl::Buffer’ | opencv_core | C:\opencv-4.10.0\modules\core\src\opengl.cpp | 1190 | ||
| Error | C2039 | ‘PIXEL_PACK_BUFFER’: is not a member of ‘cv::ogl::Buffer’ | opencv_core | C:\opencv-4.10.0\modules\core\src\opengl.cpp | 1202 | ||
| Error | C2039 | ‘ARRAY_BUFFER’: is not a member of ‘cv::ogl::Buffer’ | opencv_core | C:\opencv-4.10.0\modules\core\src\opengl.cpp | 1348 | ||
| Error | C2039 | ‘ARRAY_BUFFER’: is not a member of ‘cv::ogl::Buffer’ | opencv_core | C:\opencv-4.10.0\modules\core\src\opengl.cpp | 1364 | ||
| Error | C2039 | ‘ARRAY_BUFFER’: is not a member of ‘cv::ogl::Buffer’ | opencv_core | C:\opencv-4.10.0\modules\core\src\opengl.cpp | 1380 | ||
| Error | C2039 | ‘ARRAY_BUFFER’: is not a member of ‘cv::ogl::Buffer’ | opencv_core | C:\opencv-4.10.0\modules\core\src\opengl.cpp | 1398 | ||
| Error | C2039 | ‘ARRAY_BUFFER’: is not a member of ‘cv::ogl::Buffer’ | opencv_core | C:\opencv-4.10.0\modules\core\src\opengl.cpp | 1404 | ||
| Error | C2039 | ‘ARRAY_BUFFER’: is not a member of ‘cv::ogl::Buffer’ | opencv_core | C:\opencv-4.10.0\modules\core\src\opengl.cpp | 1459 | ||
| Error | C2039 | ‘ELEMENT_ARRAY_BUFFER’: is not a member of ‘cv::ogl::Buffer’ | opencv_core | C:\opencv-4.10.0\modules\core\src\opengl.cpp | 1538 | ||
| Error | C2039 | ‘ELEMENT_ARRAY_BUFFER’: is not a member of ‘cv::ogl::Buffer’ | opencv_core | C:\opencv-4.10.0\modules\core\src\opengl.cpp | 1542 |
I understand this but you need to be aware that this may not work. Even if we generate the bindings they may not function as expected because you are calling functions which rely on CUDA functions depreciated over 14 years ago and may not have been user tested in OpenCV since then, if at all.
See my previous reply
Where do I find these instances of Buffer:: ? They are not in opengl.hpp