# OpticalFlowDual\_TVL1\_Impl::calc keep allocates memory

**URL:** <https://forum.opencv.org/t/opticalflowdual-tvl1-impl-calc-keep-allocates-memory/20244>\
**Category:** C++\
**Tags:** cuda, imgproc, optflow\
**Created:** [February 17, 2025, 6:56am UTC](https://forum.opencv.org/t/opticalflowdual-tvl1-impl-calc-keep-allocates-memory/20244 "2025-02-17T06:56:22Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![n37jan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.opencv.org/n37jan/32/12039_2.png) [@n37jan](https://forum.opencv.org/u/n37jan)\
**Post date:** [February 17, 2025, 6:56am UTC](https://forum.opencv.org/t/opticalflowdual-tvl1-impl-calc-keep-allocates-memory/20244/1 "2025-02-17T06:56:22Z")

</div>

Hi

I’m building an application that uses OpenCV’s OpticalFlowDual\_TVL1\_Impl. It is basically gets sequence of image frames, and do optical flow motion vector estimation. While I was doing performance profiling using Nsight System, I found that there’s memory allocation and freeing everytime I’m calling OpticalFlowDual\_TVL1\_Impl::calc.

I found where the memory allocation happens in the source code:

```auto
    void OpticalFlowDual_TVL1_Impl::calc(InputArray _frame0, InputArray _frame1, InputOutputArray _flow, Stream& stream)
    {
        const GpuMat frame0 = _frame0.getGpuMat();
        const GpuMat frame1 = _frame1.getGpuMat();

        BufferPool pool(stream);
        **GpuMat flowx = pool.getBuffer(frame0.size(), CV_32FC1);**
        **GpuMat flowy = pool.getBuffer(frame0.size(), CV_32FC1);**

        calcImpl(frame0, frame1, flowx, flowy, stream);

        GpuMat flows[] = {flowx, flowy};
        cuda::merge(flows, 2, _flow, stream);
    }

```

The BufferPool’s getBuffer method is keep doing memory allocation, and deallocates it automatically when it is out of scope every time the ‘calc’ is called.

I’ve looked up the BufferPool, and it is basically memory pool that tries to reduce actual memory allocation/deallocation calls.

So, I have followed the example on here: [OpenCV: cv::cuda::BufferPool Class Reference](https://docs.opencv.org/3.4/d5/d08/classcv_1_1cuda_1_1BufferPool.html#a715fecc85a804935c8c5a6c4eb4f905e)

```auto
...
setBufferPoolUsage(true);
setBufferPoolConfig(getDevice(), 1024 * 1024 * 64, 2);

Stream stream1;

...
cv::Ptr<cv::cuda::OpticalFlowDual_TVL1> motionEstimatorCUDA;
motionEstimatorCUDA = cv::cuda::OpticalFlowDual_TVL1::create(tau, lambda, theta, nscales, warps, epsilon, iterations, scaleStep, gamma, useInitialFlow);
...
...
motionEstimator->calc(mat1, mat2, outputFlow);

```

But it is still keep allocating/deallocating memories whenever pool.getBuffer is called.

What am I missing?

---

<div class="post-metadata">

**Author:** ![cudawarped](https://avatars.discourse-cdn.com/v4/letter/c/9dc877/32.png) [@cudawarped](https://forum.opencv.org/u/cudawarped)\
**Post date:** [February 17, 2025, 12:30pm UTC](https://forum.opencv.org/t/opticalflowdual-tvl1-impl-calc-keep-allocates-memory/20244/2 "2025-02-17T12:30:02Z")

</div>

> [@n37jan](#):
>
> But it is still keep allocating/deallocating memories whenever pool.getBuffer is called.
> 
> What am I missing?

In Nsight System how are you determining that memory allocation is being performed, calls to `cudaMallocPitch`? If so how do you know these are related to flowx[y] and not from somewhere else?

Internally there are a lot of allocations performed

> <https://github.com/opencv/opencv_contrib/blob/ce3c6681c9bf0e5cf46704a6ce0883078bdba074/modules/cudaoptflow/src/tvl1flow.cpp#L216>

although looking at the code these should only happen the first time `OpticalFlowDual_TVL1_Impl::calcImpl()` is called.

I’ve just run a quick test below to verify `BufferPool` is working correctly and I cannot see any issues with that.

```auto
constexpr int w = 1024*10, h = w;
constexpr int n = 5;
GpuMat frame0(h, w, CV_8UC1);
GpuMat frame1(h, w, CV_8UC1);
DeviceInfo info;

{
    const size_t memFreeStart = info.freeMemory();
    for (int i = 0; i < 5; i++) {
        const size_t memFreeBeforeAllocation = info.freeMemory();
        GpuMat flowx(frame0.size(), frame0.type());
        GpuMat flowy(frame1.size(), frame1.type());
        const size_t memFreeAfterAllocation = info.freeMemory();
        std::cout << "Memory allocated in loop: " << (memFreeBeforeAllocation - memFreeAfterAllocation) / (1024.0 * 1024.0) << "MB" << std::endl;
    }
    const size_t memFreeEnd = info.freeMemory();
    std::cout << "Memory allocated after loop: " << (memFreeStart - memFreeEnd) / (1024.0 * 1024.0) << "MB" << std::endl;
}

{
    setBufferPoolUsage(true);
    setBufferPoolConfig(getDevice(), h * w * 2, 1);
    Stream stream = Stream();
    BufferPool pool(stream);
    const size_t memFreeStart = info.freeMemory();
    for (int i = 0; i < 5; i++) {
        const size_t memFreeBeforeAllocation = info.freeMemory();
        GpuMat flowx = pool.getBuffer(frame0.size(), frame0.type());
        GpuMat flowy = pool.getBuffer(frame1.size(), frame1.type());
        const size_t memFreeAfterAllocation = info.freeMemory();
        std::cout << "Memory allocated in loop: " << (memFreeBeforeAllocation - memFreeAfterAllocation) / (1024.0 * 1024.0) << "MB" << std::endl;
    }
    const size_t memFreeEnd = info.freeMemory();
    std::cout << "Memory allocated after loop: " << (memFreeStart - memFreeEnd) / (1024.0 * 1024.0) << "MB" << std::endl;
}

```

---

<div class="post-metadata">

**Author:** ![n37jan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.opencv.org/n37jan/32/12039_2.png) [@n37jan](https://forum.opencv.org/u/n37jan)\
**Post date:** [February 17, 2025, 3:58pm UTC](https://forum.opencv.org/t/opticalflowdual-tvl1-impl-calc-keep-allocates-memory/20244/3 "2025-02-17T15:58:40Z")

</div>

Hi. Thanks for your reply.

> [@cudawarped](#):
>
> In Nsight System how are you determining that memory allocation is being performed, calls to `cudaMallocPitch`? If so how do you know these are related to flowx[y] and not from somewhere else?
> 
> Internally there are a lot of allocations performed

Yes, I found that ‘cudaMallocPitch’ is called twice, and other memories are fine, they’re allocated just once.

I confirmed it with debug mode build, so I could step into the source code, so the ‘getBuffer’ is calling the ‘cudaMallocPitch’ every time.

The ‘getBuffer’ internally calls ‘gpuMat::create’ and it is always create new array because the ‘row’ and ‘col’ is always set to zeros

 ![image](https://us1.discourse-cdn.com/flex020/uploads/opencv/original/2X/c/cf08fb86c11df8e9b3efcd6196bf648967549657.png)

On your code

> [@cudawarped](#):
>
> ```auto
> setBufferPoolUsage(true);
> setBufferPoolConfig(getDevice(), h * w * 2, 1);
> Stream stream = Stream();
> BufferPool pool(stream);
> const size_t memFreeStart = info.freeMemory();
> for (int i = 0; i < 5; i++) {
> const size_t memFreeBeforeAllocation = info.freeMemory();
> GpuMat flowx = pool.getBuffer(frame0.size(), frame0.type());
> GpuMat flowy = pool.getBuffer(frame1.size(), frame1.type());
> const size_t memFreeAfterAllocation = info.freeMemory();
> std::cout << "Memory allocated in loop: " << (memFreeBeforeAllocation - memFreeAfterAllocation) / (1024.0 * 1024.0) << "MB" << std::endl;
> }
> 
> ```

the ‘BufferPool pool(stream)’ is defined before the for-loop, but ‘OpticalFlowDual\_TVL1\_Impl’ is creating its pool everytime the ‘calc’ is called. Maybe that is why?

---

<div class="post-metadata">

**Author:** ![cudawarped](https://avatars.discourse-cdn.com/v4/letter/c/9dc877/32.png) [@cudawarped](https://forum.opencv.org/u/cudawarped)\
**Post date:** [February 17, 2025, 4:29pm UTC](https://forum.opencv.org/t/opticalflowdual-tvl1-impl-calc-keep-allocates-memory/20244/4 "2025-02-17T16:29:10Z")

</div>

> [@n37jan](#):
>
> The ‘getBuffer’ internally calls ‘gpuMat::create’ and it is always create new array because the ‘row’ and ‘col’ is always set to zeros

I would check this again, if `rows` and `cols` are zero then `rows !=_rows` as `_rows,_cols` are the size (frame0.size()) passed into `getBuffer`

```auto
pool.getBuffer(frame0.size(), CV_32FC1);

```

You want to check the status of `allocSuccess`

> <https://github.com/opencv/opencv/blob/21402668a17f3b0178fc6fa031136f2454472709/modules/core/src/cuda/gpu_mat.cu#L183>

but make sure it is called by

```auto
        GpuMat flowx = pool.getBuffer(frame0.size(), CV_32FC1);
        GpuMat flowy = pool.getBuffer(frame0.size(), CV_32FC1);

```

and not the other create methods inside the `OpticalFlowDual_TVL1_Impl::calcImpl()` function. If its false then step into the `allocate` method to see what the issue is.

> [@n37jan](#):
>
> the ‘BufferPool pool(stream)’ is defined before the for-loop, but ‘OpticalFlowDual\_TVL1\_Impl’ is creating its pool everytime the ‘calc’ is called. Maybe that is why?

I wondered about that but in my test code moving the `BufferPool` creation inside the loop still worked as expected.

---

<div class="post-metadata">

**Author:** ![n37jan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.opencv.org/n37jan/32/12039_2.png) [@n37jan](https://forum.opencv.org/u/n37jan)\
**Post date:** [February 17, 2025, 7:21pm UTC](https://forum.opencv.org/t/opticalflowdual-tvl1-impl-calc-keep-allocates-memory/20244/5 "2025-02-17T19:21:53Z")

</div>

Hi, thanks for your reply and advice.

After some work on my code per your advice, now the allocation/deallocation of Bufferpool is working as expected. However, now I’m having different problem.

I should reveal more on my code to get advice. Here’s my code:

```auto
class OpenCVOptFlow
{
public:
    OpenCVOptFlow(cudaStream_t cudaStream)
    {
        // Optical Flow Parameters
        const double tau = 0.25;
        double lambda = 0.1;
        const double theta = 0.3;
        int nscales = 2;
        const int warps = 4;
        const double epsilon = 0.01;
        int iterations = 2;
        const double scaleStep = 0.5;
        const double gamma = 0;
        const int medianFiltering = 0;
        const bool useInitialFlow = false;

        cv::cuda::setBufferPoolUsage(true);
        cv::cuda::setBufferPoolConfig(cv::cuda::getDevice(), 1024 * 1024, 10);
        myOpenCVCUDAStream = cv::cuda::StreamAccessor::wrapStream(cudaStream);
        myMotionEstimatorCUDA = cv::cuda::OpticalFlowDual_TVL1::create(tau, lambda, theta, nscales, warps, epsilon, iterations, scaleStep, gamma, useInitialFlow);
    }

    void run(unsigned char* I0, 
        unsigned char* I1,        
        int numRows,
        int numCols,
        int srcPitch,
        float* dstFlow,
        int dstPitch)
    {
        // wrapping native GPU pointers with cv::cuda::GpuMat
        cv::cuda::GpuMat srcMat = cv::cuda::GpuMat(numRows, numCols, CV_8U, I0, srcPitch);
        cv::cuda::GpuMat prevMat = cv::cuda::GpuMat(numRows, numCols, CV_8U, I1, srcPitch);
        cv::cuda::GpuMat dstFlow = cv::cuda::GpuMat(numRows, numCols, CV_32FC2, dstFlow, dstPitch);

        // run Optical flow
        myMotionEstimatorCUDA->calc(srcMat, prevMat, dstFlow, myOpenCVCUDAStream); // running optical flow
        ...
        ...
    }

private:
    cv::cuda::Stream myOpenCVCUDAStream;
    cv::Ptr<cv::cuda::OpticalFlowDual_TVL1> myMotionEstimatorCUDA;
}

```

With this code, the memory allocation inside the OpticalFlowDual\_TVL1::calc is just happening one time. However, I’m getting error on this part:

```auto
void OpticalFlowDual_TVL1_Impl::calcImpl(const GpuMat& I0, const GpuMat& I1, GpuMat& flowx, GpuMat& flowy, Stream& stream)
{
...
...
        if (!useInitialFlow_)
        {
            u1s[nscales_-1].setTo(Scalar::all(0), stream); <- cudaErrorInvalidValue
            u2s[nscales_-1].setTo(Scalar::all(0), stream);
        }
}

```

The GpuMat::setTo function calls cudaMemcpy2DAsync or cudaMemcpy2D depending on the stream like this, and I’m getting ‘cudaErrorInvalidValue’:

```auto
GpuMat& cv::cuda::GpuMat::setTo(Scalar value, Stream& stream)
{
    CV_DbgAssert( !empty() );
    CV_DbgAssert( depth() <= CV_64F && channels() <= 4 );

    if (value[0] == 0.0 && value[1] == 0.0 && value[2] == 0.0 && value[3] == 0.0)
    {
        // Zero fill

        if (stream)
            CV_CUDEV_SAFE_CALL( cudaMemset2DAsync(data, step, 0, cols * elemSize(), rows, StreamAccessor::getStream(stream)) ); <- cudaErrorInvalidValue
        else
            CV_CUDEV_SAFE_CALL( cudaMemset2D(data, step, 0, cols * elemSize(), rows) ); <- cudaErrorInvalidValue

        return *this;
    }

....

```

I tried to give cv::cuda::stream::Null() on ‘calc’ but the error is the same. What am I doing wrong?

By the way, the OpenCV version is 4.8, and 64bit compiled with VS2022 on Win10.

Thank you.

---

<div class="post-metadata">

**Author:** ![cudawarped](https://avatars.discourse-cdn.com/v4/letter/c/9dc877/32.png) [@cudawarped](https://forum.opencv.org/u/cudawarped)\
**Post date:** [February 18, 2025, 6:19am UTC](https://forum.opencv.org/t/opticalflowdual-tvl1-impl-calc-keep-allocates-memory/20244/6 "2025-02-18T06:19:56Z")

</div>

There could be multiple causes, I need to see a complete [MRE](https://stackoverflow.com/help/minimal-reproducible-example) if you want me to look into why its failing.

Its not a good idea to have the wrapped stream as a member of your class. Are you sure that the cuda stream your wrapping inside your class is not destroyed before the call to `myMotionEstimatorCUDA->calc`? It would be better to create the stream internally or pass it to your `run` method.

---

<div class="post-metadata">

**Author:** ![n37jan](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.opencv.org/n37jan/32/12039_2.png) [@n37jan](https://forum.opencv.org/u/n37jan)\
**Post date:** [February 20, 2025, 6:17am UTC](https://forum.opencv.org/t/opticalflowdual-tvl1-impl-calc-keep-allocates-memory/20244/7 "2025-02-20T06:17:35Z")

</div>

I’ve checked it with Nsight, and it turns out that the memory allocated by the Bufferpool was in different device. I’m currently using 2 graphic cards, and the pool’s memory allocated in device 1 while the device 0 was selected. Very strange.

I’m quite sure that cuda stream is alive during the myMotionEstimatorCUDA-\>calc call.

Anyway, thank you for your support.

---

<div class="post-metadata">

**Author:** ![cudawarped](https://avatars.discourse-cdn.com/v4/letter/c/9dc877/32.png) [@cudawarped](https://forum.opencv.org/u/cudawarped)\
**Post date:** [February 20, 2025, 6:36am UTC](https://forum.opencv.org/t/opticalflowdual-tvl1-impl-calc-keep-allocates-memory/20244/8 "2025-02-20T06:36:32Z")

</div>

> [@n37jan](#):
>
> I’ve checked it with Nsight, and it turns out that the memory allocated by the Bufferpool was in different device. I’m currently using 2 graphic cards, and the pool’s memory allocated in device 1 while the device 0 was selected. Very strange.

Make sure the device you are manually creating streams and other CUDA objects on is the same as the one OpenCV is using. The memory allocated in the pool should be on the device returned by the call to `getDevice()` so I would interrogate this.

The first thing I would do is explicitely set the device (call `cudaSetDevice`) before calling any other CUDA code to the device you want to use and checking that everything is allocated on the device you selected.