cv::videoCapture possible memory leak with file read/save

C++
,
1

Hi there

I might have stumbled upon a memory leak. Googling suggests others may have seen it too but I can’t find a solution.

Each time I run this example function I never get back all the memory that was allocated by the cv::VideoCapture and cv::VideoWriter objects.

void vidMerge()
{
    std::string fp1 ("path/to/movie1.avi");
    std::string fp2 ("path/to/movie2.avi");
    std::string fpO ("path/to/movieOutput.avi");

    cv::VideoCapture inputA (fp1);
    cv::VideoCapture inputB (fp2);

    int codec = cv::VideoWriter::fourcc('M', 'J', 'P', 'G');
    cv::Size engineSize (inputA.get(cv::CAP_PROP_FRAME_WIDTH), inputA.get(cv::CAP_PROP_FRAME_HEIGHT));
            
    cv::VideoWriter output (fpO, 
                    codec,
                    24,
                    engineSize);


    cv::Mat frameA;
    cv::Mat frameB;
    cv::Mat frameO;

    int totalFrames = std::min (inputA.get(cv::CAP_PROP_FRAME_COUNT), inputB.get(cv::CAP_PROP_FRAME_COUNT));

    for ( int f = 0; f < totalFrames; f++)
    {
        inputA.read(frameA);
        inputB.read(frameB);

        frameO = 0.5 * (frameA + frameB);
        output.write(frameO);
    }

    output.release();
    inputA.release();
    inputB.release();
    frameO.release();
}

This is with OpenCV4.5.1 on Windows 10 x64, using the default backend api (as shown in the test case above).

I have found this issue to be consistent between Debug and Release OpenCV builds.

I’ve been going through heap profiling and can see how the lost memory is the result of something I’ve done.

Please let me know if you have seen this or have any ideas.

Many thanks, Jeff

2

I just tested this against OpenCV 4.0.0 (75ed282). The same behaviour, so apparently this isn’t a new situation. But also, it makes me question if I’m doing something wrong, rather than it being a bug. Thoughts anyone?

3

regardless of the issue, be careful with this:

those are going to be CV_8U types and addition wraps around. see if this expression works: (frameA >> 1) + (frameB >> 1). that’s distributing the 0.5* into the parenthesis and then transforming it (floating point!) into a cheap integer operation.

regarding the issue:

what backend do you get? ffmpeg? gstreamer? nail it down using apiPreference argument to videocapture and videowriter constructors

how much memory is approximately lost per round? what resolution do your videos have? to be a true memory leak, and not just an allocator being lazy about releasing the memory back to the OS, consumption must grow arbitrarily with repeated object constructions+destructions.

1 Like
4

Thanks for the advice on the expression! I will investigate and try to learn from your suggestion.

With respect to the issue, sorry I should have clarified the backend - especially since I was wondering if that was a consideration. By using getBackendName() I found that the readers are: MSMF, and the writer is: CV_MJPG. I confirmed this by explicitly setting the backend with these:

int apiPrefReaders = cv::CAP_MSMF;
int apiPrefWriter = cv::CAP_OPENCV_MJPEG;

Memory usage
Round 1: 15MB-95MB
Round 2: 95MB-187MB
Round 3: 187MB-257MB
So roughly ~80MB each time.

The video is 1280x720, and as you probably saw, MJPG @24fps.

I did try specifying DSHOW, FFMPEG, both read and write to be MSMF, VFW. These permutations so far either lead to no output file (nor CPU usage), or expected CPU usage and a 0 byte file result.

Thanks again for your input. Looking forward your thoughts on this.

Cheers, Jeff

5

Case closed! Thanks very much indeed, @crackwitz!

For anyone who sees memory leak looking stuff with Windows 10 and OpenCV 4.0.0/ 4.51 / 4.52 when using videocapture, you may have a similar situation to me. That is to say, MSoft MediaFoundation as a backend for the reader might not be behaving as well as it should.

Instead, explicitly set your apiBackend for the reader:

cv::VideoCapture videoReader ("pathtoMovie.avi",  cv::CAP_OPENCV_MJPEG);

Not only do I see no leaked memory, but it’s actually a little faster.

On the topic of the expression:

I did try this but it’s not a syntax that I can get away with in c++. I tried a few things but I was unable to get a version of your suggestion to work. Would you mind explaining a bit more verbosely what a better alternative is to what I was doing please? Is your point that I’m combining a float operation on int matrices?

6

OpenCV has its own AVI and MJPEG routines. they are always present and they are a reliable fallback.

MSMF is somewhat “new”. if there’s a bug in there, it’s certainly worth submitting a bug report on github.

I would wonder which object leaks memory, the VideoCapture object or the VideoWriter object. that would be something to work out for a “minimal reproducible example”.

looks like “bit shift” isn’t one of those operations:

https://docs.opencv.org/master/d1/d10/classcv_1_1MatExpr.html#details

the idea is: if you have two pixels that are both 255 (uint8), and you add them, your result has to be uint8 again, but 510 doesn’t fit in that range. it’ll become 254. that’s how integers wrap around. then you *0.5 this and you’re left with 127.0… for two white pixels blended together.

halving both before the addition keeps the addition from overflowing. with a bit shift, or integer division by 2, you’d get 127 + 127 = 254 as the result. the exact mathematical result would have been 255.0, so this also has downsides.

or you convert both operands into CV_16U (CV_16UC3) before the calculation, which are 16 bit integers. no overflow there. or you convert both into CV_32F (CV_32FC3).

the choice depends on your desired result type, but also on resource usage. 32 bit floats use more memory than 8 or 16 bit integers. integer operations are faster to calculate than floating point operations.

7

This is an excellent explanation! Thanks very much for the comprehensive answer and sharing your thinking. I had thought that it would max out the 8bit value on overflow and imply a clamp on the values rather than wrapping around. I was focusing so much on the memory stuff I wasn’t even paying attention.

The bit shift option would have been cool, but all good - plenty of other options available.

Good point on reporting MSMF as a possible bug. I’ll carve out a more concise example and post it in github as suggested.

Once again many thanks!