How to deal with large images

Python
,
1

Hi, for example, I have a very large image that weighs 3 GB opened on RAM

I want to do some operations that create a copy of the image like a color conversion and threshold

So I have this…

import numpy as np
import cv2

# Load a color image in BGR
OriginalImage = cv2.imread('LargeImage.png')    #  3GB Size

# Convert BGR to HSV
HSVCopy = cv2.cvtColor(OriginalImage , cv2.COLOR_BGR2HSV)  #  + 3GB Size

#Image Threshold
ret,TRCopy= cv2.threshold( cv2.split(HSVCopy )[1],127,255,cv2.THRESH_BINARY) #+ 1 GB to RAM

As you see I will be using like +7 GB of RAM to make all these operations, can I pass Original Image to all functions to save RAM space?

import numpy as np
import cv2

# Load a color image in BGR
OriginalImage = cv2.imread('LargeImage.png')    #  3GB Size

# Convert BGR to HSV
OriginalImage = cv2.cvtColor(OriginalImage , cv2.COLOR_BGR2HSV) 

#Image Threshold
ret,OriginalImage = cv2.threshold( cv2.split(OriginalImage )[1],127,255,cv2.THRESH_BINARY)

When I do this sometimes it doesn’t work with filters or other functions.

2

what you want to do is called “in-place operation”.

it doesn’t work like that (simply assigning to the same variable).

you have to pass the same array as the dst argument, if the function has that.

consult the docs for specific signatures.

some functions do not support in-place operation. docs will sometimes say if in-place operation is supported or not. the rest of the time, you just have to try it and hope that it always works.

example for cvtColor (OpenCV: Color Space Conversions):

# foo is some array
cv.cvtColor(src=foo, code=cv.COLOR_BGR2HSV, dst=foo)
# values in foo have been changed
2 Likes
3

In opencv by default number of pixels must be less than 2^30. Limit can be set using system variable OPENCV_IO_MAX_IMAGE_PIXELS

1 Like
4

Hi Laurent,
I have the exact same problem. I am trying to write / read a big size (10 GB) image (100,000 x 100,000, single channel gray) but am unable to do so. In fact any image over 2.139 GB can’t be written and anything above 1.06 GB can’t be read. I changed environment variable OPENCV_IO_MAX_IMAGE_PIXELS to 10,000,000,000. But there was no change in the largest image that I could read (2,13 GB). I have searched the net but the suggested solutions all point to changing value of OPENCV_IO_MAX_IMAGE_PIXELS. I am using OpenCV V4.5 on Visual Studio 2015, WIN 10, 12GB RAM, Can someone please help out?

5

if those settings have no effect, you’ve found an actual limit of the underlying image file format library (whatever it is) or of OpenCV’s source code.

what file format do you deal with?

6

Hi,

Can you post error message?
Is it :

cv2.error: OpenCV(4.5.2-dev) G:\Lib\opencv\modules\imgcodecs\src\loadsave.cpp:77: error: (-215:Assertion failed) pixels <= CV_IO_MAX_IMAGE_PIXELS in function ‘cv::validateInputImageSize’

or

cv2.error: OpenCV(4.5.2-dev) G:\Lib\opencv\modules\core\src\alloc.cpp:73: error: (-4:Insufficient memory) Failed to allocate 12884901888 bytes in function ‘cv::OutOfMemoryError’

?

Example wih python and windows

Microsoft Windows [version 10.0.19041.985]
(c) Microsoft Corporation. Tous droits réservés.

C:\Users\Laurent>python
Python 3.8.5 (tags/v3.8.5:580fbb0, Jul 20 2020, 15:57:54) [MSC v.1924 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> import cv2 as cv
>>> img = np.zeros(shape=(33000,33000), dtype=np.uint8)
>>> cv.imwrite("test.png",img)
True
>>> img2 = cv.imread("test.png")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
cv2.error: OpenCV(4.5.2-dev) G:\Lib\opencv\modules\imgcodecs\src\loadsave.cpp:77: error: (-215:Assertion failed) pixels <= CV_IO_MAX_IMAGE_PIXELS in function 'cv::validateInputImageSize'

>>> exit()

C:\Users\Laurent>set OPENCV_IO_MAX_IMAGE_PIXELS=17179869184

C:\Users\Laurent>python
Python 3.8.5 (tags/v3.8.5:580fbb0, Jul 20 2020, 15:57:54) [MSC v.1924 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> import cv2 as cv
>>> img2 = cv.imread("test.png")
>>> img2.shape
(33000, 33000, 3)
>>>
7

I stepped into the code and found that it fails at line #209 in grfmt_bmp.cpp
///////////////////////////////////////////////////////////////
// FIXIT: use safe pointer arithmetic (avoid ‘int’), use size_t, intptr_t, etc
CV_Assert(((uint64)m_height * m_width * nch < (CV_BIG_UINT(1) << 30)) && “BMP reader implementation doesn’t support large images >= 1Gb”);
///////////////////////////////////////////////////////////////
May be I can modify the code to CV_BIG_UINT(1) << 40. But I am not sure what will happen. It may start failing elsewhere. After reading into one Mat I need to copy this image to another Mat.

1 Like
8

I am using bmp file format. cv::validateInputImageSize() in LoadSave.cpp does not fail as I have set the environment variables to large values (CV_IO_MAX_IMAGE_WIDTH = CV_IO_MAX_IMAGE_HEIGHT = 100,000, CV_IO_MAX_IMAGE_PIXELS = 17179869184).

9

Are you using bmp format?
I think bmp file size is limited 4GB : 4 bytes The size of the BMP file in bytes

10

I agree. But I am not able to read / write even 3 GB files.

11

Are you using c++ python java…?

12

this seems to require a patch to OpenCV.

OpenCV has value checks that are cautious.

you can’t simply override/change the checks. you can, but then things might break.

you have to make sure all the code following the check handles those values correctly.

that includes whatever library is used, but also OpenCV’s code itself.

13

I am using VC++ (VS2015), WIN 10 on i5 PC with RAM 12 GB.

Regards,

Kishor Durve

14

Something is weird : can you post error message?

can you try this code :

void main(void)
{
    system("echo %OPENCV_IO_MAX_IMAGE_PIXELS%");
    Mat img(33000, 33000, CV_8UC3);
    RNG r;
    r.fill(img, RNG::UNIFORM, 0, 256);
    cout << "Saving image\n";
    imwrite("test.png", img);
    cout << "Loading image\n";
    Mat img2 = imread("test.png");
    cout << "Image size " << img2.size() << "\n";
}

and launch it form windows console (cmd)
and post result

15

I have tried your code.

//////////////////////////////////
OPENCV_IO_MAX_IMAGE_PIXELS is 17179869184

I could write and read png image file of size 30000x30000.
I tried image file (png) of size 48000x48000 and could write and read this also.
You were right - the problem seems to be related to bmp file format.
//////////////////////////////////

Thanks for your help.

Regards,

Kishor Durve

16

Apparently, the max size of a BMP image is 32768x32768, and with three bytes per pixel that would (uncompressed) be around three gigs. With an alpha channel, four gigs. Likely, the pixel size limit is hit here.