OverflowError: size does not fit in an int

[ziky90]

Hi,

I am getting an error very similar to #1475 with the Pillow==3.4.2, that is supposed to be fixed.

Actually I'm getting:

File "/home/jan/my_script.py", line 230, in save
    img = Image.fromarray(image, 'RGB')
  File "/usr/local/lib/python2.7/dist-packages/PIL/Image.py", line 2187, in fromarray
    return frombuffer(mode, size, obj, "raw", rawmode, 0, 1)
  File "/usr/local/lib/python2.7/dist-packages/PIL/Image.py", line 2137, in frombuffer
    return frombytes(mode, size, data, decoder_name, args)
  File "/usr/local/lib/python2.7/dist-packages/PIL/Image.py", line 2070, in frombytes
    im.frombytes(data, decoder_name, args)
  File "/usr/local/lib/python2.7/dist-packages/PIL/Image.py", line 741, in frombytes
    s = d.decode(data)
OverflowError: size does not fit in an int

Details about my image are following:
shape = (31000, 30030, 3)
dtype = uint8

My Python version is Python 2.7.12

[wiredfool]

Can you post a minimal script that reproduces the error?

[ziky90]

@wiredfool Here is the minimal script that reproduces the error:

from PIL import Image
import  numpy as np

image = np.random.randint(255, size=(31000, 30030, 3))
image = image.astype('uint8')

# next line will cause the OverflowError
img = Image.fromarray(image, 'RGB')

[wiredfool]

The underlying issue in #1475 isn't actually fixed, Image.frombuffer is limited to a 2gb chunk. I think even with a mmap-able image (e.g. RGBX or RGBA), we'd still be running into an integer overflow issue in the raw decoder.

I kept getting memory errors on the randint line, but running this as:

from PIL import Image
import  numpy as np

a = np.zeros((31000, 30030, 3), dtype=np.uint8)

# next line will cause the OverflowError
img = Image.fromarray(a, 'RGB')

Gives me the same issue, it's failing in decode.c:127, the same spot as in #1475.

And attempting to mmap this:

a = np.zeros((31000, 30030, 4), dtype=np.uint8)
img = Image.fromarray(a, 'RGBX')

Fails in a new and interesting way:

  File "numpy_overflow.py", line 12, in <module>
    img = Image.fromarray(a, 'RGBX')
  File "/home/erics/vpy27-dbg/local/lib/python2.7/site-packages/Pillow-3.4.2-py2.7-linux-x86_64.egg/PIL/Image.py", line 2187, in fromarray
    return frombuffer(mode, size, obj, "raw", rawmode, 0, 1)
  File "/home/erics/vpy27-dbg/local/lib/python2.7/site-packages/Pillow-3.4.2-py2.7-linux-x86_64.egg/PIL/Image.py", line 2132, in frombuffer
    core.map_buffer(data, size, decoder_name, None, 0, args)
MemoryError: Integer overflow in ysize
[84908 refs]

which is in this code: map.c:345

    if (!PyArg_ParseTuple(args, "O(ii)sOn(sii)", &target, &xsize, &ysize,
                          &codec, &bbox, &offset, &mode, &stride, &ystep))
        return NULL;

    if (!PyImaging_CheckBuffer(target)) {
        PyErr_SetString(PyExc_TypeError, "expected string or buffer");
        return NULL;
    }

    if (stride <= 0) {
        if (!strcmp(mode, "L") || !strcmp(mode, "P"))
            stride = xsize;
        else if (!strncmp(mode, "I;16", 4))
            stride = xsize * 2;
        else
            stride = xsize * 4;
    }

    if (ysize > INT_MAX / stride) {
        PyErr_SetString(PyExc_MemoryError, "Integer overflow in ysize");
        return NULL;
    }

[monjoybme]

Is this issue solved? I'm facing same issue in latest version of pillow too. Please help.

[radarhere]

@monjoybme As of Pillow 6.0.0, I would expect you not to be receiving, 'OverflowError: size does not fit in an int', but instead 'ValueError: not enough image data'. I have created PR #3791 to move the problem one step further and solve the issue completely.

[monjoybme]

@radarhere Could you tell me how can I solve ValueError: not enough image data issue. My function is below.

def np_to_pil(np_img):
  """
  Convert a NumPy array to a PIL Image.
  Args:
    np_img: The image represented as a NumPy array.
  Returns:
     The NumPy array converted to a PIL Image.
  """
  if np_img.dtype == "bool":
    np_img = np_img.astype("uint8") * 255
  elif np_img.dtype == "float64":
    np_img = (np_img * 255).astype("uint8")
return Image.fromarray(np_img)

I'm using this https://github.com/deroneriksson/python-wsi-preprocessing/blob/c82dcafff48718da041ab99b7534ba697db2cc43/deephistopath/wsi/util.py#L43

[radarhere]

To solve the error now, you will need to compile Pillow from my PR's branch.

git clone https://github.com/radarhere/Pillow
cd Pillow
git checkout int
python setup.py install

If my PR is merged into Pillow master before July 1, then you should be able to go back to using standard Pillow from 6.1.0 onwards.

[monjoybme]

@radarhere Thanks a lot. I compiled and getting new error ValueError: not enough image data

`File "WsiToImagePatch.py", line 35, in np_to_pil

return Image.fromarray(np_img)

File "/usr/local/lib/python3.5/dist-packages/PIL/Image.py", line 2554, in fromarray

return frombuffer(mode, size, obj, "raw", rawmode, 0, 1)

File "/usr/local/lib/python3.5/dist-packages/PIL/Image.py", line 2497, in frombuffer

return frombytes(mode, size, data, decoder_name, args)

File "/usr/local/lib/python3.5/dist-packages/PIL/Image.py", line 2430, in frombytes

im.frombytes(data, decoder_name, args)

File "/usr/local/lib/python3.5/dist-packages/PIL/Image.py", line 812, in frombytes

raise ValueError("not enough image data")

ValueError: not enough image data
`

[radarhere]

Okay. I would have expected that from Pillow 6.0, or from the current Pillow master, but not from my 'int' branch. Could you provide a self-contained example, I presume some form of WsiToImagePatch.py?

[monjoybme]

I'm using this code https://github.com/deroneriksson/python-wsi-preprocessing/blob/c82dcafff48718da041ab99b7534ba697db2cc43/deephistopath/wsi/util.py#L57

[radarhere]

You're linking to a function. I don't know what the input is - 'np_img'.

For example, this script uses that function, but it works fine even without my PR.

from PIL import Image
import numpy as np

def np_to_pil(np_img):
  """
  Convert a NumPy array to a PIL Image.
  Args:
    np_img: The image represented as a NumPy array.
  Returns:
     The NumPy array converted to a PIL Image.
  """
  if np_img.dtype == "bool":
    np_img = np_img.astype("uint8") * 255
  elif np_img.dtype == "float64":
    np_img = (np_img * 255).astype("uint8")
  return Image.fromarray(np_img)

np_img = np.ndarray(shape=(1, 1))
np_to_pil(np_img)

The particular input that you are feeding it is triggering the problem. If it's just that your input is too large, I would really think that this should have been solved by my PR. If it's not fixed, then something unexpected is happening, and to investigate further, I would like to be able to trigger the problem myself. To do that, I need to have the specific input.

[monjoybme]

Thanks a lot. I've shared (only with you) the code here https://github.com/monjoybme/WSI_to_patch/invitations.
In line 939, when I set NUM_TOP_TILES >= 13000 I'm facing OverFlowError. But when NUM_TOP_TILES <= 12000, the code is working fine. For reproducibility you just have to copy any .svs file into the folder /data/training_slides/test-001.svs. For easy purpose rename the .svs file name to test-001.svs. Please share your email if you need .svs file. Thanks for your kind support.