ResourceWarning: unclosed file when calling save / thumbnail on an ImageFile instance (for Python 3)

[kanoc]

This is a similar issue to #477, but the context is a bit different. Executing this code with python -Wall will generate a ResourceWarning (can be reproduced with an arbitrary image):

from PIL import Image

with Image.open('test_img.png') as img:
    img.save('test_img.jpg', format='JPEG', quality=80)

This code will not produce a warning:

from PIL import Image

with open('test_img.png', 'rb') as image_file:
    with Image.open(image_file) as img:
        img.save('test_img.jpg', format='JPEG', quality=80)

I've spent a few hours debugging this, and I think I've nailed it down. Apologies if I've misunderstood something.

The warning is caused by the line PIL/Image.py:762, when the file object was opened by Image.open:

return self.im.pixel_access(self.readonly)

Looking at he C bindings in _imaging.c:2799 I can see that because the return value of the call above is discarded, the line _imaging.c:2824 gets executed

PyObject_Del(self);

which will destroy the file object without closing it, thus generating a ResourceWarning: unclosed file... error.

To verify this, I've added a line to close the file before returning from the load method:

self.fp.close()
return self.im.pixel_access(self.readonly)

With this, no ResourceWarning was generated.

Tested with Pillow v2.5.1 and Python 3.3.2.

[wiredfool]

I don't really understand your references to Imaging-1.1.7/_imaging.c. _imaging.c has changed since then, and your line numbers reference the middle of _draw_pieslice.

[kanoc]

Sorry about that. Updated the line numbers using the _imaging.c file from pillow v2.5.1.

[wiredfool]

I'm seeing the resource warning, but I really don't understand it. Something is calling delete far sooner than it should, since as far as I can tell, everything is still in scope and reachable, and not letting the with close everything. Your line numbers still don't make sense to me. Perhaps you can link to specific lines on this page: https://github.com/python-pillow/Pillow/blob/master/_imaging.c

It was also my understanding that it was the responsibility of the individual image plugins to close their file descriptors, but I'm not seeing that in practice. I'm going to have to review that.

[kanoc]

The resource warning happens before the with has a chance to close anything. It happens in the save method (https://github.com/python-pillow/Pillow/blob/master/PIL/Image.py#L1595), when the load is executed (https://github.com/python-pillow/Pillow/blob/master/PIL/Image.py#L1633).

Forget my previous analysis, I was wrong. I've found the exact line that causes the warning: https://github.com/python-pillow/Pillow/blob/master/PIL/ImageFile.py#L234. This clears the reference to the file object, and when the ImageFile.load exits the file object is cleaned up before it is closed (triggering the resource warning).

By commenting out the line https://github.com/python-pillow/Pillow/blob/master/PIL/ImageFile.py#L234, the resource warnings go away. I'm not suggesting this as a solution, just as a proof.

[calvingiles]

I have been chasing similar issues in Image.load(). In python 3 I could create the ResourceWarning with the following test script with -Wdefault flags in the root of the code:

from PIL import Image
from numpy import array

img = Image.open('Tests/images/flower.jpg')
array(img)

If I wrap the second half of the Image.load() function in a try/finally and close it I can eliminate the ResourceWarning in python3 (I don't think you get it in python2) but Most of the tests fail!

So my awful workaround that fails tests is:

def load(self):
    """
    Allocates storage for the image and loads the pixel data.  In
    normal cases, you don't need to call this method, since the
    Image class automatically loads an opened image when it is
    accessed for the first time. This method will close the file
    associated with the image.

    :returns: An image access object.
    """
    if self.im and self.palette and self.palette.dirty:
        # realize palette
        self.im.putpalette(*self.palette.getdata())
        self.palette.dirty = 0
        self.palette.mode = "RGB"
        self.palette.rawmode = None
        if "transparency" in self.info:
            if isinstance(self.info["transparency"], int):
                self.im.putpalettealpha(self.info["transparency"], 0)
            else:
                self.im.putpalettealphas(self.info["transparency"])
            self.palette.mode = "RGBA"

    try:
        if self.im:
            if HAS_CFFI and USE_CFFI_ACCESS:
                if self.pyaccess:
                    return self.pyaccess
                from PIL import PyAccess
                self.pyaccess = PyAccess.new(self, self.readonly)
                if self.pyaccess:
                    return self.pyaccess
            return self.im.pixel_access(self.readonly)
    finally:
        self.close()

I notice that you have numpy tests so perhaps they need extending to catch this case?

I am happy to help with this as it impacts scikit-learn.

[kanoc]

@calvingiles the case you describe has the same cause as described in my previous comment. Comment out the line https://github.com/python-pillow/Pillow/blob/master/PIL/ImageFile.py#L234 and the warnings go away.

What happens is:

1. numpy.array calls Image.__getattr__ (https://github.com/python-pillow/Pillow/blob/master/PIL/Image.py#L598)
2. Image.__getattr__ calls Image.tobytes (https://github.com/python-pillow/Pillow/blob/master/PIL/Image.py#L627)
3. Image.tobytes calls (via self.load()) ImageFile.load (https://github.com/python-pillow/Pillow/blob/master/PIL/ImageFile.py#L125)
4. And then, the key line is https://github.com/python-pillow/Pillow/blob/master/PIL/ImageFile.py#L234:

self.fp = None # might be shared

This line clears the only reference to the file object, therefore when the ImageFile.load exits, the file object is cleaned up without being closed. And this causes the ResourceWarning.

The only fix I can think of at moment would be to clear ImageFile.fp only if it was opened by ImageFile itself in ImageFile.__init__ (https://github.com/python-pillow/Pillow/blob/master/PIL/ImageFile.py#L84).

Until this is fixed, the workaround for your case would be:

from PIL import Image
from numpy import array

with open('Tests/images/flower.jpg', 'rb') as img_file:
    with Image.open(img_file) as img:
        array(img)

Update: fixed missing quotes.

[calvingiles]

Thanks. I will suggest this workaround for scikit-learn/scikit-learn and we can keep an eye on this. Any idea when a fix may get implemented?

[meawoppl]

@kanoc the letters rb need to be in quotes like "rb".

[hugovk]

I've added failing tests for these in this branch:

https://github.com/hugovk/Pillow/tree/ResourceWarning
https://travis-ci.org/hugovk/Pillow/builds/45600222

The image save fails for Python 3.4 and the NumPy array fails for 3.2, 3.3 and 3.4.

[aclark4life]

@kanoc Is it similar enough that we can close either this one or #477 to combine issues?

[hugovk]

(I merged latest master into my branch with tests, as expected they still fail in the same way:
https://github.com/hugovk/Pillow/tree/ResourceWarning
https://travis-ci.org/hugovk/Pillow/builds/55929699 )

[kanoc]

@aclark4life The issue reported in #477 was fixed and #477 was closed. I think the reference to #477 from pklaus/PhotoBrowser@8248b78 is wrong, the ResourceWarning there is caused by this issue.

As the cause of the ResourceWarning reported here is different than the one reported in #477 and #477 is closed, I don't see why these issues should be combined.