Undefined Behavior: using numpy.array() to convert instances of PIL.TiffImagePlugin.TiffImageFile to ndarrays after opening TIFF with Image.open()

[JordanPavlic]

Since I spent multiple days trying to address this issue in my project and cannot find any documented information on why this could be occurring on either StackExchange or the Pillow Issues board, I'm posting it here for some help.

What did you do?

Attempted to import a .tif image using Image.open() and then convert the Image to an array.

What did you expect to happen?

Successful conversion of the pillow image to a numpy ndarray.
As documented per #3301

What actually happened?

Both numpy.array() and numpy.asarray() returned an instance of Pil.TiffImagePlugin.TiffImageFile instead of a numpy ndarray when called once. Calling numpy.asarray() or numpy.array() more than once on the instance of Pil.TiffImagePlugin.TiffImageFile causes the numpy.asarray() or numpy.array() to return a numpy ndarray even if no assignment of the returned value from the first call occurs. it's as if the call to numpy.array() or numpy.asarray() is mutating the Pil.TiffImagePlugin.TiffImageFile instance.

I ran the following tests to show the oddity of the behavior, each time I got the console print message "tempfile.tif: Cannot read TIFF header." This error message was not part of my code base and was not raise with an exception. The image I'm using is a scanned .tiff image in grayscale sent through Microsoft outlook.

Case 1:

Code:

Result:

Case 2:

Code:

Result:

Case 3 :

Code:

Result:

Case 4:

Code:

Result:

Case 5:

Code:

Result:

Unfortunately, I'm unable to post the image I'm analyzing as it contains confidential information from my employer. However, since the behavior of the library itself is the really strange thing I'm hoping that should be enough to start looking into why this strange behavior is occurring.

However, here is some file information for the file that I can provide:

What versions of Pillow and Python are you using?

[radarhere]

You mention #3301. In that issue, users are instructed to use asarray. You are just using array.

[JordanPavlic]

I also evaluated with asarray instead of array and confirmed the issue is still present. See Case 4 in "What actually happened?" above.

[radarhere]

If it helps, the error isn't technically coming from our code either - it is coming from libtiff, at either https://gitlab.com/libtiff/libtiff/blob/master/libtiff/tif_open.c#L291 or https://gitlab.com/libtiff/libtiff/blob/master/libtiff/tif_open.c#L415

[radarhere]

Regarding your comment about the returned values, I actually can't reproduce this with Pillow 5.3. The following script just gives me five True values.

from PIL import Image
import numpy as np

im = Image.open("Tests/images/hopper.tif")
nd = np.array(im)
a = str(nd)
b = str(np.array(im))
print(a == b)

im = Image.open("Tests/images/hopper.tif")
np.array(im)
nd = np.array(im)
a = str(nd)
b = str(np.array(im))
print(a == b)

im = Image.open("Tests/images/hopper.tif")
nd1 = np.array(im)
nd2 = np.array(im)
a = str(nd2)
b = str(np.array(nd1))
print(a == b)

im = Image.open("Tests/images/hopper.tif")
nd = np.asarray(im)
a = str(nd)
b = str(np.asarray(im))
print(a == b)

im = Image.open("Tests/images/hopper.tif")
np.asarray(im)
nd = np.asarray(im)
a = str(nd)
b = str(np.asarray(im))
print(a == b)

[JordanPavlic]

That's definitely helpful!

Another strange addition to this issue:
Code:

Result:

I added the above code and started to run my main script again, which iterates through all files in the drive location. It appears that at the numpy.array(img) the console is now waiting for keyboard input to continue. At the bottom of the result you can see that I'm on numpy.array(img), but execution doesn't continue until I press a key. I wonder if something in libtiff is holding up execution?

regarding your last post, can you post your test image? I would like to run your test code on my system and then post back the results.

Edit: I've also confirmed the same behavior on my end using numpy.asarray() as well.

[radarhere]

https://github.com/python-pillow/Pillow/raw/master/Tests/images/hopper.tif

[JordanPavlic]

Just ran the following and got your result:

I ran using your hopper.tif here just to clarify by the way.

Code:

Result:

An interesting thing I noticed is that I did not get the internal libtif error this time around. I'll see if I can replicate the process for creating the image I'm analyzing without any confidential materials present.

[JordanPavlic]

Also, rerunning your tests with the image I'm analyzing currently gives me the following result:

Edit: as with the above, I realized this was waiting on keyboard input to continue as well. Here are the results when I hit enter for each step:

[JordanPavlic]

Here is an example of an image that's causing the issue to appear:
error_causing_image.zip

[radarhere]

Thanks, but I'm still not seeing a problem.

Here's another thought - the description of needing to hit Enter, the sqlalchemy lines being printed out, and the .filePath that suggests that you're using Twisted (but maybe not) - all of this sounds like interference from other libraries. It would be ideal if you could put together a script that could be run by itself, from start to finish, that shows the problem.

[JordanPavlic]

I just did a quick check of both pip list and conda list and I don't believe I have twisted installed. The .filePath comes from one of my internal classes I use to hold file metadata and the filepath itself is a pathlib object.

I pulled out your test script and modified it to run on it's own with the image in the same directory. Hopefully this is what you were after, if not just let me know and I'll see what I can do.

It still seems to wait for key input and fails the tests on my end. I've attached the code I'm running as well as the image. I'll go ahead and reinstall both packages for numpy and pillow just to see if that resolves it.

Code:

Result:

Here is a copy of the files executed:
Test Code.zip

[JordanPavlic]

I reinstalled both numpy and pillow and no change.

[radarhere]

Thanks for your efforts, but I still can't reproduce. I've also tried on our CI jobs, and found no success there either.

[JordanPavlic]

Thanks radarhere, I really appreciate all the help on this. I'm going to try a clean conda environment with only the required dependencies next and see if that works. I'll let you know how it goes.