setup.py should not set zip_safe to True

[AkarinVS]

What did you do?

on vanilla macOS with xcode and necessary prerequisites installed (as static libraries), run

python3 setup.py install && python3 -c "from PIL import Image"

What did you expect to happen?

The import should be successful.

What actually happened?

Core version: None
Pillow version: 8.1.2
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File "...../lib/python3.8/site-packages/Pillow-8.1.2-py3.8-macosx-10.15-x86_64.egg/PIL/Image.py", line 97, in <module>
ImportError: The _imaging extension was built for another version of Pillow or PIL:
Core version: None
Pillow version: 8.1.2

The real cause is not because the _imaging.so native module is compiled for another version (as we just built it), but because Python is unable to dlopen it at all.

What are your OS, Python and Pillow versions?

• OS: macOS 10.15
• Python: 3.8.8
• Pillow: 8.1.2

Suspected Cause

We specified zip_safe to be generally True unless it's a debugging build or mingw build.

Pillow/setup.py

Line 912 in 51bbefe

zip_safe=not (debug_build() or PLATFORM_MINGW),

However, Python is unable to import a dynamic library from a zip file. Therefore, this package should definitely not be labeled zip_safe. I have checked many installations, and on many,easy_install somehow knows to extract the content of the egg into its a directory, thus hiding this issue on lots of platforms. I'm not yet sure why this doesn't happen on macOS. But nevertheless, I don't think we should use zip_safe: True on any platform: as https://docs.python.org/3/library/zipimport.html explicitly disallows importing dynamic modules.

[radarhere]

Hi. What version of setuptools are you using?

ImportError: The _imaging extension was built for another version of Pillow or PIL:
Core version: None
Pillow version: 8.1.2

This looks like #4784 (comment). If so, the solution might be to upgrade setuptools to at least 49.3.2.

[AkarinVS]

I was using setuptools 49.2.1, however, I tried both 49.3.2 and the latest 54.2.0, the results are the same. As long as the egg is not extracted (due to zip_safe=True), it can't import the _imaging native module.

[radarhere]

I just have to imagine that there's something else at work here, since this is specific to your installation, and not a problem experienced by other macOS users.

The real cause is not because the _imaging.so native module is compiled for another version (as we just built it), but because Python is unable to dlopen it at all.

If this were the case, wouldn't you expect that to trigger an ImportError instead?

If you believe that this is as straightforward as zip_safe though, you're free to create your own PR suggesting your change.

Would you be able to provide the complete output of the commands that you're running?

And also, to try and stamp out my theory, could you show the output of python3 -c "import setuptools;print(setuptools.__version__)"? We do have issues raised here from time to time where people unintentionally use a different Python version.

[hugovk]

I also could not reproduce:

• OS: macOS 10.14
• Python: 3.9.2
• Pillow: 8.1.2 (master)
• setuptools: 54.2.0

[wiredfool]

I saw something similar to this recently on Ubuntu 18.04 with Py3.9/deadsnakes and a new venv. I didn't get to the core of it, but I think I wound up creating the virtualenv with a different tool (either -mvenv or virtualenv, I don't remember which was the initial and which was the final solution) and the problem went away.

At the time I chalked it up to the somewhat insane splitting of python's packaging across 5 different .debs, where the ability to install from pip or have virtual environments is not included by default, but I'm not clear if that was actually the problem.

[AkarinVS]

if your installation is working, could you please check that the installed pillow egg is a single zip archive or an extracted folder? My hypothesis is that it only works for extracted egg case, and not when the egg is kept as is, in a zip archive. There are some conditions, which I'm still not yet sure of, that make setuptools to extract the egg. My proposed patch basically set `zip_safe = False` in setup.py and then it works reliably for me. Relying on setuptools to automatically detect this situation is unreliable and fragile, and as Python definitely cannot import native modules from a zip.

[radarhere]

>>> from PIL import Image
>>> Image
<module 'PIL.Image' from 'Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/Pillow-8.2.0.dev0-py3.8-macosx-11.2-arm64.egg/PIL/Image.py'>
>>> from PIL import _imaging
>>> _imaging
<module 'PIL._imaging' from 'Library/Caches/Python-Eggs/Pillow-8.2.0.dev0-py3.8-macosx-11.2-arm64.egg-tmp/PIL/_imaging.cpython-38-darwin.so'>

If I delete the -tmp directory, it is recreated when importing.

[nulano]

From https://setuptools.readthedocs.io/en/latest/userguide/miscellaneous.html#setting-the-zip-safe-flag (emphasis mine):

For some use cases (such as bundling as part of a larger application), Python packages may be run directly from a zip file. Not all packages, however, are capable of running in compressed form, because they may expect to be able to access either source code or data files as normal operating system files. So, setuptools can install your project as a zipfile or a directory, and its default choice is determined by the project’s zip_safe flag.

You can pass a True or False value for the zip_safe argument to the setup() function, or you can omit it. If you omit it, the bdist_egg command will analyze your project’s contents to see if it can detect any conditions that would prevent it from working in a zipfile. It will output notices to the console about any such conditions that it finds.

Currently, this analysis is extremely conservative: it will consider the project unsafe if it contains any C extensions or datafiles whatsoever. This does not mean that the project can’t or won’t work as a zipfile! It just means that the bdist_egg authors aren’t yet comfortable asserting that the project will work. [...]

Since Pillow does not need to access any source code or data files as regular files, this sounds like setting zip_safe=True should be safe. IIUC the only reason setuptools disables it by default for projects with C extensions is that it can't be sure they don't access other "regular" files that are part of its distribution, which Pillow doesn't.

[nulano]

I was using setuptools 49.2.1, however, I tried both 49.3.2 and the latest 54.2.0, the results are the same. As long as the egg is not extracted (due to zip_safe=True), it can't import the _imaging native module.

Perhaps the old build using "setuptools 49.2.1" is still cached?

[AkarinVS]

I have completely removed my python install and rebuild everything, now it seems to work with latest setuptools 54.2.0 and it extracts the native modules into ~/Library/Cache/Python-Eggs/. Case closed. Dup of #4784. Thanks for all the explanations!