Cross-compile issue against Synology toolchain

[th0ma7]

What did you do?

SynoCommunity provide open-source packages to run on Synology NAS. Our project page is https://github.com/SynoCommunity/spksrc

We used to provide pillow wheel part of the our default python2 package. It is also provide with some native python apps we package needed this wheel. While preparing our new python39 package I tried re-including it "as-is" but it fails to cross-compiling on some archs using the synology toolchain, namely arm and ppc.

What did you expect to happen?

inability to find some of the needed include files

What actually happened?

What are your OS, Python and Pillow versions?

• OS: Linux
• Python: 3.7-3.9
• Pillow: 8.2-8.4

Example of build failure on arm and ppc archs:
https://github.com/SynoCommunity/spksrc/pull/4937/checks?sha=5ca113eb0b3b58169b6a6885d6a1383a35ba9560
Our local patch we maintain to allow it to build (we add to the environment STAGING_INSTALL_PREFIX variable)
https://github.com/SynoCommunity/spksrc/blob/master/cross/pillow/patches/001-fix-paths.patch

What would be really nice is to have a work-around where I guess it would use the environment CFLAGS pointing to the toolchain to find the needed includes.

[th0ma7]

Reading https://pillow.readthedocs.io/en/stable/releasenotes/3.3.0.html
I now strongly presume that --disable-platform-guessing should be set by default.
Instead you might need to add --enable-platform-guessing for those who need it.

In my use-case I would need the ability to set an environment variable such as:

ENV += PILLOW_PLATFORM_GUESSING=disable
ENV += FRIBIDI_INSTALL=system

PyNaCl does that for SODIUM_INSTALL for instance.

EDIT: Adding @Safihre and @hgy59 to the loop

[radarhere]

When I test, --disable-platform-guessing is not true by default. To make it true, you can use the build flag - https://pillow.readthedocs.io/en/stable/installation.html#build-options

Build flag: --disable-platform-guessing. Skips all of the platform dependent guessing of include and library directories for automated build systems that configure the proper paths in the environment variables (e.g. Buildroot).

So. that is python3 setup.py build_ext --disable-platform-guessing install.

---

To change the include directories using CFLAGS,
https://pillow.readthedocs.io/en/stable/installation.html#external-libraries

If they are installed in a non-standard location, you may need to configure setuptools to use those locations by editing setup.py or setup.cfg, or by adding environment variables on the command line:

CFLAGS="-I/usr/pkg/include" python3 -m pip install --upgrade Pillow

And -L can be similarly used for library directories.

So looking at your Makefile, I had initially thought of something like

ENV += CFLAGS="-L$(STAGING_INSTALL_PREFIX)/lib -I$(STAGING_INSTALL_PREFIX)/include"

However, when I test, I don't find this to be necessary. Removing the patch and just adding in build_ext --disable-platform-guessing, it passes all CIs. Does that resolve your situation?

[th0ma7]

So. that is python3 setup.py build_ext --disable-platform-guessing install.

This is where it becomes interesting: as I am building multiple wheels I can't pass this specific argument to that single wheel build out of the list of wheels being processed.

That's why an environment variable allowing to do that would be needed, such as PILLOW_PLATFORM_GUESSING=disable. Similarly this could be used also for PILLOW_FRIBIDI_INSTALL=system.

This way pip will have access to these environment variables and if Pillow is able to make use of them I'd be all set.

There are other cases where this is available such as SODIUM_INSTALL=system for PyNaCl or PYCARES_USE_SYSTEM_LIB=1 for pycares, etc.

[radarhere]

This is where it becomes interesting: as I am building multiple wheels I can't pass this specific argument to that single wheel build out of the list of wheels being processed.

So this isn't just about spksrc? I would have thought that the branch and GHA run from my last comment meant that I'd found a solution.

[th0ma7]

not entirely sure I follow? Yes, this is for spksrc framework. Although the way the framework works, it doesn't allow passing specific arguments to "one" of the many requirements being built at once through pip using the package requirements.txt file. The way to circumvent this lack of argument passing is by using environment variables.

Aaaahhh, re-re-reading, I just noticed your commit https://github.com/radarhere/spksrc/commit/256770199a4e65cfe8a0b4effb706f4dfda37adf

I had not followed correctly your sub-links, now I get it. I'll have another look at it and confirm, thnx.

[radarhere]

You've created SynoCommunity/spksrc#5023, and it passes. Is that all that you needed from us?

[th0ma7]

I took more time to analyse the situation and tried once more to figure out what the situation really is.

First off, thnx for the proposed change. Oddly enough, the change your proposing does look like working but it doesn't. When looking at the logs the build process goes outside from the build environment and uses the host gcc and include files instead of the Synology toolchain as per provided through the various environment variables CC,CFLAGS, LDFLAGS, etc.

Also, the cross/pillow/Makefile you edited is our "legacy" method to cross-compile wheels that we're trying to move away from to only use (when possible) a pip call with a requirement.txt file.

I've created an new PR with this as focus SynoCommunity/spksrc#5023

When cross-compiling python310 + Pillow wheel I get:

  In file included from /usr/include/stdlib.h:55:0,
                   from /home/spksrc/pillow-build-arg/spksrc/spk/python310/work-88f6281-6.1/install/var/packages/python310/target/include/python3.10/Python.h:34,
                   from src/_imagingmorph.c:14:
  /usr/include/bits/floatn.h:75:1: error: unknown machine mode '__TC__'
  /usr/include/bits/floatn.h:87:1: error: unknown type name '__float128'
  In file included from /home/spksrc/pillow-build-arg/spksrc/spk/python310/work-88f6281-6.1/install/var/packages/python310/target/include/python3.10/pythread.h:128:0,
                   from /home/spksrc/pillow-build-arg/spksrc/spk/python310/work-88f6281-6.1/install/var/packages/python310/target/include/python3.10/Python.h:136,
                   from src/_imagingmorph.c:14:
  /usr/include/pthread.h:682:6: warning: '__regparm__' attribute directive ignored [-Wattributes]
  /usr/include/pthread.h:694:3: warning: '__regparm__' attribute directive ignored [-Wattributes]
  /usr/include/pthread.h:717:6: warning: '__regparm__' attribute directive ignored [-Wattributes]
  /usr/include/pthread.h:730:3: warning: '__regparm__' attribute directive ignored [-Wattributes]
  /usr/include/pthread.h:739:6: warning: '__regparm__' attribute directive ignored [-Wattributes]
  error: command '/home/spksrc/pillow-build-arg/spksrc/toolchain/syno-88f6281-6.1/work/arm-marvell-linux-gnueabi/bin/arm-marvell-linux-gnueabi-gcc' failed with exit code 1
  ----------------------------------------
  ERROR: Failed building wheel for Pillow
  Running setup.py clean for Pillow
Failed to build Pillow
ERROR: Failed to build one or more wheels

I presume that this means Pillow is still including standard include paths (ex: /usr/include/) instead of only using what's in the environment. Thus the include files found on the running host (x86_64) contradicts what's in the build environment for cases where I'm cross-compiling for armv5, armv7 or ppc archs.

EDIT: Re-reading from the doc online (https://pillow.readthedocs.io/en/latest/installation.html#building-from-source) using pip the BUILD_ARGS needs to be converted as --global-option="build_ext" --global-option="--enable-[feature]". But that would then affect all wheels from the requirement file being built at that given time. Again, using environment variables would alleviate that in my use case (e.g. PILLOW_PLATFORM_GUESSING=disable).

[radarhere]

Where does that error come from? When I look at https://github.com/SynoCommunity/spksrc/runs/4638182461, I instead see a Pillow failure with "The headers or library files could not be found for jpeg"

[th0ma7]

I caught the error when invoking the build manually locally with the docker image (for instance make arch-armv7-6.1). Indeed the jpeg error you notice was before my last commit fixing this SynoCommunity/spksrc@8b24d77
The error should show in the logs once the auto-build complete.

And in my PR its now obvious to me that the BUILD_ARGS will not be taken into account as pip will only honor --global-option arguments.

[th0ma7]

@radarhere based on our exchanges I've adapted our framework in order to allow passing arguments to pip using --global-option which seems to work all-right. I think this will be a nice addition to how our framework is working.

Related to Pillow more specifically, while using --global-option works out, the side effect when building multiple wheels in one shot using a requirement file is that it the options are passed to all the wheels being built at once. As long as there isn't any conflicting options between wheels its probably fine. But I'll still point out that the best option for this use-case would be to allow environment variables which then become specific for the wheel being built really needing such variable.

All in all, I'd say my issue is now probably solved. But my ask would now be to add a new feature to allow using environment variable equivalent to each of the --enable|disable flags, specific to pillow (with PILLOW_ prefix). Thnx in advance and for the help, cheers!

[th0ma7]

I've circumvented the issue by further adapting our framework so each cross-compiled wheel gets compiled individually instead of parsing the entire requirement file in one shot. This allowed afterward to pass per-wheel --global-option leading to solving my use case entirely.

Still, PILLOW_ prefix environment variables would be welcomed but is now more of a nice to have. Closing the issue. Thnx for you help.