Building from source does not support freetype and raqm

[ccoulombe]

• OS: Centos
• Python: 3.10
• Pillow: 9.4.0
• freetype : 6.16.1

Building Pillow 9.4.0 from source does not seem to properly provide RAQM support.
Building with the bundled raqm (v0.9.0) or a system raqm (v0.9.0) fails to provide freetype or raqm support.

Using bundled raqm

Build commands

cd Pillow-9.4.0
export CPATH=$EBROOTGENTOO/include:$CPATH
export LIBRARY_PATH=$EBROOTGENTOO/lib64:$LIBRARY_PATH
sed -i -e 's;/sbin/ldconfig;ldconfig;g' setup.py
python setup.py build_ext --debug --vendor-raqm bdist_wheel |& tee bundled.out

The build setup summary is ok:

--------------------------------------------------------------------
PIL SETUP SUMMARY
--------------------------------------------------------------------
version      Pillow 9.4.0
platform     linux 3.10.2 (main, Feb  4 2022, 19:10:35)
             [GCC 9.3.0]
--------------------------------------------------------------------
--- JPEG support available
--- OPENJPEG (JPEG2000) support available (2.3)
--- ZLIB (PNG/ZIP) support available
*** LIBIMAGEQUANT support not available
--- LIBTIFF support available
--- FREETYPE2 support available
--- RAQM (Text shaping) support available (bundled)
--- LITTLECMS2 support available
--- WEBP support available
--- WEBPMUX support available
--- XCB (X protocol) support available
--------------------------------------------------------------------

But once installed in a virtual environment, freetype and raqm are mentioned as not installed but the libraries are correctly linked:

$ python -m PIL | rg installed
*** FREETYPE2 support not installed
*** RAQM (Bidirectional Text) support not installed
*** LIBIMAGEQUANT (Quantization method) support not installed

$ ldd $VIRTUAL_ENV/lib/python3.10/site-packages/PIL/_imagingft.cpython-310-x86_64-linux-gnu.so:
	linux-vdso.so.1 (0x00007ffceb9fd000)
	libfreetype.so.6 => /cvmfs/soft.computecanada.ca/gentoo/2020/usr/lib64/libfreetype.so.6 (0x00002b4085e78000)
	libharfbuzz.so.0 => /cvmfs/soft.computecanada.ca/gentoo/2020/usr/lib64/libharfbuzz.so.0 (0x00002b4085f38000)
	libfribidi.so.0 => /cvmfs/soft.computecanada.ca/gentoo/2020/usr/lib64/libfribidi.so.0 (0x00002b4086041000)

And running a short example results in the following error:

ImportError: The _imagingft C module is not installed

but the library exists.

Using system raqm

Building a system RAQM results in the same as above.

Build commands:

cd Pillow-9.4.0
export CPATH=$HOME/libraqm/INST/usr/local/include:$EBROOTGENTOO/include:$CPATH
export LIBRARY_PATH=$HOME/libraqm/INST/usr/local/lib64:$EBROOTGENTOO/lib64:$LIBRARY_PATH
sed -i -e 's;/sbin/ldconfig;ldconfig;g' setup.py
python setup.py build_ext --debug --enable-raqm bdist_wheel |& tee system.out

Setup from the build output:

PIL SETUP SUMMARY
--------------------------------------------------------------------
version      Pillow 9.4.0
platform     linux 3.10.2 (main, Feb  4 2022, 19:10:35)
             [GCC 9.3.0]
--------------------------------------------------------------------
--- JPEG support available
--- OPENJPEG (JPEG2000) support available
--- ZLIB (PNG/ZIP) support available
*** LIBIMAGEQUANT support not available
--- LIBTIFF support available
--- FREETYPE2 support available
--- RAQM (Text shaping) support available
--- LITTLECMS2 support available
--- WEBP support available
--- WEBPMUX support available
--- XCB (X protocol) support available

But once installed in a virtual environment, freetype and raqm are mentioned as not installed but the libraries are correctly linked:

(495) Pillow-9.4.0 (main *) $ python -m PIL | rg installed
*** FREETYPE2 support not installed
*** RAQM (Bidirectional Text) support not installed
*** LIBIMAGEQUANT (Quantization method) support not installed

(495) Pillow-9.4.0 (main *) $ ldd $VIRTUAL_ENV/lib/python3.10/site-packages/PIL/_imagingft.cpython-310-x86_64-linux-gnu.so:
	linux-vdso.so.1 (0x00007ffe3f9b6000)
	libfreetype.so.6 => /home/coulombc/libraqm/INST/usr/local/lib64/libfreetype.so.6 (0x00002aaeeba95000)
	libraqm.so.0 => /home/coulombc/libraqm/INST/usr/local/lib64/libraqm.so.0 (0x00002aaeebb88000)
	libc.so.6 => /cvmfs/soft.computecanada.ca/gentoo/2020/lib64/libc.so.6 (0x00002aaeebbc6000)
	libharfbuzz.so.0 => /home/coulombc/libraqm/INST/usr/local/lib64/libharfbuzz.so.0 (0x00002aaeebd80000)
	libfribidi.so.0 => /cvmfs/soft.computecanada.ca/gentoo/2020/usr/lib64/libfribidi.so.0 (0x00002aaeec045000)

---

This looks like a bug to me but could a wrong detection from Pillow. Anyone successfully built Pillow with RAQM? What am I missing here?

[radarhere]

What version of CentOS are you using?

I wouldn't have thought that you needed to run sed -i -e 's;/sbin/ldconfig;ldconfig;g' setup.py. If you'd like to talk about why that is, maybe we can fix that within Pillow itself.

once installed in a virtual environment

So Pillow works without a problem in your normal environment?

[ccoulombe]

It is a multi-layer system, the centos layer is 7.9.

I wouldn't have thought that you needed to run sed -i -e 's;/sbin/ldconfig;ldconfig;g' setup.py. If you'd like to talk about why that is, maybe we can fix that within Pillow itself.

This is needed since the path of ldconfig is hardcoded in the setup, relying on the ldconfig from $PATH would be flexible to different systems.

No, Pillow does not work properly. In summary, the setup after the build says everything is ok, but once installed in a virtual environment, support for freetype or raqm is not available.

Here's a fresh build that reproduce the issue:

~ $ module load python/3.10
~ $ git clone https://github.com/python-pillow/Pillow.git -b 9.4.0
~ $ makevenv 
(8666) ~ $ pip install oldest-supported-numpy cython
(8666) ~ $ cd Pillow/
(8666) Pillow ((9.4.0) ) $ sed -i -e 's;/sbin/ldconfig;ldconfig;g' setup.py
(8666) Pillow ((9.4.0) *) $ export CPATH=$EBROOTGENTOO/include:$CPATH
(8666) Pillow ((9.4.0) *) $ export LIBRARY_PATH=$EBROOTGENTOO/lib64:$LIBRARY_PATH
(8666) Pillow ((9.4.0) *) $ python setup.py build_ext --inplace --debug --vendor-raqm bdist_wheel |& tee out.out
running build_ext
...
Using vendored version of raqm
...
--------------------------------------------------------------------
PIL SETUP SUMMARY
--------------------------------------------------------------------
version      Pillow 9.4.0
platform     linux 3.10.2 (main, Feb  4 2022, 19:10:35)
             [GCC 9.3.0]
--------------------------------------------------------------------
--- JPEG support available
--- OPENJPEG (JPEG2000) support available (2.3)
--- ZLIB (PNG/ZIP) support available
*** LIBIMAGEQUANT support not available
--- LIBTIFF support available
--- FREETYPE2 support available
--- RAQM (Text shaping) support available (bundled)
--- LITTLECMS2 support available
--- WEBP support available
--- WEBPMUX support available
--- XCB (X protocol) support available
--------------------------------------------------------------------

--------------------------------------------------------------------
PIL SETUP SUMMARY
--------------------------------------------------------------------
version      Pillow 9.4.0
platform     linux 3.10.2 (main, Feb  4 2022, 19:10:35)
             [GCC 9.3.0]
--------------------------------------------------------------------
--- JPEG support available
--- OPENJPEG (JPEG2000) support available
--- ZLIB (PNG/ZIP) support available
*** LIBIMAGEQUANT support not available
--- LIBTIFF support available
--- FREETYPE2 support available
--- RAQM (Text shaping) support available (bundled)
--- LITTLECMS2 support available
--- WEBP support available
--- WEBPMUX support available
--- XCB (X protocol) support available
--------------------------------------------------------------------

(8666) Pillow ((9.4.0) *) $ pip install dist/Pillow-9.4.0-cp310-cp310-linux_x86_64.whl --force-reinstall
(8666) Pillow ((9.4.0) *) $ cd
(8666) ~ $ python -c 'from PIL import features; print(features.check("raqm"))'
None
(8666) ~ $ python -c 'from PIL import features; print(features.check("freetype2"))'
False

[ccoulombe]

Importing the library _imagingft raises a missing symbol:

(8666) Pillow ((9.4.0) *) $ cd src/PIL
(8666) PIL ((9.4.0) *) $ python -c 'import _imagingft'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: /home/coulombc/wheels_builder/Pillow/src/PIL/_imagingft.cpython-310-x86_64-linux-gnu.so: undefined symbol: FT_Get_Transform
(8666) PIL ((9.4.0) *) $ nm -u _imagingft.cpython-310-x86_64-linux-gnu.so |& rg Transform
                 U FT_Get_Transform
                 U FT_Vector_Transform

freetype version is: libfreetype.so.6.16.1
harfbuzz version is : 2.6.4

Ah, those versions are too old for raqm.

Forcing the path to the libfreetype and libharfbuzz built as part of raqm fixed importing _imagingft.

[ccoulombe]

The issue was the version of freetype and harfbuzz that were too old for raqm.
Using libfreetype.so.6.18.0 and libharfbuzz.so.0.30000.0, I can now successfully:

(8666) ~ $ python -c 'from PIL import features; print(features.check("freetype2"))'
True
(8666) ~ $ python -c 'from PIL import features; print(features.check("raqm"))'
True

In light of this, and in order to help wheels builders/maintainers, I'd suggest to be more explicit when :

• the exception is raised about import _imagingft
• checking module
• checking feature module
  If an another exception is not acceptable, maybe raise a user warning?

Also, I think the setup should verify that the version of dependencies are met.

[radarhere]

freetype version is: libfreetype.so.6.16.1

From what I can see, this is actually freetype 2.9.1

We should support it, according to https://pillow.readthedocs.io/en/stable/deprecations.html#freetype-2-7

Support for FreeType 2.7 has been removed.

We recommend upgrading to at least FreeType 2.10.4, which fixed a severe vulnerability introduced in FreeType 2.6 (CVE-2020-15999).

However, libraqm raised the minimum version of FreeType required in HOST-Oman/libraqm#151.

[ccoulombe]

The main culprit was the version of harfbuzz (2.6.4).
According to raqm v0.9.0, freetype should be >= 2.11 and harfbuzz >= 3.0.0

[radarhere]

PR #7044 has replaced "/sbin/ldconfig" with "ldconfig". This should be available in Pillow 9.5.0 in a few days.

[ccoulombe]

Great, thanks!

[nulano]

It might be worth checking the version numbers of used libraries before starting a build, but setup.py is already quite complex, and other than the bundled raqm (which is not the default option), Pillow usually supports quite old versions of dependencies. It is expected that most users will build using their system raqm library, which should import correctly.

I imagine you got an "implicit function declaration" warning (which really should be an error IMO) during the build. I'd suggest using CFLAGS="-Werror=implicit-function-declaration" to treat it as an error as is done for CI in #4963. Perhaps a compile-time version check could be added to the bundled raqm sources to make it more obvious?

Forcing the path to the libfreetype and libharfbuzz built as part of raqm fixed importing _imagingft.

I could be misunderstanding here, but is (part of) the issue here that your system raqm was not correctly importing freetype and harfbuzz?

[ccoulombe]

Agreed, then the question : How can one normal user easily verify the version when they are not documented in the readme nor the installation docs? raises.
I am not a normal user and I can dig around to found the answers.

But! A setup or any package should verify that its requirements are met. So many do with very complex setups, think of numpy or scipy. An idea could be to change the setup for meson or cmake ? Or simply verify them in the current setup script.

Sure, I could turn the warning into an error.

Yes, both, the bundled and system raqm did not linked to a recent version of freetype and harfbuzz, causing the symbol not found. We updated the version of both freetype and hafbuzz and installed a system raqm to satisfy all requirements.

If the exception raised would have been redirected to stderr or a warning issued, I would have found the culprit sooner.

[ccoulombe]

From a previous comment:

In light of all this, and in order to help wheels builders/maintainers, I'd suggest to be more explicit when :

the exception is raised about import _imagingft
checking module
checking feature module
If an another exception is not acceptable, maybe raise a user warning?

With the above, I'd also suggest:

• documenting the minimal version for all requirements, some are documented, but others are not
• adding requirements verification to the setup/build

[nulano]

How can one normal user easily verify the version when they are not documented in the readme nor the installation docs?

I'd also suggest:

• documenting the minimal version for all requirements, some are documented, but others are not
• adding requirements verification to the setup/build

AFAIK there are no explicit minimum versions (except for FreeType, as noted above), simply a requirement that the tested systems' default library versions are supported. IIRC, support for some library versions was unintentionally dropped in the past simply due to lack of testing.

Looking at the latest CI run, the following are the minimum versions currently tested:

--- TKINTER support ok, loaded 8.5                                 # centos-7
--- FREETYPE2 support ok, loaded 2.8.0                             # centos-7
--- LITTLECMS2 support ok, compiled for 2.6                        # centos-7
--- WEBP support ok, loaded 0.3.0                                  # centos-7
--- WEBP Transparency support ok
*** WEBPMUX support not installed
*** WEBP Animation support not installed
--- JPEG support ok, compiled for libjpeg-turbo 1.2.90             # centos-7
--- OPENJPEG (JPEG2000) support ok, loaded 2.3.1                   # centos-7
--- ZLIB (PNG/ZIP) support ok, loaded 1.2.7                        # centos-7
--- LIBTIFF support ok, loaded 4.0.3                               # centos-7
--- RAQM (Bidirectional Text) support ok, loaded 0.7.0             # centos-7
--- LIBIMAGEQUANT (Quantization method) support ok, loaded 2.16.0  # fedora-36
--- XCB (X protocol) support ok     # no idea, XCB does not expose its version number

However, much older versions are likely supported, although it is unclear which ones due to lack of CI testing.
I agree that the documentation should probably be updated - it might be worth discussing adding explicit minimum version requirements.