JPEG encoding with Pillow PyPI wheel is 6x slower than distro package

[wangqr]

What did you do?

Run the attached script to encode a RGB image to JPEG

What did you expect to happen?

The encoding should take about 1ms

What actually happened?

The encoding takes about 1ms with distribution provided package, but ~6ms with Pillow-8.0.1-cp3*-cp3*-manylinux1_x86_64.whl from PyPI

What are your OS, Python and Pillow versions?

I have tested on ArchLinux, Ubuntu and CentOS. Here are the results (all from same computer):

• Archlinux

# python-pillow
Python 3.8.6 (default, Sep 30 2020, 04:00:38)
[GCC 10.2.0]
Pillow 7.2.0
0.8895997999999963 ms/iter

# python-pillow-git (619a7b9f)
Python 3.8.6 (default, Sep 30 2020, 04:00:38)
[GCC 10.2.0]
Pillow 8.1.0.dev0
0.8732269799998903 ms/iter

# Pillow from PyPI
Python 3.8.6 (default, Sep 30 2020, 04:00:38)
[GCC 10.2.0]
Pillow 8.0.1
5.752080059999571 ms/iter

• Ubuntu

# python3-pillow
Python 3.8.5 (default, Jul 28 2020, 12:59:40)
[GCC 9.3.0]
Pillow 7.0.0
1.8287872446002438 ms/iter

# Pillow from PyPI
Python 3.8.5 (default, Jul 28 2020, 12:59:40)
[GCC 9.3.0]
Pillow 8.0.1
8.810945131699555 ms/iter

• CentOS

# python3-pillow
Python 3.6.8 (default, Apr 16 2020, 01:36:27)
[GCC 8.3.1 20191121 (Red Hat 8.3.1-5)]
Pillow 5.1.1
2.030805503507145 ms/iter

# Pillow from PyPI
Python 3.6.8 (default, Apr 16 2020, 01:36:27)
[GCC 8.3.1 20191121 (Red Hat 8.3.1-5)]
Pillow 8.0.1
8.873232635483145 ms/iter

Code

#!/bin/python3

import timeit
import PIL
import PIL.Image
import io
import sys

print('Python', sys.version)
print('Pillow', PIL.__version__)
im = PIL.Image.open('lena.png').convert('RGB')  # http://optipng.sourceforge.net/pngtech/img/lena.png
def test():
    byio = io.BytesIO()
    im.save(byio, 'JPEG')
print(timeit.timeit(test, number=10000) / 10, 'ms/iter')

[nulano]

I'm pretty sure the difference is due to the fact that wheels use the libjpeg library, while the distro version most likely uses libjpeg-turbo.

[wangqr]

In that case can we switch to libjpeg-turbo in future PyPI build? People usually use the PyPI package to install a newer version than the distro provided one. And they might be surprised by the significant performance drop.

libjpeg-turbo is also under the IJG license if we only use libjpeg interface. It is long used by major linux distros as default libjpeg implementation, and is faster than libjpeg. It is already in python-pillow/pillow-depends. It is also used in CI build.

[radarhere]

I've created python-pillow/pillow-wheels#237 to resolve this by switching to libjpeg-turbo in the macOS and Linux wheels.