WebP image saving broken from 10.3.0 to 10.4.0

[jax-b]

What did you do?

Installed the latest version of 10.4.0 to my AWS lambda test environment

What did you expect to happen?

No issues opening and saving webp files

What actually happened?

WebP files failed to open without changing to a intermediary BytesIO
After changing to a intermediary BytesIO i still had trouble saving the image

Reverting Pillow to 10.3.0 restores operation

What are your OS, Python and Pillow versions?

• OS: AWS Linux 2
• Python: 3.10.0
• Pillow: 10.4.0

pil features

10.4.0

--------------------------------------------------------------------
Pillow 10.4.0
Python 3.10.14 (main, May 24 2024, 16:11:37) [GCC 7.3.1 20180712 (Red Hat 7.3.1-17)]
--------------------------------------------------------------------
Python executable is /var/lang/bin/python3.10
System Python files loaded from /var/lang
--------------------------------------------------------------------
Python Pillow modules loaded from /opt/python/PIL
Binary Pillow modules loaded from /opt/python/PIL
--------------------------------------------------------------------
--- PIL CORE support ok, compiled for 10.4.0
*** TKINTER support not installed
--- FREETYPE2 support ok, loaded 2.13.2
--- LITTLECMS2 support ok, loaded 2.16
/opt/python/PIL/features.py:42: UserWarning: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /opt/python/PIL/../pillow.libs/libsharpyuv-898c0cb5.so.0.1.0)
warnings.warn(str(ex))
*** WEBP support not installed
/opt/python/PIL/features.py:154: UserWarning: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /opt/python/PIL/../pillow.libs/libsharpyuv-898c0cb5.so.0.1.0)
warnings.warn(str(ex))
*** WEBP Transparency support not installed
*** WEBPMUX support not installed
*** WEBP Animation support not installed
--- JPEG support ok, compiled for libjpeg-turbo 3.0.3
--- OPENJPEG (JPEG2000) support ok, loaded 2.5.2
--- ZLIB (PNG/ZIP) support ok, loaded 1.2.7
--- LIBTIFF support ok, loaded 4.6.0
*** RAQM (Bidirectional Text) support not installed
*** LIBIMAGEQUANT (Quantization method) support not installed
--- XCB (X protocol) support ok
--------------------------------------------------------------------

10.3.0

--------------------------------------------------------------------
Pillow 10.3.0
Python 3.10.14 (main, May 24 2024, 16:11:37) [GCC 7.3.1 20180712 (Red Hat 7.3.1-17)]
--------------------------------------------------------------------
Python executable is /var/lang/bin/python3.10
System Python files loaded from /var/lang
--------------------------------------------------------------------
Python Pillow modules loaded from /opt/python/PIL
Binary Pillow modules loaded from /opt/python/PIL
--------------------------------------------------------------------
--- PIL CORE support ok, compiled for 10.3.0
*** TKINTER support not installed
--- FREETYPE2 support ok, loaded 2.13.2
--- LITTLECMS2 support ok, loaded 2.16
--- WEBP support ok, loaded 1.3.2
--- WEBP Transparency support ok
--- WEBPMUX support ok
--- WEBP Animation support ok
--- JPEG support ok, compiled for libjpeg-turbo 3.0.2
--- OPENJPEG (JPEG2000) support ok, loaded 2.5.2
--- ZLIB (PNG/ZIP) support ok, loaded 1.2.7
--- LIBTIFF support ok, loaded 4.6.0
*** RAQM (Bidirectional Text) support not installed
*** LIBIMAGEQUANT (Quantization method) support not installed
--- XCB (X protocol) support ok
--------------------------------------------------------------------

Test Code

from io import BytesIO
from PIL import UnidentifiedImageError
from PIL.WebPImagePlugin import Image
from fastapi import (
    APIRouter,
    File,
    Path,
    UploadFile,
    FastAPI, 
    Request
)
from typing import Annotated

router = APIRouter()

@router.post("/{id}/images")
async def images_post(
    upload_file: Annotated[UploadFile, File()],
    id: Annotated[
        str,
        Path(),
    ],
) -> str | None:
    try:
        # Load Image
        img = Image.open(BytesIO(await upload_file.read()))
    except UnidentifiedImageError as e:
        raise HTTPException(
            status_code=status.HTTP_400_BAD_REQUEST,
            detail="File uploaded is not a image",
        ) from e
    except Exception as e:
        raise HTTPException(
            status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
            detail="Error on uploading the file (server side)",
        ) from e

    full_size_bytes = BytesIO()
    img.save(full_size_bytes, format="webp")

app = FastAPI()
@app.exception_handler(RequestValidationError)
async def validation_exception_hander(request: Request, exc: Exception) -> dict:
    error_message = f"Unexpected error occurred: {exc}"
    return JSONResponse(status_code=422, content={"detail": error_message})

app.include_router(router)

Traceback

Traceback is from my cloudwatch logs which use Mangum as a intermediary to run the API

 File "/opt/python/mangum/protocols/http.py", line 58, in run
    await app(self.scope, self.receive, self.send)
  File "/opt/python/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/opt/python/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/opt/python/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/opt/python/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/opt/python/starlette/middleware/cors.py", line 93, in __call__
    await self.simple_response(scope, receive, send, request_headers=headers)
  File "/opt/python/starlette/middleware/cors.py", line 148, in simple_response
    await self.app(scope, receive, send)
  File "/opt/python/starlette/middleware/exceptions.py", line 65, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/opt/python/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/opt/python/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/opt/python/starlette/routing.py", line 756, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/opt/python/starlette/routing.py", line 776, in app
    await route.handle(scope, receive, send)
  File "/opt/python/starlette/routing.py", line 297, in handle
    await self.app(scope, receive, send)
  File "/opt/python/starlette/routing.py", line 77, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/opt/python/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/opt/python/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/opt/python/starlette/routing.py", line 72, in app
    response = await func(request)
  File "/opt/python/fastapi/routing.py", line 278, in app
    raw_response = await run_endpoint_function(
  File "/opt/python/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
  File "/var/task/src/source_file.py", line 1590, in images_post
    img.save(full_size_bytes, format="webp")
  File "/opt/python/PIL/Image.py", line 2553, in save
    save_handler = SAVE[format.upper()]

[radarhere]

Hi. Would it be possible for you to put together some example code that only involves Pillow (not fastapi)? Two examples, one showing your opening problem, and the other showing your saving problem.

Also, your traceback does actually show the error itself. What is it?

[jax-b]

i mean its only image open and image save to a BytesIO buffer. This just has the fastapi code around it to give better context.
The traceback show that the it looks like the WebP handler is not registered within Pillow? its throwing a UnidentifiedImageError

[nulano]

Hi. Would it be possible for you to put together some example code that only involves Pillow (not fastapi)? Two examples, one showing your opening problem, and the other showing your saving problem.

Well, it is clear from the report that webp is not installed correctly with Pillow 10.4.0:

/opt/python/PIL/features.py:42: UserWarning: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /opt/python/PIL/../pillow.libs/libsharpyuv-898c0cb5.so.0.1.0)
warnings.warn(str(ex))
*** WEBP support not installed

So the question really is, @jax-b how did you install Pillow 10.4.0?

---

Looking at the filename, it seems you somehow managed to install pillow-10.4.0-cp310-cp310-manylinux_2_28_x86_64.whl on a system with glibc<2.27 (edit: Amazon Linux 2 has glibc 2.26, so clearly older than 2.28), which should not be possible.

$ find . -iname libsharpyuv\*
./pillow-10.4.0-cp310-cp310-manylinux_2_28_x86_64/pillow.libs/libsharpyuv-898c0cb5.so.0.1.0
./pillow-10.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64/pillow.libs/libsharpyuv-aaa00b5c.so.0.1.0

The pillow-10.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl wheel that would be correct for your system does not require glibc 2.27:

$ ldd -v pillow.libs/libsharpyuv-aaa00b5c.so.0.1.0
        linux-vdso.so.1 (0x00007ffe557fe000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x0000792d8318d000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x0000792d83188000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x0000792d82e00000)
        /lib64/ld-linux-x86-64.so.2 (0x0000792d8328e000)

        Version information:
        pillow.libs/libsharpyuv-aaa00b5c.so.0.1.0:
                libpthread.so.0 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libpthread.so.0
                libm.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libm.so.6
                libc.so.6 (GLIBC_2.14) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
        /lib/x86_64-linux-gnu/libm.so.6:
                ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2
                libc.so.6 (GLIBC_2.36) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_ABI_DT_RELR) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.4) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_PRIVATE) => /lib/x86_64-linux-gnu/libc.so.6
        /lib/x86_64-linux-gnu/libpthread.so.0:
                libc.so.6 (GLIBC_2.36) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_ABI_DT_RELR) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
        /lib/x86_64-linux-gnu/libc.so.6:
                ld-linux-x86-64.so.2 (GLIBC_2.2.5) => /lib64/ld-linux-x86-64.so.2
                ld-linux-x86-64.so.2 (GLIBC_2.3) => /lib64/ld-linux-x86-64.so.2
                ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2

[jax-b]

both were installed as part of the serverless python requirements for lambda library. which then zip up the requirements for lambda and uploads it to AWS.
it installs the modules to a custom target directory using a standard pip command

python -m pip install -r requirements.txt -t /tmp/dir

the requirements file is mostly the same just pinned at different versions of pillow

pillow==10.X.0
webp

the opening problem is fixed by the intermediary buffer so I wouldn't worry about that too much (it works between versions).

from PIL.WebPImagePlugin import Image
from io import BytesIO

test_webp: bytes = None
with open("test.webp", "rb") as f:
    test_webp = f.read()

img = Image.open(BytesIO(test_webp))

full_size_bytes = BytesIO()
img.save(full_size_bytes, format="webp")

with open("test_out.webp", "wb") as f:
    f.write(full_size_bytes)

if it works on one version but not another using the same base docker container i dont think that the issue would be the platform? Could how Pillow looks for the GLIBC version have changed or was the minimum version of GLIBC changed between 10.3.0 and 10.4.0.

my build system is WSL Ubuntu 22.04

Linux wsl 5.15.133.1-microsoft-standard-WSL2 #1 SMP Thu Oct 5 21:02:42 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Distributor ID: Ubuntu
Description:    Ubuntu 22.04.4 LTS
Release:        22.04
Codename:       jammy

[jax-b]

Something to note is looking at the wheel that works on Lambda its installing the cp310-cp310-manylinux_2_28_x86_64 version and it has no problems running?

[nulano]

You can try the --platform manylinux_2_17_x86_64 pip parameter to force it to download only compatible wheels as I suggested in #7648 (comment):

python -m pip install --platform manylinux_2_17_x86_64 --only-binary :all: -r requirements.txt -t /tmp/dir

Something to note is looking at the wheel that works on Lambda its installing the cp310-cp310-manylinux_2_28_x86_64 version and it has no problems running?

Indeed, it looks like the version of webp included in the 10.3.0 wheel only needs glibc 2.14:

$ ldd -v pillow-10.3.0-cp310-cp310-manylinux_2_28_x86_64/pillow.libs/libsharpyuv-20f78091.so.0.0.1
        linux-vdso.so.1 (0x00007ffdb2957000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007de977813000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007de97780e000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007de977400000)
        /lib64/ld-linux-x86-64.so.2 (0x00007de977914000)

        Version information:
        pillow-10.3.0-cp310-cp310-manylinux_2_28_x86_64/pillow.libs/libsharpyuv-20f78091.so.0.0.1:
                libpthread.so.0 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libpthread.so.0
                libm.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libm.so.6
                libc.so.6 (GLIBC_2.14) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
        /lib/x86_64-linux-gnu/libm.so.6:
                ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2
                libc.so.6 (GLIBC_2.36) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_ABI_DT_RELR) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.4) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_PRIVATE) => /lib/x86_64-linux-gnu/libc.so.6
        /lib/x86_64-linux-gnu/libpthread.so.0:
                libc.so.6 (GLIBC_2.36) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_ABI_DT_RELR) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
        /lib/x86_64-linux-gnu/libc.so.6:
                ld-linux-x86-64.so.2 (GLIBC_2.2.5) => /lib64/ld-linux-x86-64.so.2
                ld-linux-x86-64.so.2 (GLIBC_2.3) => /lib64/ld-linux-x86-64.so.2
                ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2

Could how Pillow looks for the GLIBC version have changed or was the minimum version of GLIBC changed between 10.3.0 and 10.4.0.

The minimum glibc version has not changed, that remains to be 2.17 for the manylinux_2_17_x86_64 wheels and 2.28 for the manylinux_2_28_x86_64 wheels. However, just because that is the requirement, it does not mean that it definitely won't work on older glibc, just that it is not supported.

The included version of webp was updated from 1.3.2 to 1.4.0 in #7973.
The new version now uses the logf and expf glibc functions, which were updated in glibc 2.27.
As you can see, compiling on a glibc 2.28 system creates a dependency on the new version of these functions:

$ objdump -T pillow-10.3.0-cp310-cp310-manylinux_2_28_x86_64/pillow.libs/libsharpyuv-20f78091.so.0.0.1 | grep -E 'logf|expf'
$ objdump -T pillow-10.4.0-cp310-cp310-manylinux_2_28_x86_64/pillow.libs/libsharpyuv-898c0cb5.so.0.1.0 | grep -E 'logf|expf'
0000000000000000      DF *UND*  0000000000000000 (GLIBC_2.27) expf
0000000000000000      DF *UND*  0000000000000000 (GLIBC_2.27) logf
$ objdump -T pillow-10.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64/pillow.libs/libsharpyuv-aaa00b5c.so.0.1.0 | grep -E 'logf|expf'
0000000000000000      DF *UND*  0000000000000000 (GLIBC_2.2.5) expf
0000000000000000      DF *UND*  0000000000000000 (GLIBC_2.2.5) logf

[radarhere]

If it is helpful, here's a Pillow wheel built with libwebp 1.3.2 - pillow-10.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.zip

[jax-b]

That solves the issue