Pillow 11.1.0 not reading JPEG comment

[ssokolow]

What did you do?

Loaded the attached image using the provided code.

What did you expect to happen?

As documented in the Image file formats page, there should be a comment field in the info dict containing the same data as this exiftool output:

User Comment                    : {"prompt": "a photograph of an astronaut riding a horse", "negative_prompt": "", "seed": 0, "use_stable_diffusion_model": "sd-v1-5", "clip_skip": false, "use_controlnet_model": null, "control_alpha": null, "use_vae_model": "", "sampler_name": "euler_a", "width": 512, "height": 768, "num_inference_steps": 25, "guidance_scale": 7.5, "use_lora_model": null, "use_embeddings_model": null, "tiling": null, "use_face_correction": null, "use_upscale": null}

What actually happened?

The comment field is missing.

What are your OS, Python and Pillow versions?

• OS: Kubuntu Linux 22.04 LTS
• Python: 3.10.12
• Pillow: 11.1.0

--------------------------------------------------------------------
Pillow 11.1.0
Python 3.10.12 (main, Feb  4 2025, 14:57:36) [GCC 11.4.0]
--------------------------------------------------------------------
Python executable is [REDACTED]/.venv/bin/python3
Environment Python files loaded from [REDACTED]/.venv
System Python files loaded from /usr
--------------------------------------------------------------------
Python Pillow modules loaded from [REDACTED]/.venv/lib/python3.10/site-packages/PIL
Binary Pillow modules loaded from [REDACTED]/.venv/lib/python3.10/site-packages/PIL
--------------------------------------------------------------------
--- PIL CORE support ok, compiled for 11.1.0
--- TKINTER support ok, loaded 8.6
--- FREETYPE2 support ok, loaded 2.13.2
--- LITTLECMS2 support ok, loaded 2.16
--- WEBP support ok, loaded 1.5.0
--- JPEG support ok, compiled for libjpeg-turbo 3.1.0
--- OPENJPEG (JPEG2000) support ok, loaded 2.5.3
--- ZLIB (PNG/ZIP) support ok, loaded 1.2.11, compiled for zlib-ng 2.2.2
--- LIBTIFF support ok, loaded 4.6.0
--- RAQM (Bidirectional Text) support ok, loaded 0.10.1, fribidi 1.0.8, harfbuzz 10.1.0
*** LIBIMAGEQUANT (Quantization method) support not installed
--- XCB (X protocol) support ok
--------------------------------------------------------------------

from PIL import Image
im = Image.open('a_photograph_of_an_astronaut_riding_a_horse_4MZPQQN0.jpeg')
print(im.info)

[radarhere]

Hi. JpegImagePlugin's info["comment"] retrieves data from the COM marker. See https://en.wikipedia.org/wiki/JPEG#cite_ref-51

Instead, the comment in your image is actually contained within the EXIF data, under the UserComment tag. See https://web.archive.org/web/20240329145259/https://www.awaresystems.be/imaging/tiff/tifftags/privateifd/exif/usercomment.html

This should print what you're after.

from PIL import Image, ExifTags
im = Image.open('a_photograph_of_an_astronaut_riding_a_horse_4MZPQQN0.jpeg')
exif = im.getexif()
ifd = exif.get_ifd(ExifTags.IFD.Exif)
comment = ifd[ExifTags.Base.UserComment]
print(comment.decode())

[ssokolow]

1. Perhaps there should be a note in the docs to that effect.
2. Pydantic rejects the resulting JSON because, for some reason, comment.decode() prepends UNICODE to the output.

EDIT: Never mind. I'm just converting it wrong. repr() shows there are also a bunch of nulls, so it's clear I need to dir() around.

This is the repr() of what comment.decode() is producing:

'UNICODE\x00\x00{\x00"\x00p\x00r\x00o\x00m\x00p\x00t\x00"\x00:\x00 \x00"\x00a\x00 \x00p\x00h\x00o\x00t\x00o\x00g\x00r\x00a\x00p\x00h\x00 \x00o\x00f\x00 \x00a\x00n\x00 \x00a\x00s\x00t\x00r\x00o\x00n\x00a\x00u\x00t\x00 \x00r\x00i\x00d\x00i\x00n\x00g\x00 \x00a\x00 \x00h\x00o\x00r\x00s\x00e\x00"\x00,\x00 \x00"\x00n\x00e\x00g\x00a\x00t\x00i\x00v\x00e\x00_\x00p\x00r\x00o\x00m\x00p\x00t\x00"\x00:\x00 \x00"\x00"\x00,\x00 \x00"\x00s\x00e\x00e\x00d\x00"\x00:\x00 \x000\x00,\x00 \x00"\x00u\x00s\x00e\x00_\x00s\x00t\x00a\x00b\x00l\x00e\x00_\x00d\x00i\x00f\x00f\x00u\x00s\x00i\x00o\x00n\x00_\x00m\x00o\x00d\x00e\x00l\x00"\x00:\x00 \x00"\x00s\x00d\x00-\x00v\x001\x00-\x005\x00"\x00,\x00 \x00"\x00c\x00l\x00i\x00p\x00_\x00s\x00k\x00i\x00p\x00"\x00:\x00 \x00f\x00a\x00l\x00s\x00e\x00,\x00 \x00"\x00u\x00s\x00e\x00_\x00c\x00o\x00n\x00t\x00r\x00o\x00l\x00n\x00e\x00t\x00_\x00m\x00o\x00d\x00e\x00l\x00"\x00:\x00 \x00n\x00u\x00l\x00l\x00,\x00 \x00"\x00c\x00o\x00n\x00t\x00r\x00o\x00l\x00_\x00a\x00l\x00p\x00h\x00a\x00"\x00:\x00 \x00n\x00u\x00l\x00l\x00,\x00 \x00"\x00u\x00s\x00e\x00_\x00v\x00a\x00e\x00_\x00m\x00o\x00d\x00e\x00l\x00"\x00:\x00 \x00"\x00"\x00,\x00 \x00"\x00s\x00a\x00m\x00p\x00l\x00e\x00r\x00_\x00n\x00a\x00m\x00e\x00"\x00:\x00 \x00"\x00e\x00u\x00l\x00e\x00r\x00_\x00a\x00"\x00,\x00 \x00"\x00w\x00i\x00d\x00t\x00h\x00"\x00:\x00 \x005\x001\x002\x00,\x00 \x00"\x00h\x00e\x00i\x00g\x00h\x00t\x00"\x00:\x00 \x007\x006\x008\x00,\x00 \x00"\x00n\x00u\x00m\x00_\x00i\x00n\x00f\x00e\x00r\x00e\x00n\x00c\x00e\x00_\x00s\x00t\x00e\x00p\x00s\x00"\x00:\x00 \x002\x005\x00,\x00 \x00"\x00g\x00u\x00i\x00d\x00a\x00n\x00c\x00e\x00_\x00s\x00c\x00a\x00l\x00e\x00"\x00:\x00 \x007\x00.\x005\x00,\x00 \x00"\x00u\x00s\x00e\x00_\x00l\x00o\x00r\x00a\x00_\x00m\x00o\x00d\x00e\x00l\x00"\x00:\x00 \x00n\x00u\x00l\x00l\x00,\x00 \x00"\x00u\x00s\x00e\x00_\x00e\x00m\x00b\x00e\x00d\x00d\x00i\x00n\x00g\x00s\x00_\x00m\x00o\x00d\x00e\x00l\x00"\x00:\x00 \x00n\x00u\x00l\x00l\x00,\x00 \x00"\x00t\x00i\x00l\x00i\x00n\x00g\x00"\x00:\x00 \x00n\x00u\x00l\x00l\x00,\x00 \x00"\x00u\x00s\x00e\x00_\x00f\x00a\x00c\x00e\x00_\x00c\x00o\x00r\x00r\x00e\x00c\x00t\x00i\x00o\x00n\x00"\x00:\x00 \x00n\x00u\x00l\x00l\x00,\x00 \x00"\x00u\x00s\x00e\x00_\x00u\x00p\x00s\x00c\x00a\x00l\x00e\x00"\x00:\x00 \x00n\x00u\x00l\x00l\x00}'

[radarhere]

If you're concerned about \x00, then this

comment = ifd[ExifTags.Base.UserComment]
print(repr(comment.replace(b"\x00", b"").decode()))

will give

'UNICODE{"prompt": "a photograph of an astronaut riding a horse", "negative_prompt": "", "seed": 0, "use_stable_diffusion_model": "sd-v1-5", "clip_skip": false, "use_controlnet_model": null, "control_alpha": null, "use_vae_model": "", "sampler_name": "euler_a", "width": 512, "height": 768, "num_inference_steps": 25, "guidance_scale": 7.5, "use_lora_model": null, "use_embeddings_model": null, "tiling": null, "use_face_correction": null, "use_upscale": null}'

[ssokolow]

I'm concerned about receiving a mysteriously incompletely decoded blob of not-quite-JSON where I don't want to be responsible for anticipating all other possible forms it might take in edge cases in the wild.

The tool I'm writing is explicitly intended as a replacement for another tool which I keep running into papercut bugs in, and, at minimum, it would be a bad look if my own creation had papercut bugs because I chose the hacky solution.

If there's no .actually_decode() that I can call to foist getting it right off on Pillow, I might as well add exifread to my dependencies alongside Pillow, since I know it Just Works and gets it right for this use-case.

EDIT: Actually, now I'm wondering if it's not my fault but Pillow's that I'm having so much trouble figuing out which Pydantic method I need to call to ingest the form Easy Diffusion stores embedded prompt data in when using PNG instead of JPEG. Time for more repr().

[radarhere]

I've created #8788 to update the documentation to read

A comment about the image, from the COM marker. This is separate from the UserComment tag that may be stored in the EXIF data.

Let us know if there is anything specific you would like further assistance with.

[ssokolow]

That works for the documentation.

I think the only other thing I'd like help with is understanding if that more or less undocumented .get_ifd() approach is a viable solution or not. (i.e. Whether there's a way to ask Pillow to not stop at what appears at first glance to be some kind of of binary blob consisting of a format-identifying ASCII prefix (UNICODE) followed by UTF-16 and decode all the way to a Python unicode string for me.)

(By more or less undocumented, I mean that it's apparently a young enough method for DuckDuckGo to not have picked it up yet, and Google takes me to docs where I slam face-first into an Any return type with no description.)

[radarhere]

As far as Pillow's documentation goes, I can offer

• https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.Exif to explain the use of get_ifd()
• https://pillow.readthedocs.io/en/stable/reference/ExifTags.html to explain the nature of ExifTags
• https://pillow.readthedocs.io/en/stable/_modules/PIL/ExifTags.html#Base to list the ExifTags

If you're talking about get_ifd(tag: int) → dict[int, Any], yes, get_ifd() returns a dictionary mapping to integers to... anything, but to be fair, there are reasonable number of Exif tags, and there can even be custom Exif tags that may not be documented anywhere online.

[ssokolow]

...so is my first impression correct that, for ExifTags.Base.UserComment, the current situation with Pillow is "Here's the binary blob it's storing. You're on your own beyond that."?

...because there's even a Stack Overflow question that never received an answer from 4 years ago asking how to decode it with a reply that starts "Did you find anything? My quick search led to this: According to JEITA CP-3451, UserComment is rather special thing with a hairy character encoding hack in the first 8 bytes."

Not the kind of thing I'm willing to take responsibility for maintaining and getting right across all possible values when people apparently can't even link to the spec it's defined in. I have enough experience with that kind of self-describing any-codepage encoding nonsense with various standardized things stored in X11 XAtoms.

[radarhere]

Perhaps another way of looking at it would be that Pillow is providing the exact data that was requested. I think some users do want to access that kind of raw information.

First, I'm not sure where a specific method for parsing a single EXIF tag should live in our API. I know the obvious answer might be to just store it in info["comment"]. Maybe.
Second, if there isn't any documentation available for the way that encoders write data to the UserComment tag, it's not clear how we could write Pillow code to parse that.

[ssokolow]

Perhaps another way of looking at it would be that Pillow is providing the exact data that was requested. I think some users do want to access that kind of raw information.

True... but, in my case, what I'm looking for is an alternative to having to depend on both Pillow and exifread, just so I can extract the JSON from both PNG and JPEG files without having to write my own decoder.

Second, if there isn't any documentation available for the way that encoders write data to the UserComment tag, it's not clear how we could write Pillow code to parse that.

It's less that it's undocumented (here you go) and more that I don't want to maintain an implementation of that sort of thing. (I've been a little once-burned, twice-shy ever since running into X11's Compound Text Encoding... thankfully, it seems so dead that even X.org utilities just assume it'll be a single blob of latin1 these days.)

[ssokolow]

Ugh. My own testing shows that exifread is broken in the presence of non-ASCII text and a quick look at their source suggests it blows up when it encounters things like CJK text because they implemented the comment.replace(b"\x00", b"") you suggested, causing their UTF-8 decoder to barf when encountering a subsequence of UTF-16 low bytes from codepoints beyond the 0-127 ASCII compatibility range.

I guess I have no choice but to implement the UNICODE variant of EXIF UserComment on top of Pillow, commit a bunch of test fixtures, and hope there are no edge cases I'm missing where Easy Diffusion can be induced to emit the ASCII\0\0\0, JIS\0\0\0\0\0, or Undefined (all NULLs) variants. (Undefined is literally spec'd as "use the system codepage and hope for the best".)

(Which is annoying, because I've already discovered it apparently contains three independent JSON serializers with different quirks for the three different ways to ask it to dump the prompting data to .json files. (Autosave, single image download, and download all.) If it's also doing that for embedding into .jpeg, .png, and .webp, that'll be a combinatorial explosion of unit tests.)

EDIT: It looks like the equally hacky implementation that works as a replacement for comment.replace(b"\x00", b"").decode() when the encoding tag is UNICODE\0 is comment[8:].decode('utf-16be'). At least, it passes Pydantic validation and the correct text is in the fields where I put some CJK characters.

[radarhere]

#8788 has now been merged to update the documentation.

It sounds like you've found your way to an answer. Is there anything further?

[ssokolow]

Nothing on Pillow's end. Thanks. :)

(I do, however, have some choice profanities for whoever designed Easy Diffusion's encoding and structuring scheme for serializing metadata into PNG files.)