Resize not equivalent between Pillow <9 and Pillow 9.0.0

[NielsRogge]

Hi Pillow folks,

Thanks for creating this awesome library. We're using Pillow to prepare images for several computer vision algorithms in HuggingFace Transformers.

However, we noticed that, after Pillow 9.0.0, some integration tests of our models were failing (models didn't get the exact same output tensors anymore on the same images). Visually there's not really an impact on the models, they perform similar, but it's strange that Pillow's resize method gives different results. Is this expected?

To reproduce:

!pip install -q datasets
from datasets import load_dataset
from PIL import Image

# load image
ds = load_dataset("hf-internal-testing/fixtures_ade20k", split="test")
image = Image.open(ds[0]["file"])

Then, resize as follows:

from PIL import Image

resized_image = image.resize(size=(640, 640), resample=Image.BICUBIC)

(I did this both on Pillow 7.1.2 and Pillow 9.0.1). I save a resized image as follows:

import numpy as np

resized_image_array = np.asarray(resized_image)
np.save("resized_image_7_1_2.npy", resized_image_array)

When comparing the values by turning them into NumPy arrays and using assert allclose:

import numpy as np

resized_image_old = np.load("saved_resized_image_7_1_2.npy")

np.testing.assert_allclose(resized_image_old, resized_image_array)

I'm getting:

AssertionError: 
Not equal to tolerance rtol=1e-07, atol=0

Mismatched elements: 261534 / 1228800 (21.3%)
Max absolute difference: 255
Max relative difference: 255.
 x: array([[[ 86, 138, 186],
        [ 85, 137, 185],
        [ 85, 137, 185],...
 y: array([[[ 86, 138, 186],
        [ 85, 137, 185],
        [ 85, 137, 187],...

[radarhere]

Hi. I don't quite follow your example - I don't see where "saved_resized_image_7_1_2.npy", resized_image_old or resized_image_array are created. Would you be able to clarify?

Also, what version of NumPy are you using?

[NielsRogge]

I've updated the code snippet above to include the saving of the resized image as a NumPy array.

This is all done in Google Colab. Numpy version is 1.21.5.

[radarhere]

On my local machine, I don't find a problem when compiling from source.

Testing in Google Colab, I was able to reproduce this - but only when I didn't compile from source.

So it's not a change in Pillow's code itself, but presumably one of the dependencies in our wheels. The image that you're specifying is a JPEG, so my guess would be that this is because we switched to libjpeg-turbo in our wheels.

[radarhere]

@NielsRogge did that answer your question?

[NielsRogge]

Yes, visually the differences are minor, so I'm going to update the expected outputs of our models. Thanks for looking into it!