Performance when reading TIFF file with small tiles

[zzjjbb]

What did you do?

I use the method Image.open(tiff_filename).load() to read a tiff file from a SMB server.

What did you expect to happen?

25MB data is read from the server during reading.

What actually happened?

It is very slow and downloads hundreds megabytes of data.
Currently, I change to reading the content into memory first and adding a BytesIO wrapper.
raw = f.read(); Image.open(BytesIO(raw)).load()
This can gives me the same result and the speed and network traffic is expected.

My findings

The tiff file is a ~5000x5000 8bit image with no compression. The library first loads the header and gets the data of ~5000 tiles. As the offsets in the tiles list show, each tile has 5KB size. However, pillow reads a much larger size for each tile, which is a const MAXBLOCK = 65536 (bytes) = ~65KB.

Pillow/src/PIL/ImageFile.py

Line 280 in 76919ad

s = read(self.decodermaxblock)

What are your OS, Python and Pillow versions?

• OS: Windows
• Python: 3.12.7
• Pillow: 10.2.0

--------------------------------------------------------------------
Pillow 10.2.0
Python 3.12.7 | packaged by Anaconda, Inc. | (main, Oct  4 2024, 13:17:27) [MSC v.1929 64 bit (AMD64)]
--------------------------------------------------------------------
Python modules loaded from ......\site-packages\PIL
Binary modules loaded from ......\site-packages\PIL
--------------------------------------------------------------------
--- PIL CORE support ok, compiled for 10.2.0
--- TKINTER support ok, loaded 8.6
--- FREETYPE2 support ok, loaded 2.13.2
--- LITTLECMS2 support ok, loaded 2.16
--- WEBP support ok, loaded 1.3.2
--- WEBP Transparency support ok
--- WEBPMUX support ok
--- WEBP Animation support ok
--- JPEG support ok, compiled for libjpeg-turbo 3.0.1
--- OPENJPEG (JPEG2000) support ok, loaded 2.5.0
--- ZLIB (PNG/ZIP) support ok, loaded 1.3
--- LIBTIFF support ok, loaded 4.6.0
*** RAQM (Bidirectional Text) support not installed
*** LIBIMAGEQUANT (Quantization method) support not installed
*** XCB (X protocol) support not installed
--------------------------------------------------------------------
BLP
Extensions: .blp
Features: open, save, encode
--------------------------------------------------------------------
BMP image/bmp
Extensions: .bmp
Features: open, save
--------------------------------------------------------------------
BUFR
Extensions: .bufr
Features: open, save
--------------------------------------------------------------------
CUR
Extensions: .cur
Features: open
--------------------------------------------------------------------
DCX
Extensions: .dcx
Features: open
--------------------------------------------------------------------
DDS
Extensions: .dds
Features: open, save
--------------------------------------------------------------------
DIB image/bmp
Extensions: .dib
Features: open, save
--------------------------------------------------------------------
EPS application/postscript
Extensions: .eps, .ps
Features: open, save
--------------------------------------------------------------------
FITS
Extensions: .fit, .fits
Features: open
--------------------------------------------------------------------
FLI
Extensions: .flc, .fli
Features: open
--------------------------------------------------------------------
FTEX
Extensions: .ftc, .ftu
Features: open
--------------------------------------------------------------------
GBR
Extensions: .gbr
Features: open
--------------------------------------------------------------------
GIF image/gif
Extensions: .gif
Features: open, save, save_all
--------------------------------------------------------------------
GRIB
Extensions: .grib
Features: open, save
--------------------------------------------------------------------
HDF5
Extensions: .h5, .hdf
Features: open, save
--------------------------------------------------------------------
ICNS image/icns
Extensions: .icns
Features: open, save
--------------------------------------------------------------------
ICO image/x-icon
Extensions: .ico
Features: open, save
--------------------------------------------------------------------
IM
Extensions: .im
Features: open, save
--------------------------------------------------------------------
IMT
Features: open
--------------------------------------------------------------------
IPTC
Extensions: .iim
Features: open
--------------------------------------------------------------------
JPEG image/jpeg
Extensions: .jfif, .jpe, .jpeg, .jpg
Features: open, save
--------------------------------------------------------------------
JPEG2000 image/jp2
Extensions: .j2c, .j2k, .jp2, .jpc, .jpf, .jpx
Features: open, save
--------------------------------------------------------------------
MCIDAS
Features: open
--------------------------------------------------------------------
MPEG video/mpeg
Extensions: .mpeg, .mpg
Features: open
--------------------------------------------------------------------
MSP
Extensions: .msp
Features: open, save, decode
--------------------------------------------------------------------
PCD
Extensions: .pcd
Features: open
--------------------------------------------------------------------
PCX image/x-pcx
Extensions: .pcx
Features: open, save
--------------------------------------------------------------------
PIXAR
Extensions: .pxr
Features: open
--------------------------------------------------------------------
PNG image/png
Extensions: .apng, .png
Features: open, save, save_all
--------------------------------------------------------------------
PPM image/x-portable-anymap
Extensions: .pbm, .pgm, .pnm, .ppm
Features: open, save
--------------------------------------------------------------------
PSD image/vnd.adobe.photoshop
Extensions: .psd
Features: open
--------------------------------------------------------------------
QOI
Extensions: .qoi
Features: open
--------------------------------------------------------------------
SGI image/sgi
Extensions: .bw, .rgb, .rgba, .sgi
Features: open, save
--------------------------------------------------------------------
SPIDER
Features: open, save
--------------------------------------------------------------------
SUN
Extensions: .ras
Features: open
--------------------------------------------------------------------
TGA image/x-tga
Extensions: .icb, .tga, .vda, .vst
Features: open, save
--------------------------------------------------------------------
TIFF image/tiff
Extensions: .tif, .tiff
Features: open, save, save_all
--------------------------------------------------------------------
WEBP image/webp
Extensions: .webp
Features: open, save, save_all
--------------------------------------------------------------------
WMF
Extensions: .emf, .wmf
Features: open, save
--------------------------------------------------------------------
XBM image/xbm
Extensions: .xbm
Features: open, save
--------------------------------------------------------------------
XPM image/xpm
Extensions: .xpm
Features: open
--------------------------------------------------------------------
XVTHUMB
Features: open
--------------------------------------------------------------------

[radarhere]

Are you able to provide a copy of the file?

[zzjjbb]

Are you able to provide a copy of the file?

Thanks for the reminder! I should have it uploaded at first...
IMG07.zip
These tiff files are the output of some industrial camera and the one I upload here is simply some noise. I need to zip it since GitHub do not like tiff files in comment.

[radarhere]

Thanks.

So, you're aware that you could work around this by reading the file into memory first.
And you have probably also realised that you could

from PIL import ImageFile
ImageFile.MAXBLOCK = 1024

to reduce the amount of data read in each block.

But instead, you're hoping that Pillow by default will perform smaller read operations.

We could make an assumption that the data for each tile will stop by the start of the next tile. Does #8609 solve your problem?

[radarhere]

Pillow 11.2.1 has now been released with the fix for this.