Add support for high bit depth multichannel images

[wiredfool]

Pillow (and PIL) is currently able to open 8 bit per channel multi-channel images (such as RGB) but is able to open higher bit depth images (e.g. I16, I32, or Float32 images) if they are single channel (e.g., grayscale).

Previous References

This has been requested many times: #1828, #1885, #1839, #1602, and farther back.

Requirements

• We should be able to support common GIS formats as well as high bit depth RGB(A) images.
• At least 4 channels, but potentially more (see Add tests for opening 2-5 layer uint16 greyscale TIFFs #1839)
• Different pixel formats, including I16, I32, and Float.
• There should be definitions for the array interface to exchange images with numpy/scipy
• There should be enough support to read and write TIFFs and raw image data.
• Support for resize, crop, and convert operations at the very least.

Background Reference Info

The rough sequence for image loading is:

• Image file is opened

• Each of the ImagePlugin _accept functions have a chance to look at the first few bytes to determine if they should attempt to open the file

• The *ImagePlugin._open method is called giving the image plugin a chance to read more of the image and determine if it still wants to consider it a valid image of it's particular type. If it does, it passes back a tile definition which includes a decoder and an image size.

• If there is a successful _open call, at some point later *ImagePlugin._load may be called on the image, which runs the decoder producing a set of bytes in a raw mode. This is where things like compression are handled, but the output of the decoder is not necessarily what we're storing in our internal structures.

• The image is unpacked (Unpack.c) from the raw mode (e.g. I16;BS) into a storage (Storage.c) mode (I).

• It's now possible to operate on the image (e.g. crop, pixel access, etc)

  There are 3 (or 4) image data pointers, as defined in Imaging.h:

struct ImagingMemoryInstance {

    /* Format */
    char mode[IMAGING_MODE_LENGTH]; /* Band names ("1", "L", "P", "RGB", "RGBA", "CMYK", "YCbCr", "BGR;xy") */
    int type;       /* Data type (IMAGING_TYPE_*) */
    int depth;      /* Depth (ignored in this version) */
    int bands;      /* Number of bands (1, 2, 3, or 4) */
    int xsize;      /* Image dimension. */
    int ysize;

    /* Colour palette (for "P" images only) */
    ImagingPalette palette;

    /* Data pointers */
    UINT8 **image8; /* Set for 8-bit images (pixelsize=1). */
    INT32 **image32;    /* Set for 32-bit images (pixelsize=4). */

    /* Internals */
    char **image;   /* Actual raster data. */
    char *block;    /* Set if data is allocated in a single block. */

    int pixelsize;  /* Size of a pixel, in bytes (1, 2 or 4) */
    int linesize;   /* Size of a line, in bytes (xsize * pixelsize) */

    /* Virtual methods */
    void (*destroy)(Imaging im);
};

The only one that is guaranteed to be set is **image, which is an array of pointers to row data.

Changes Required

• Definitions for all of the modes that we're planning, and potentially a [format];MB[#bands] style generic mode.

Core Imaging Structure

• The imaging structure has the fields required to add the additional channels. (type, bands, pixelsize, linesize)
• The **image pointer can be used for any width of pixel.
• We may or may not want to set the **image32 pointer.
• Currently type of IMAGING_TYPE_INT32 and IMAGING_TYPE_FLOAT32 imply 1 band. This will change.
• Consider promoting int16 to IMAGING_TYPE_INT16

Storage

• Updates to Storage.c, Unpack.c, Pack.c, Access.c, PyAccess.py, and Convert.c

Ways to Help

We need a better definition of the format requirements. What are the various types of images that are used in GIS, Medical, or other fields that we'd want to interpret? We need small, redistributable versions of images that we can test against.

[in progress]

[terramars]

I'm having the same problem with 16 bit single-channel paletted TIFFs, created by GDAL. It would be "really" nice if Pillow could play nicely with GIS and scientific image formats, as GDAL is a pain in the ass and I'd rather not use it.

tiffinfo as follows:

TIFFReadDirectory: Warning, Unknown field with tag 33550 (0x830e) encountered.
TIFFReadDirectory: Warning, Unknown field with tag 33922 (0x8482) encountered.
TIFFReadDirectory: Warning, Unknown field with tag 34735 (0x87af) encountered.
TIFFReadDirectory: Warning, Unknown field with tag 34737 (0x87b1) encountered.
TIFFReadDirectory: Warning, Unknown field with tag 42113 (0xa481) encountered.
TIFF Directory at offset 0x34293c6 (54694854)
Image Width: 10774 Image Length: 12577
Bits/Sample: 16
Sample Format: unsigned integer
Compression Scheme: LZW
Photometric Interpretation: palette color (RGB from colormap)
Samples/Pixel: 1
Rows/Strip: 1
Planar Configuration: single image plane
Color Map: (present)
Tag 33550: 4.999617,4.999789,0.000000
Tag 33922: 0.000000,0.000000,0.000000,679006.067110,9955209.915048,0.000000
Tag 34735: 1,1,0,7,1024,0,1,1,1025,0,1,1,1026,34737,22,0,2049,34737,7,22,2054,0,1,9102,3072,0,1,32736,3076,0,1,9001
Tag 34737: WGS 84 / UTM zone 36S|WGS 84|
Tag 42113: 0
Predictor: horizontal differencing 2 (0x2)

[bodokaiser]

Any updates on this?

[wiredfool]

Unfortunately, no.

[vfdev-5]

@wiredfool what do you think about to add the support of multichannel images as sequence of Image ? For example, 4 channels image with uint16 is represented (more less equivalently) by
['<PIL.Image.Image image mode=I;16 size=... >', '<PIL.Image.Image image mode=I;16 size=...>', ..., '<PIL.Image.Image image mode=I;16 size=...>']. I mean by that, maybe, to provide a class inheriting from Image and tuple and override all method to work on a tuple of images... Sure that it looks like a hack, however it could unlock more features (and create issues :) ) at least while working with Image.fromarray.

[wiredfool]

To do anything useful with it, we'd have to have support in the C layer, so it would have to be at the core imaging layer, and especially Unpack/Pack.

[vfdev-5]

@wiredfool following your "Ways to help",

We need a better definition of the format requirements. What are the various types of images that are used in GIS, Medical, or other fields that we'd want to interpret?

For GIS, as there is a huge amount of different formats (for example, gdal format list), this can be left for GIS libraries as gdal, rasterio etc.
However, a support of Image.fromarray on input multi-channel (3,4,5,...) arrays of dtype np.uint16, np.float32 would be, imho, essential.

We need small, redistributable versions of images that we can test against.

For GIS imagery, this can be easily created manually with gdal, rasterio.

I would like to give a hand on this, so, feel free to ask me.