pycaffe preprocessing incorrect for certain options in master (fixed in dev)

[shelhamer]

In master the pycaffe preprocessing for deployment nets -- models that use input: fields instead of data layers -- is incorrect for certain combinations of input scaling and means.

• pycaffe scales the input before mean subtraction, instead of after
• pycaffe represents images as floats in [0, 1], while some Caffe models represent images in [0, 255] -- most notably the ImageNet models

However when the input scale in master is set to 255 this reproduces the correct behavior for the ImageNet models.

This bug does not affect models that use data layers as the input preprocessing is handled by ForwardPrefilled() and the data layers instead of pycaffe's Net.preprocess().

Input preprocessing in Python is fixed and merged to dev in #816. A new stable release is on the way, but in the meantime pycaffe users working with deployment nets should try dev to ensure correct model output.

Sorry for the bug! Thanks to @Denominator for a report.

[zayd]

Great to see this fix, was just about to submit a PR!

img_as_float in load_image() io.py rescales the image to [0, 1]. This necessitates the raw_scale parameter which complicates things. Should input: just try to replicate the data layers? (i.e. just load, mean subtract and scale the image?)

[shelhamer]

Closing since the fix is released to master, but the discussion is still open.

We decided to adopt the raw_scale parameter since the Python convention is to represent images as [0, 1] floats or [0, 255] uint8. Since uint8 lacks precision, we chose the [0, 1] float to match expectation. While we could have picked loading images into [0, 255] float to match the ImageNet model data layer configuration, that is only the configuration of that particular model (as a side effect of OpenCV image loading). For instance, LeNet runs on [0, 1] intensity images. DataLayer just loads whatever the leveldb/lmdb was populated with, and there isn't necessarily any one agreed upon data scale for images.

Since the Python interface is meant to work for all inputs, even non-image data, we thought having a raw_scale field and respecting the Python convention for images otherwise was the best approach.

If you see a simpler way that still supports the full diversity of inputs properly, chime in or submit a PR.

As a last comment, don't forget that the Python and MATLAB input preprocessing really shouldn't exist at all. Ideally, all the data transformation would be defined in the model prototxt and carried out by data layers, including a InputDataLayer that works like the current input fields do as a placeholder for data loaded online.