Add support for multiple labels to be specified in the image_data_layer

[elezar]

Although there are a number of pull requests pending where mulit-label support has been discussed, many of them seem to have been abandoned.

For reference for example: #523 #3268

The latter may still be active (last updated in December 2015), but they also deal with the loss or accuracy layers, and not with the data layers themselves.

This pull requests adds the ability to specify multiple labels in a text file directly e.g.:

images/cat.jpg 0
images/fish-bike.jpg 1 9

instead of having to maintain separate labels and data sets.

It is also possible to specify a list of labels to ignore following the labels per item:

images/cat.jpg 0;9
images/fish-bike.jpg 1 9

Ignoring a label equates to setting the entries in the label top blob to some predefined value (-1). In the same way that multiple labels could be specified, multiple ignore labels could also be specified as a list. Here a semi-colon (;) has been chosen as the label and ignore list separator.

The following points should be noted:

• If a single label is specified per line in the input file, the current behavior is maintained. That is to say that the label top for the layer is an (N,1) dimensional blob, with the entries equal to the label of the image.
• If multiple labels (including ignore labels) are specified per line, the label top layer is an (N, K) dimensional blob, with K=max_label + 1. In this case the entries in the blob are zero if the label is not present in the line, or 1 if this is the case. max_label is determined when reading the file for the first time.

Some extensions that I would like to add:

• For this to be useful, the loss and accuracy layers should also be extended. The best would most likely be to work in conjunction with other pull requests such as multi-label softmax support #3268 in this regard. @xdshang, any thoughts on this? What other accuracy or loss layers make sense to extend to the multi-label case? The sigmoid cross entropy loss, for example.
• The separators (between individual labels) should be flexible. Files may be comma-separated, or space separated, for example.

I would also like to look at adapting #3471 to use the new data layer.

Thoughts and comments would be welcome.

Note, I had to add std::endl outputs to the unit tests. I assume that this makes more sense in any case.

[shenyunhang]

This is exactly what I need. Have you test it on PASCAL VOC dataset?

[elezar]

I have, and have been working on a simple tutorial. I will push it soon.

[elezar]

@shenyunhang Please see the latest push. I have included an example that uses the PASCAL VOC dataset with the multi-label image data layer. The example can be run as both an iPython notebook, from the script.

The format of the iPython notebook is "borrowed" from @beijbom's PR #3471 (I will update it further once that PR has been accepted and merged).

[shenyunhang]

@elezar Thank you very much. I am trying on it.
However, I have some data with the following format:

filename ;ignore1 ignore2

Here I can't make sure what labels are associated with the data line.
It seem image_data_layer can't work with such format. But it is OK. I will modify some places to make it work for me.

[elezar]

@shenyunhang I will look at adding a unit test for your example you tomorrow. There should not be a specific reason that this does not work, but I may have missed something in the implementation. You are also welcome to submit a pull request to the feature branch at zalando/caffe is you get it working before then.

[elezar]

@shenyunhang were you able to get the layer working for you? If so, do you mind sharing your changes with me?

[elezar]

@shenyunhang I have updated the image data layer to support ignore labels when no labels have been specified.

[elezar]

@beijbom Since you worked on the Python implementation of the example, would you mind looking at notebook that I have included in this PR? @shelhamer This also contains a prototxt version of the network which could be used to address some of the TODOs mentioned in #3471 (comment)

[beijbom]

hey @elezar , I think the notebook could use some more explanation & comments, but it's getting there! :). It will be very useful to have multi-label support in the image data-layer, as this is a very common user-case, and the python data-layers are slow for production stuff.

cheers

[elezar]

@beijbom Thanks. I have tried to improve the notebook (once again using your notebook as reference) and pushed the changes.

[elezar]

@shelhamer could you perhaps have a look at this PR and let me know what work is still required?

[wuhang]

@elezar
I'm a new caffe, how do i classification new image by multi lable, can give me a complete classification of examples? thanks.

[elezar]

@wuhang I think a good place to start is @beijbom's tutorial. There is an iPython notebook included in master.

[elezar]

Closing this PR as stale.