Real time data augmentation and online batch generation.

[jrabary]

Mxnet looks very promising and I would like to give it a try. I want to train a triplet network like google facenet. One of the key issue to get a succefull training is the selection on a good triplet data to fed to the network. Since generating all possible triplets is not tractable, selecting them during the training is a good compromise. So is it currently possible to add additional steps it the data iterator and in the SGD step such that one can do some statistics on the current batch using the current model in order to select good triplets ? The idea is to transform the source data set with just labeled image into triplet during training. Currently, I use fuel and blocks frameworks and chain different transformer to achieve this. May be Mxnet already have such kind of data pipeline transformation, or can I plug fuel into Mxnet ?

[tqchen]

Can you give a it more detail on what kind of triplet information you would like to have.

• Say get multiple mini-batch of data, run a scoring over current network, and generate the pairs training set?

It is possible, in a sense that all the things after data loading goes into python side, and you can always insert such augmentation step (pairing) on python side to generate the triplets. It requires a bit tweaking on current code.

[jrabary]

Precisely, I generate the batch of triplets as follow :

• start with parent dataset that contains labeled images ( image + class )
• at each step
  • create a batch with an evenly distributed images per class ( say n classes with m images each)
  • from this mini batch, Create a batch of triplets formed with an anchor image, positive image and negative image. The negative images are selected wrt to current network score.
  • update parameters using this batch of triplets.
    with fuel, all of this is done by chaining different data transformer. The SGD optimizer just fetches the batch from the pipeline. Do we have similar feature with Mxnet ?

[tqchen]

The data transforming pipeline on the IO side was not as rich as you mentioned. However, the pipline you mentioned should be able to implemented using the current python API

[jrabary]

How should I get started for that :

• customizing the data iterator ?
• changing the model.fit method ?

[tqchen]

Starting from the getting data part of model.fit should be the easiest way. Customizing data iterator with a pipeline should make the approach more modularized and suitable for future usecases

[jrabary]

After checking a closed look into model.fit method, I notice that it just need a DataIterator object. Then, what is the requirement for an iterator to make it works with model.fit ? Does fuel data stream http://fuel.readthedocs.org/en/latest/overview.html will work out of the box ?

[tqchen]

I think if you write an adapter to implement the DataIter, things might work out of the box