Incompatibilities in BatchNorm.

[mathmanu]

BatchNorm in NVIDIA/caffe is not compatible with BatchNorm in BVLC/caffe.

There is also no compatibility b/w engine:CAFFE and engine:CUDNN BatchNorm in NVIDIA/caffe itself. (Blob shapes are different).

Kindly fix these issues - so that we can use pre-trained models for fine tuning.

Please refer to:
NVIDIA/DIGITS#629
and
BVLC#3919
as well where similar issues are discussed.

I have some suggestions to fix these issues:

1. Rename the NVIDIA/caffe's BatchNorm to BatchNormScale, since it now includes Scaling as well.

2. Put a check/exit in CUDNN BatchNormScale reshape function, if the top and bottom blobs are same - so that the user will get a warning.

3. Fix the inconsistency in blob shape between engine:CAFFE and engine:CUDNN

4. Currenty I have to specify so many parameters in the new BatchNorm layer. Thi is un-necessary.

layer {
  name: "bn_conv1"
  bottom: "conv1"
  top: "conv1"
  type: "BatchNorm"
  param { #scale
    lr_mult: 1
    decay_mult: 1
  }
  param { #shift/bias
    lr_mult: 1
    decay_mult: 1
  } 
  param { #global mean
    lr_mult: 0
    decay_mult: 0
  }
  param { #global var
    lr_mult: 0
    decay_mult: 0
  }

  batch_norm_param {
    scale_filler {
    type: "constant"
    value: 1
    }
    bias_filler {
      type: "constant"
      value: 0
    }
    engine: CUDNN
  }
}

(4a). In BatchNormScale, If you change the oder of the blobs to: gloabl_mean, and global_variance, scale, bias, global_counter, then I don't have to specify 4 param fields for lr_mult and decay_mult - but only 2.

(4b). If the definition of scale and bias fields in BatchNormParameter is changed to:
optional float scale_filler = 5 [default = 1];
optional float bias_filler = 6 [default = 0];
Then I don't have to specify these also in the prototxt.

5. Keep the original BatchNorm from BVLC/caffe as it is, un-touched - so that compatibility to BVLC/caffe is not affected and old BVLC/caffe models can be used for fine tuning. If possible, provide a CUDNN version of this original BatchNorm without scaling as well, so that it can be accelerated.

[RSly]

hi,
there are some valuable suggestions here.

I wonder if anything has improved with regards to batch normalization in NVCaffe?

thanks

[lukeyeager]

/cc @borisgin

[borisgin]

I wrote new BatchNorm which is

1. backward compatible both with previous nvcaffe-0.15 and with BVLC caffe .prototxt and snapshots (can read both)
2. supports BN in-place (the same input and output layers)
3. no need to specify lr_mult... in .prototxt.
4. supports cudnn 6

It will be released with new nvcaffe

[RSly]

Thanks @borisgin
Is it possible to already test it on a development branch?
Also with all the different versions, it would be nice to create and maintain a BN layer definition template :)

[mathmanu]

Thanks @borisgin for the update.

Will there be reverse compatibility as well i.e. can the NVIDIA/caffe CUDNN trained models be used for fine tuning in BVLC/caffe CPU and GPU modes? This is because there are several frameworks (such as FasterRCNN, SSD etc) built around BVLC/caffe, and I am wondering whether they will understand this new model trained in NVIDIA/caffe.

If complete compatibility is not there, isn't better to give this layer a new name, such as BatchNormScale, and make sure that the layers BatchNorm and BatchNormScale co-exist. That way we will be able to check the type and write a utility to do the reverse conversion, if needed.

[borisgin]

You can use old BVLC prototxt and models with NV-caffe, so you can train old models on new nv-caffe. you can't load new nvcaffe models into BVLC caffe since BVLC caffe does not have fused BN and scale layer. having different name for the layer would not help, since for this you should have such layer in BVLC caffe. it would be much simpler just to replace old BVLC BN layer with new one from Nvcaffe.

[fslzj]

The gpu_diff() of blobs[2] and blobs[3] should be always 0. It is necessary set the 3rd and 4th param (blobs_[2] and blobs[3]) as param{ lr_mult:0 decay_mult:0}?
In addition, what does the blobs_[4] represent? It seems not be never used.

[szm-R]

Hi, @borisgin did you merged your batch norm layer with nvcaffe? because I can't still use models trained with bvlc on NV and it's really annoying :(

[borisgin]

Yes, it merged, Can you send a link on the Model which you use, please?

[szm-R]

The version of NVcaffe I'm using is 0.15.14 and this is one of the models I've tested which results in this error:
F1008 11:16:40.578579 5197 net.cpp:797] Check failed: target_blobs.size() == source_layer.blobs_size() (5 vs. 3) Incompatible number of blobs for layer conv1/7x7_s2/bn