batch normalization : moving average with different batch size

[matthieu637]

This formulation of decay moving average is more accurate if the batch size is changing over time.

[Noiredd]

I have restarted the tests that failed prior to #5973, it passes now. Anyone with better understanding of batch norm to state their opinion of this?

PS: Are you planning to change the GPU version too?

[matthieu637]

More information :
If the batch size is fixed, as usually in supervised learning, it won't change the result.
In case of multiple batch size, for instance in reinforcement learning, it will be a little more accurate : small batches have less impact than bigger.