Added batchnorm layer with tests

[ducha-aiki]

based on fixed PRs from Russell91 (base test) and ChenglongChen (implementation)

[weiliu89]

I think the code misses computing the second part of derivative over mean during back propagation.

[weiliu89]

// I think we should add these following code between line 244 and 246. Correct me if I am wrong

// EX across spatial
caffe_cpu_gemv(CblasNoTrans, N_ * C_, H_ * W_, Dtype(1), bottom_diff, spatial_multiplier_.cpu_data(), Dtype(0), spatial_mean_.mutable_cpu_data());
// EX across batch
caffe_cpu_gemv(CblasNoTrans, N_, C_, Dtype(1), spatial_mean_.cpu_data(), batch_sum_multiplier_.cpu_data(), Dtype(0), batch_mean_.mutable_cpu_data());

caffe_cpu_gemm(CblasNoTrans, CblasNoTrans, N_, C_, 1, Dtype(1), batch_sum_multiplier_.cpu_data(), batch_mean_.cpu_data(), Dtype(0), spatial_mean_.mutable_cpu_data());
caffe_cpu_gemm(CblasNoTrans, CblasNoTrans, N_ * C_, H_ * W_, 1, Dtype(-1), spatial_mean_.cpu_data(), spatial_multiplier_.cpu_data(), Dtype(1), bottom_diff);

[weiliu89]

During backward, it misses step: Average 2 as shown in the image I attached.
Correct me if I am wrong

[ChenglongChen]

@weiliu89,
The second part in Eq (3) can be dropped as x_i - \mu has zero mean? As a result Average 2 is not necessary.

[weiliu89]

@ChenglongChen
I think you are correct. Do you have any idea how to compute the derivatives over input x? I have no idea. Do you have any reference on how to do it?

[ducha-aiki]

Replaced by #1965