Implement BatchNormInternal for cuda

[mindest]

Description: Implement BatchNormInternal for cuda

• type binding: input/output, scale/bias, mean/var are of separate types, support fp16 case;
• add corresponding forward test cases;
• reenable gradient test case.

Motivation and Context

• Original version of BatchNormalization does not support training in fp16, and has issue when updating running_mean/_var.

[SherlockNoMad]

let's rename this to saved_inv_var to reflect the reality.
and comment that this test data will only work for CUDA and not CPU.

[SherlockNoMad]

If the cudnn is actually returning saved_inv_std, would this UT also work for CPU impl?

[mindest]

It should, but I also infer from the result given in the calculation that

• when calculating saved_inv_std and y, it uses biased std/var
• when calculating running_var, it uses unbiased std/var

As for the CPU implementation, it always uses the biased one for calculation. Not sure why cudnn has such inconsistency itself and which is more reasonable.

And the above difference makes the running_var output differ for CPU/CUDA given the same input data.

[SherlockNoMad]

ic... thank you for the detail investigation.
Could you please also help document this subtle difference in the kernel and UT comment?

I think the CPU impl is the correct one, as in the ONNX spec, we explicitly mentioned that the variance should be population variance, aka biased variance.

When the number of sample is large, the difference between biased-var and unbiased-var would be small. Let's note this done and move on.

[SherlockNoMad]

Hi @mindest, thanks a lot for the PR.
It's good to merge, after adding the comment about the subtle unbiased-var result by cudnnBatchNorm.

Please ask Vincent for sign-off if I am not online.