Skip to content

Add fp16 support for BatchNormalization Forward/Backward#7218

Closed
mindest wants to merge 2 commits intomasterfrom
linmin/batch_norm
Closed

Add fp16 support for BatchNormalization Forward/Backward#7218
mindest wants to merge 2 commits intomasterfrom
linmin/batch_norm

Conversation

@mindest
Copy link
Contributor

@mindest mindest commented Apr 2, 2021

Description

  • Add fp16 support for BatchNormalization ForwardTraining/Backward
  • Bind inputs mean and var with outputs mean and var, respectively

Motivation and Context

  • Currently BatchNormalization does not support fp16 and should be kept in fp32 when training
  • Fix the problem that running_mean and running_var do not change in training mode

@pranav-prakash
Copy link
Contributor

pranav-prakash commented Apr 2, 2021

Note that as I mentioned in #6946 the cuda kernel for BatchNorm breaks the ONNX spec in outputtting inverse standard deviation instead of variance for the saved statistic. While doing so avoids the need for an extra operation here since we can use the cudnn output directly, I'm not sure if it's worth breaking spec for that (and that too it's probably not a significant performance gain).

But then again the spec for batchnorm was recently updated to drop the redundant "saved mean/var" output (onnx/onnx#3379), so I think those two outputs can just be removed entirely (at least for opset 14).

@SherlockNoMad SherlockNoMad added the training issues related to ONNX Runtime training; typically submitted using template label Apr 5, 2021
@SherlockNoMad
Copy link
Contributor

Hi @mindest, which model need this op? is this urgently needed?

@mindest mindest closed this May 10, 2021
@mindest mindest deleted the linmin/batch_norm branch August 5, 2021 05:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

training issues related to ONNX Runtime training; typically submitted using template

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants