softrelu activation clipping bug in MKLDNN

[safrooze]

Description

With MKLDNN (enabled in mxnet-cuxxmkl builds), using softrelu activation, the output is clipped to maximum value of 88.37625885009766. This regression was introduced in 1.2.0 when MKLDNN was enabled for activations other than relu in #10336.

Looking at MKLDNN implementation of softrelu, I don't see any overflow prevention logic, so most likely the exp() function overflows and results in this bug.

Environment info (Required)

mxnet-cu90mkl >= 1.2.0b20180403

Minimum reproducible example

nd.Activation(nd.array([100]), act_type='softrelu')

output:

[ 88.37625885]
<NDArray 1 @cpu(0)>

@jinhuang415

[safrooze]

@pengzhao-intel @zheng-da

[pengzhao-intel]

@safrooze thanks for the analysis :) @ZhennanQin please help take a look for this issue.

[ZhennanQin]

@safrooze Thanks for reporting this. We confirmed this is a bug in mkldnn, and will fix this in future. considering input > 88 is a corner case in most of models, most users shouldn't face this problem when using soft_relu with mkldnn. If you're having trouble with this on a certain model, please let us know.

[azai91]

@ZhennanQin do we have an ETA when this would be fixed? should we fallback if input too large?

[ZhennanQin]

@azai91 , this defect will be addressed within the next sprint, with expected target date around 15th Aug.

[safrooze]

@ZhennanQin Is this fixed in MKLDNN now?

[ZhennanQin]

@safrooze Patch is ready for internal test. Will push to github repo in next promotion.

[safrooze]

@ZhennanQin Is this pushed now?

[ZhennanQin]

@safrooze Yes, the commit is uxlfoundation/oneDNN@23f45a2. Thanks.

[apeforest]

@safrooze Please verify at your earliest convenience if this issue has been fixed. Thanks!

[apeforest]

@safrooze I have verified that using the master from intel/mkl-dnn module the problem has been fixed.

from mxnet import ndarray as nd

nd.Activation(nd.array([100]), act_type='softrelu')

[100.]
<NDArray 1 @cpu(0)>

@sandeep-krishnamurthy @nswamy Please close this issue.