[TEST][ONNX] ONNX test flaky sqrt, div, logsoftmax

[yangchen-MS]

With commit be07fac, the following test input produced failure for Sqrt op:

indata = np.asarray([[0.69074488, 0.29226714,-0.56205946, 0.01265035]], dtype='float32')

With the input above, I got:

outdata:
[[-0.90204036 -1.30051804 -2.15484476 -1.58013487]]
tvm_out:
[[-0.90204048 -1.30051827 -2.15484476 -1.58013499]]

[yangchen-MS]

Sorry, I pasted a wrong test data. Here is the correct one:

x = np.asarray([[[[ 0.94189703, 0.24268626, 0.25545985],
                  [ 0.95224684, 0.57228124, 0.63862652],
                  [ 0.93324018, 0.87410188, 0.62078589]],
                 [[ 0.21460213, 0.86773658, 0.62907255],
                  [ 0.18104482, 0.82294154, 0.26680526],
                  [ 0.411677,   0.03111285, 0.11248256]]]], dtype='float32')

Output I got on my machine:

out_np:
[[[[ 0.97051382  0.49263197  0.50543034]
   [ 0.97583139  0.75649273  0.79914111]
   [ 0.96604359  0.93493414  0.78789967]]

  [[ 0.46325168  0.9315238   0.79314095]
   [ 0.42549363  0.90716124  0.51653194]
   [ 0.64162064  0.17638835  0.33538419]]]]
tvm_out:
[[[[ 0.97051376  0.49263194  0.50543034]
   [ 0.97583133  0.75649256  0.79914111]
   [ 0.96604353  0.93493414  0.78789967]]

  [[ 0.46325165  0.9315238   0.79314089]
   [ 0.4254936   0.90716124  0.51653194]
   [ 0.64162064  0.17638835  0.33538419]]]]

[tqchen]

From the output, it is hard to tell if anything is wrong, or does it has anything to do with the specific PR, can you look a bit into the difference in the generated code and point to the exact test-case being used? Depending on the issue, maybe we need to fix the test-case(adjust the rtol) or the code

[yangchen-MS]

I should have been more careful when reporting the issue. The failure happened when running nnvm/tests/python/frontend/onnx for CPU target. My TVM built with LLVM 6.0.1. To reproduce the failure (hopefully you could reproduce it because it might be arch-related), you can replace:

x = np.random.uniform(size=in_shape).astype(dtype)

with

    x = np.asarray([[[[ 0.94189703, 0.24268626, 0.25545985],
                  [ 0.95224684, 0.57228124, 0.63862652],
                  [ 0.93324018, 0.87410188, 0.62078589]],
                 [[ 0.21460213, 0.86773658, 0.62907255],
                  [ 0.18104482, 0.82294154, 0.26680526],
                  [ 0.411677,   0.03111285, 0.11248256]]]], dtype='float32')

in function test_single_ops in nnvm/tests/python/frontend/onnx/test_forward.py, and run a similar command like below:

python -m nose -v ../nnvm/tests/python/frontend/onnx/test_forward.py

[yangchen-MS]

I think we may want to adjust 'rtol for these failing tests. The reason is that different FP-related algorithms (e.g. vectorization algorithm used by LLVM, or CUDA library) could produce results with different accuracy. For example, we might want to check only 3 or 4 most significant digits after dot.

[tqchen]

@yangchen-MS can you send in a PR to adjust the rtols and atol(of the recently reported flaky case). We can use tvm.testing.assert_allclose which sets the atol correctly.

In some cases, we might need to fix the way random numbers are generated(e.g. Div, (x/y) we will need to make sure y is not too small)

[yangchen-MS]

Yes, I will do that. Thanks.