[TEST] test_matmul topi flaky

[tqchen]

http://ci.tvm.ai:8080/blue/rest/organizations/jenkins/pipelines/tvm/branches/PR-1778/runs/3/nodes/231/log/?start=0

@grwlf can you take a look?

[sergei-mironov]

That's unfortunate. Sure, I'll investigate.

[junrushao]

It seems to me a threshold issue. Currently in test_matmul, we have atol = 0 and rtol = 0. And the numpy's assert_allclose uses atol + rtol * abs(desired) as the threshold, so when abs(desired) is pretty small, and atol = 0, the threshold can be relatively small compared with numerical precision.

Ref: https://docs.scipy.org/doc/numpy/reference/generated/numpy.testing.assert_allclose.html

[tqchen]

good point, @junrushao1994 can you send a patch?

[junrushao]

@tqchen Should I send a quick fix only for test_matmul, or modify all the assert_allclose with something like atol=1e-5?

FYI:

$ grep "assert_allclose" -rIn tvm | wc -l
     313

[tqchen]

most cases do comes with rtol i guess, and i think we are fine as long as they are not flaky

[junrushao]

@tqchen #1791

[junrushao]

Ideally it would be great to have some tools helping run a test for 1000 times before it is merged, so that the choice of thresholds won't bother further commits.

[sergei-mironov]

@junrushao1994 thank you. I was able to reproduce the error. On my machine, typical numeric error is 1e-7 or less, which is a bit higher than the default value of atol (1e-8). Most of the time, relative tolerance prevents assert from triggering but in rare cases RNG happens to generate operands close to zero. In such cases relative tolerance become infinitesimal, so the error condition hits. Increasing atol seems to be a correct fix. I am approving the PR.

As @junrushao1994 also points out, we use assert_allclose extensively. In the same time, atol are only set in rare cases, which is probably wrong as we may see now.

tvm $ git grep assert_allclose | grep -v atol | wc -l
258

[tqchen]

good point, maybe we want to send pr to change these place preemptively

[tqchen]

Close this, for now, @grwlf you are more than welcomed to send another pr to add atol to most other settings