Fix test_cublas and relax tolerance

[areusch]

• Allows test to pass on ci-gpu with new 18.04 image.

@tkonolige @tqchen

[comaniac]

FYI: It's possible that this error is caused by the CUBLAS math flag:

tvm/src/runtime/contrib/cublas/cublas.cc

Line 38 in 8131364

inline void CUBLASTryEnableTensorCore(cublasHandle_t hdl) {

According to the CUBLAS document, this flag is being deprecated. We also made some tests previously around this flag, and found that it is effective even for float32, meaning that CUBLAS kernel internally casts float32 to float16, does the computation, and casts the results back. As a result, this flag may introduce accuracy issue.

[areusch]

@comaniac i was sending this through CI for @tkonolige as he was out friday. I'll let him reply to your comment.

[tkonolige]

@comaniac Disabling the flag makes the tests pass. What should we do here? Accept lower accuracy for performance?

[comaniac]

@comaniac Disabling the flag makes the tests pass. What should we do here? Accept lower accuracy for performance?

I personally prefer to keep the accuracy, because it seems not right to tolerate 1e-2 for a single batch_matmul op. It means the end-to-end accuracy of all models with cublas.batch_matmul may be larger than 1e-2. cc @Hzfengsy @Laurawly as they added this flag at the time it hasn't been deprecated.

[Hzfengsy]

I also prefer to keep accuracy. Just as @comaniac said, 1e-2 is too low for larger end-to-end workloads

[areusch]

superseded by #8130