Flaky test_gluon_model_zoo_gpu.test_training @ Python3: MKLDNN-GPU

[marcoabreu]

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/PR-9809/4/pipeline
http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/master/389/pipeline
http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/master/391/pipeline
http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/master/387/pipeline

======================================================================

FAIL: test_gluon_model_zoo_gpu.test_training

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 198, in runTest

    self.test(*self.arg)

  File "/workspace/tests/python/gpu/../unittest/common.py", line 155, in test_new

    orig_test(*args, **kwargs)

  File "/workspace/tests/python/gpu/test_gluon_model_zoo_gpu.py", line 159, in test_training

    assert_almost_equal(cpu_out.asnumpy(), gpu_out.asnumpy(), rtol=1e-2, atol=1e-2)

  File "/workspace/python/mxnet/test_utils.py", line 493, in assert_almost_equal

    raise AssertionError(msg)

AssertionError: 

Items are not equal:

Error 84.491165 exceeds tolerance rtol=0.010000, atol=0.010000.  Location of maximum error:(9, 877), a=0.816808, b=-0.181214

 a: array([[ 0.46908194,  0.25131682,  0.27432024, ..., -0.3243659 ,

        -0.8637756 ,  0.57461524],

       [ 0.18945426,  0.32339886,  0.18884647, ...,  0.00044107,...

 b: array([[ 0.34904164,  0.3221189 ,  0.05189171, ..., -0.14858764,

        -0.9381798 ,  0.39666444],

       [ 0.3438487 ,  0.4455883 ,  0.32494336, ...,  0.15454161,...

-------------------- >> begin captured logging << --------------------

urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): data.mxnet.io

urllib3.connectionpool: DEBUG: http://data.mxnet.io:80 "GET /data/val-5k-256.rec HTTP/1.1" 200 150874780

root: INFO: downloaded http://data.mxnet.io/data/val-5k-256.rec into data/val-5k-256.rec successfully

common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1760574333 to reproduce.

--------------------- >> end captured logging << ---------------------

[zheng-da]

I think we should disable it for now. I run the tests many times. It seems both CPU and GPU occasionally generate incorrect results. I need more time to test the tests.

…

On Sun, Feb 18, 2018 at 3:13 AM Marco de Abreu ***@***.***> wrote: ´´´ FAIL: test_gluon_model_zoo_gpu.test_training Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/workspace/tests/python/gpu/test_gluon_model_zoo_gpu.py", line 150, in test_training assert_almost_equal(cpu_out.asnumpy(), gpu_out.asnumpy(), rtol=1e-2, atol=1e-2) File "/workspace/python/mxnet/test_utils.py", line 495, in assert_almost_equal raise AssertionError(msg) AssertionError: Items are not equal: Error 76.641708 exceeds tolerance rtol=0.010000, atol=0.010000. Location of maximum error:(9, 304), a=0.754152, b=-0.052508 a: array([[ 0.45648926, -0.19809955, 0.35798055, ..., 1.0708873 , -0.3380539 , -0.22070615], [ 0.42138988, 0.02357741, 0.38420224, ..., 1.0584493 ,... b: array([[ 0.28696495, -0.17991166, 0.37088192, ..., 1.0100358 , -0.3728746 , -0.14392784], [ 0.27283525, 0.03558442, 0.45698166, ..., 0.9900025 ,... -------------------- >> begin captured logging << -------------------- urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): data.mxnet.io urllib3.connectionpool: DEBUG: http://data.mxnet.io:80 "GET /data/val-5k-256.rec HTTP/1.1" 200 150874780 root: INFO: downloaded http://data.mxnet.io/data/val-5k-256.rec into data/val-5k-256.rec successfully --------------------- >> end captured logging << --------------------- ´´´ — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#9820>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAETUbgbE8AKGK5QtCNCn119Qr8M2JNlks5tWAXYgaJpZM4SJpzP> .

[marcoabreu]

Sounds good. Since we've now got the CI randomness PR merged into the master, it should be no problem to generate faulty tests.

[marcoabreu]

@piiswrong @szha are you fine with disabling this test?

[marcoabreu]

Ah wait. I just checked a few runs of test failures and it ONLY happens with MKLDNN. It looks to me like this is not a flaky test but could rather have a problem in the underlying implementation.

@zheng-da @mli can you please make sure this gets resolved? This test is one of the biggest failure reasons and there might be a valid test failure.

[pengzhao-intel]

@marcoabreu Thanks for the testing.
Seems it's a non-deterministic failure both in GPU and CPU backend.
We will take look this case to figure out the root cause w/ high priority.

BTW, the root cause of #9843, #9844 & #9845 may be the same as this one so I think we can duplicate these issues and track the progress in one thread.

[ptrendx]

At least #9845 is not MKLDNN specific - I've seen (deterministic) failures in that test before, see #8780.

[zheng-da]

@marcoabreu I was testing it yesterday. It seems the problem exists when MKLDNN is compiled into MXNet, so it should be a bug caused by the code inside USE_MKLDNN. I'll look into the problem this week and try to fix it as soon as possible.

I agree with @pengzhao-intel . This problem might be related to the other flaky failures.

[zheng-da]

I run more tests. The bug occurs when MKLDNN is compiled into MXNet and we use the threaded engine, which is the default executing engine. If we use the naive executing engine, the problem doesn't show up. It means that the bug isn't in the operator implementation. It should be in the executor engine or in the NDArray.