Hanging flaky test test_operator.test_norm @ Python 3: GPU Win

[marcoabreu]

test_operator.test_norm causes hangs at Python 3: GPU Win.

http://jenkins.mxnet-ci.amazon-ml.com/view/Master%20Jobs/job/incubator-mxnet/job/master/1109/

test_operator.test_pad ... ok

test_operator.test_instance_normalization ... ok

test_operator.test_l2_normalization ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1234 to reproduce.

ok

test_operator.test_norm ... Sending interrupt signal to process

After 10s process did not stop

[marcoabreu]

On other occasions, it causes assertion errors:
http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/master/1108/pipeline

======================================================================

FAIL: test_operator.test_norm

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/usr/lib/python3.6/site-packages/nose/case.py", line 198, in runTest

    self.test(*self.arg)

  File "/work/mxnet/tests/python/unittest/common.py", line 157, in test_new

    orig_test(*args, **kwargs)

  File "/work/mxnet/tests/python/unittest/test_operator.py", line 3074, in test_norm

    check_numeric_gradient(norm_sym, [in_data], numeric_eps=epsilon, rtol=1e-2, atol=1e-3)

  File "/work/mxnet/python/mxnet/test_utils.py", line 914, in check_numeric_gradient

    ("NUMERICAL_%s"%name, "BACKWARD_%s"%name))

  File "/work/mxnet/python/mxnet/test_utils.py", line 493, in assert_almost_equal

    raise AssertionError(msg)

AssertionError: 

Items are not equal:

Error 1.028974 exceeds tolerance rtol=0.010000, atol=0.001000.  Location of maximum error:(0, 1, 4, 5, 0, 0), a=-0.003815, b=-0.002757

 NUMERICAL_data: array([[[[[[ 0.01990795, -0.06532669,  0.00363588, ...,  0.01215935,

            -0.00745058, -0.00047684],

           [-0.00017881, -0.10442734,  0.01651049, ..., -0.03290176,...

 BACKWARD_data: array([[[[[[ 0.01993995, -0.06537328,  0.00367729, ...,  0.01226579,

            -0.0074518 , -0.0004907 ],

           [-0.00015076, -0.10442342,  0.01648286, ..., -0.03304606,...

-------------------- >> begin captured logging << --------------------

common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=615595105 to reproduce.

--------------------- >> end captured logging << ---------------------

[eric-haibin-lin]

@anirudhacharya

[anirudhacharya]

I will look into it

[leezu]

The latter error could be due to that numpy.linalg.norm is not numerically stable (neither is the mxnet one). Possibly #11573 could fix this issue.

[szha]

FYI @anirudhacharya the test has been partially re-enabled in #11573.

[KellenSunderland]

@szha Would it make sense to close this issue and re-open if we see test issuse with test_norm again?

[anirudhacharya]

@KellenSunderland as Sheng mentioned it has only been partially enabled, there are certain test cases which are still not enabled and fix that was merged does not fix the occasional hanging of the test case on the CI.

[leezu]

@anirudh2290 @KellenSunderland Only the part that caused assertion errors during numerical checking of the gradient is disabled. It is not clear if the hang was occurring during check_numeric_gradient or during the other parts in the test. If the hang occurred outside of check_numeric_gradient it is fixed by change of underlying norm implementation, otherwise it is disabled..

You may want to rename this issue to make it clear that currently a part of the test is disabled due to numerical instability of the check_numeric_gradient.

[haojin2]

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/PR-11482/22/pipeline/858

======================================================================

FAIL: test_operator_gpu.test_norm

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 198, in runTest

    self.test(*self.arg)

  File "/usr/local/lib/python3.5/dist-packages/nose/util.py", line 620, in newfunc

    return func(*arg, **kw)

  File "/work/mxnet/tests/python/gpu/../unittest/common.py", line 172, in test_new

    orig_test(*args, **kwargs)

  File "/work/mxnet/tests/python/gpu/../unittest/test_ndarray.py", line 1324, in test_norm

    mx.test_utils.assert_almost_equal(npy_out, mx_out.asnumpy())

  File "/work/mxnet/python/mxnet/test_utils.py", line 493, in assert_almost_equal

    raise AssertionError(msg)

AssertionError: 

Items are not equal:

Error 200000.015625 exceeds tolerance rtol=0.000010, atol=0.000000.  Location of maximum error:(0, 0, 0, 0), a=0.567434, b=-0.567434

 a: array([[[[0.56743413, 0.3482769 ],

         [0.76402813, 0.21444368],

         [0.26532394, 0.35806283],...

 b: array([[[[-0.56743413, -0.3482769 ],

         [-0.76402813, -0.21444368],

         [-0.26532394, -0.35806283],...

-------------------- >> begin captured logging << --------------------

common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1479125905 to reproduce.

--------------------- >> end captured logging << ---------------------