Flaky test_random.test_exponential_generator @ Python2: MKLDNN-CPU

[marcoabreu]

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/master/395/pipeline

======================================================================

FAIL: test_random.test_exponential_generator

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/usr/local/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest

    self.test(*self.arg)

  File "/workspace/tests/python/unittest/common.py", line 155, in test_new

    orig_test(*args, **kwargs)

  File "/workspace/tests/python/unittest/test_random.py", line 374, in test_exponential_generator

    verify_generator(generator=generator_mx_same_seed, buckets=buckets, probs=probs)

  File "/workspace/python/mxnet/test_utils.py", line 1891, in verify_generator

    str(buckets), str(probs)))

AssertionError: Generator test fails, Chi-square p=[0.21031861623067064, 0.023120840905593496, 0.02566284972432512, 0.0396308101152387, 0.031248307960405564], obs_freq=[array([200561, 200496, 199515, 200039, 199389]), array([198895, 200007, 200598, 200780, 199720]), array([200509, 200524, 199203, 200596, 199168]), array([199922, 200497, 200741, 199937, 198903]), array([199609, 200258, 199725, 201138, 199270])], expected_freq=[array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32)].

buckets=[(0.0, 0.22314355131420976), (0.22314355131420976, 0.5108256237659907), (0.5108256237659907, 0.916290731874155), (0.916290731874155, 1.6094379124341005), (1.6094379124341005, inf)], probs=[0.2, 0.2, 0.2, 0.2, 0.2]

-------------------- >> begin captured stdout << ---------------------

ctx=cpu(0), dtype=float16, Scale=0.1:

ctx=cpu(0), dtype=float16, Scale=1:



--------------------- >> end captured stdout << ----------------------

-------------------- >> begin captured logging << --------------------

common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1690699840 to reproduce.

--------------------- >> end captured logging << ---------------------

[zheng-da]

It seems this problem is related to #9856

[sxjscience]

As shown in the log, the chi-square value is above the 0.05 threshold (indicates the check has passed) once in the 5 runs, i.e, 20% success rate. Currently we force the success rate to be 25% (https://github.com/apache/incubator-mxnet/blob/master/python/mxnet/test_utils.py#L1848), we could lower it to 10% to fix the test.

[marcoabreu]

That might fix the test, but could we risk masking a problem that way? I'm afraid I'm unfamiliar with the topic, so would you mind explaining how an actual failure would look like and how we could select the best threshold?

[sxjscience]

@marcoabreu A wrongly implemented random number generator would typically have success_rate = 0, i.e, producing p < 0.05 for all the 5 runs. However, since this tests the randomness, it's really tricky to set the threshold and my previous design is to force the success rate to be at least 25%.

[zheng-da]

This test also fails constantly with a special seed.

$ export MXNET_TEST_SEED=548415857
$ for i in {1..10}; do nosetests -v tests/python/gpu/test_operator_gpu.py:test_exponential_generator; done
[INFO] Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=798238534 to reproduce.
[WARNING] *** test-level seed set: all "@with_seed()" tests run deterministically ***
test_operator_gpu.test_exponential_generator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
FAIL

======================================================================
FAIL: test_operator_gpu.test_exponential_generator
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/usr/lib/python2.7/dist-packages/nose/util.py", line 620, in newfunc
    return func(*arg, **kw)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/common.py", line 155, in test_new
    orig_test(*args, **kwargs)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/test_random.py", line 369, in test_exponential_generator
    verify_generator(generator=generator_mx, buckets=buckets, probs=probs)
  File "/home/ubuntu/incubator-mxnet/python/mxnet/test_utils.py", line 1891, in verify_generator
    str(buckets), str(probs)))
AssertionError: Generator test fails, Chi-square p=[0.027708004423388152, 0.048739405069752051, 0.049679278693325266, 0.093148605055595635, 0.0099893338307405818], obs_freq=[array([198823, 200824, 200335, 199968, 200050]), array([200671, 199850, 198978, 199885, 200616]), array([200488, 200237, 199846, 198872, 200557]), array([199234, 199570, 200093, 200233, 200870]), array([200578, 198743, 200487, 200589, 199603])], expected_freq=[array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32)].
buckets=[(0.0, 0.22314355131420976), (0.22314355131420976, 0.51082562376599072), (0.51082562376599072, 0.916290731874155), (0.916290731874155, 1.6094379124341005), (1.6094379124341005, inf)], probs=[0.2, 0.2, 0.2, 0.2, 0.2]
-------------------- >> begin captured stdout << ---------------------
ctx=gpu(0), dtype=float16, Scale=0.1:
ctx=gpu(0), dtype=float16, Scale=1:
ctx=gpu(0), dtype=float32, Scale=0.1:
ctx=gpu(0), dtype=float32, Scale=1:

--------------------- >> end captured stdout << ----------------------
-------------------- >> begin captured logging << --------------------
common: INFO: Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=798238534 to reproduce.
common: WARNING: *** test-level seed set: all "@with_seed()" tests run deterministically ***
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 1 test in 3.485s

FAILED (failures=1)
[INFO] Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=422739486 to reproduce.
[WARNING] *** test-level seed set: all "@with_seed()" tests run deterministically ***
test_operator_gpu.test_exponential_generator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
FAIL

======================================================================
FAIL: test_operator_gpu.test_exponential_generator
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/usr/lib/python2.7/dist-packages/nose/util.py", line 620, in newfunc
    return func(*arg, **kw)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/common.py", line 155, in test_new
    orig_test(*args, **kwargs)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/test_random.py", line 369, in test_exponential_generator
    verify_generator(generator=generator_mx, buckets=buckets, probs=probs)
  File "/home/ubuntu/incubator-mxnet/python/mxnet/test_utils.py", line 1891, in verify_generator
    str(buckets), str(probs)))
AssertionError: Generator test fails, Chi-square p=[0.027708004423388152, 0.048739405069752051, 0.049679278693325266, 0.093148605055595635, 0.0099893338307405818], obs_freq=[array([198823, 200824, 200335, 199968, 200050]), array([200671, 199850, 198978, 199885, 200616]), array([200488, 200237, 199846, 198872, 200557]), array([199234, 199570, 200093, 200233, 200870]), array([200578, 198743, 200487, 200589, 199603])], expected_freq=[array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32)].
buckets=[(0.0, 0.22314355131420976), (0.22314355131420976, 0.51082562376599072), (0.51082562376599072, 0.916290731874155), (0.916290731874155, 1.6094379124341005), (1.6094379124341005, inf)], probs=[0.2, 0.2, 0.2, 0.2, 0.2]
-------------------- >> begin captured stdout << ---------------------
ctx=gpu(0), dtype=float16, Scale=0.1:
ctx=gpu(0), dtype=float16, Scale=1:
ctx=gpu(0), dtype=float32, Scale=0.1:
ctx=gpu(0), dtype=float32, Scale=1:

--------------------- >> end captured stdout << ----------------------
-------------------- >> begin captured logging << --------------------
common: INFO: Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=422739486 to reproduce.
common: WARNING: *** test-level seed set: all "@with_seed()" tests run deterministically ***
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 1 test in 4.569s

FAILED (failures=1)
[INFO] Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=1017842502 to reproduce.
[WARNING] *** test-level seed set: all "@with_seed()" tests run deterministically ***
test_operator_gpu.test_exponential_generator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
FAIL

======================================================================
FAIL: test_operator_gpu.test_exponential_generator
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/usr/lib/python2.7/dist-packages/nose/util.py", line 620, in newfunc
    return func(*arg, **kw)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/common.py", line 155, in test_new
    orig_test(*args, **kwargs)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/test_random.py", line 369, in test_exponential_generator
    verify_generator(generator=generator_mx, buckets=buckets, probs=probs)
  File "/home/ubuntu/incubator-mxnet/python/mxnet/test_utils.py", line 1891, in verify_generator
    str(buckets), str(probs)))
AssertionError: Generator test fails, Chi-square p=[0.027708004423388152, 0.048739405069752051, 0.049679278693325266, 0.093148605055595635, 0.0099893338307405818], obs_freq=[array([198823, 200824, 200335, 199968, 200050]), array([200671, 199850, 198978, 199885, 200616]), array([200488, 200237, 199846, 198872, 200557]), array([199234, 199570, 200093, 200233, 200870]), array([200578, 198743, 200487, 200589, 199603])], expected_freq=[array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32)].
buckets=[(0.0, 0.22314355131420976), (0.22314355131420976, 0.51082562376599072), (0.51082562376599072, 0.916290731874155), (0.916290731874155, 1.6094379124341005), (1.6094379124341005, inf)], probs=[0.2, 0.2, 0.2, 0.2, 0.2]
-------------------- >> begin captured stdout << ---------------------
ctx=gpu(0), dtype=float16, Scale=0.1:
ctx=gpu(0), dtype=float16, Scale=1:
ctx=gpu(0), dtype=float32, Scale=0.1:
ctx=gpu(0), dtype=float32, Scale=1:

--------------------- >> end captured stdout << ----------------------
-------------------- >> begin captured logging << --------------------
common: INFO: Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=1017842502 to reproduce.
common: WARNING: *** test-level seed set: all "@with_seed()" tests run deterministically ***
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 1 test in 3.078s

FAILED (failures=1)
[INFO] Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=300887491 to reproduce.
[WARNING] *** test-level seed set: all "@with_seed()" tests run deterministically ***
test_operator_gpu.test_exponential_generator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
FAIL

======================================================================
FAIL: test_operator_gpu.test_exponential_generator
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/usr/lib/python2.7/dist-packages/nose/util.py", line 620, in newfunc
    return func(*arg, **kw)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/common.py", line 155, in test_new
    orig_test(*args, **kwargs)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/test_random.py", line 369, in test_exponential_generator
    verify_generator(generator=generator_mx, buckets=buckets, probs=probs)
  File "/home/ubuntu/incubator-mxnet/python/mxnet/test_utils.py", line 1891, in verify_generator
    str(buckets), str(probs)))
AssertionError: Generator test fails, Chi-square p=[0.027708004423388152, 0.048739405069752051, 0.049679278693325266, 0.093148605055595635, 0.0099893338307405818], obs_freq=[array([198823, 200824, 200335, 199968, 200050]), array([200671, 199850, 198978, 199885, 200616]), array([200488, 200237, 199846, 198872, 200557]), array([199234, 199570, 200093, 200233, 200870]), array([200578, 198743, 200487, 200589, 199603])], expected_freq=[array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32)].
buckets=[(0.0, 0.22314355131420976), (0.22314355131420976, 0.51082562376599072), (0.51082562376599072, 0.916290731874155), (0.916290731874155, 1.6094379124341005), (1.6094379124341005, inf)], probs=[0.2, 0.2, 0.2, 0.2, 0.2]
-------------------- >> begin captured stdout << ---------------------
ctx=gpu(0), dtype=float16, Scale=0.1:
ctx=gpu(0), dtype=float16, Scale=1:
ctx=gpu(0), dtype=float32, Scale=0.1:
ctx=gpu(0), dtype=float32, Scale=1:

--------------------- >> end captured stdout << ----------------------
-------------------- >> begin captured logging << --------------------
common: INFO: Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=300887491 to reproduce.
common: WARNING: *** test-level seed set: all "@with_seed()" tests run deterministically ***
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 1 test in 3.074s

FAILED (failures=1)
[INFO] Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=520324218 to reproduce.
[WARNING] *** test-level seed set: all "@with_seed()" tests run deterministically ***
test_operator_gpu.test_exponential_generator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
FAIL

======================================================================
FAIL: test_operator_gpu.test_exponential_generator
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/usr/lib/python2.7/dist-packages/nose/util.py", line 620, in newfunc
    return func(*arg, **kw)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/common.py", line 155, in test_new
    orig_test(*args, **kwargs)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/test_random.py", line 369, in test_exponential_generator
    verify_generator(generator=generator_mx, buckets=buckets, probs=probs)
  File "/home/ubuntu/incubator-mxnet/python/mxnet/test_utils.py", line 1891, in verify_generator
    str(buckets), str(probs)))
AssertionError: Generator test fails, Chi-square p=[0.027708004423388152, 0.048739405069752051, 0.049679278693325266, 0.093148605055595635, 0.0099893338307405818], obs_freq=[array([198823, 200824, 200335, 199968, 200050]), array([200671, 199850, 198978, 199885, 200616]), array([200488, 200237, 199846, 198872, 200557]), array([199234, 199570, 200093, 200233, 200870]), array([200578, 198743, 200487, 200589, 199603])], expected_freq=[array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32)].
buckets=[(0.0, 0.22314355131420976), (0.22314355131420976, 0.51082562376599072), (0.51082562376599072, 0.916290731874155), (0.916290731874155, 1.6094379124341005), (1.6094379124341005, inf)], probs=[0.2, 0.2, 0.2, 0.2, 0.2]
-------------------- >> begin captured stdout << ---------------------
ctx=gpu(0), dtype=float16, Scale=0.1:
ctx=gpu(0), dtype=float16, Scale=1:
ctx=gpu(0), dtype=float32, Scale=0.1:
ctx=gpu(0), dtype=float32, Scale=1:

--------------------- >> end captured stdout << ----------------------
-------------------- >> begin captured logging << --------------------
common: INFO: Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=520324218 to reproduce.
common: WARNING: *** test-level seed set: all "@with_seed()" tests run deterministically ***
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 1 test in 3.156s

FAILED (failures=1)
[INFO] Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=994645507 to reproduce.
[WARNING] *** test-level seed set: all "@with_seed()" tests run deterministically ***
test_operator_gpu.test_exponential_generator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
FAIL

======================================================================
FAIL: test_operator_gpu.test_exponential_generator
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/usr/lib/python2.7/dist-packages/nose/util.py", line 620, in newfunc
    return func(*arg, **kw)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/common.py", line 155, in test_new
    orig_test(*args, **kwargs)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/test_random.py", line 369, in test_exponential_generator
    verify_generator(generator=generator_mx, buckets=buckets, probs=probs)
  File "/home/ubuntu/incubator-mxnet/python/mxnet/test_utils.py", line 1891, in verify_generator
    str(buckets), str(probs)))
AssertionError: Generator test fails, Chi-square p=[0.027708004423388152, 0.048739405069752051, 0.049679278693325266, 0.093148605055595635, 0.0099893338307405818], obs_freq=[array([198823, 200824, 200335, 199968, 200050]), array([200671, 199850, 198978, 199885, 200616]), array([200488, 200237, 199846, 198872, 200557]), array([199234, 199570, 200093, 200233, 200870]), array([200578, 198743, 200487, 200589, 199603])], expected_freq=[array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32)].
buckets=[(0.0, 0.22314355131420976), (0.22314355131420976, 0.51082562376599072), (0.51082562376599072, 0.916290731874155), (0.916290731874155, 1.6094379124341005), (1.6094379124341005, inf)], probs=[0.2, 0.2, 0.2, 0.2, 0.2]
-------------------- >> begin captured stdout << ---------------------
ctx=gpu(0), dtype=float16, Scale=0.1:
ctx=gpu(0), dtype=float16, Scale=1:
ctx=gpu(0), dtype=float32, Scale=0.1:
ctx=gpu(0), dtype=float32, Scale=1:

--------------------- >> end captured stdout << ----------------------
-------------------- >> begin captured logging << --------------------
common: INFO: Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=994645507 to reproduce.
common: WARNING: *** test-level seed set: all "@with_seed()" tests run deterministically ***
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 1 test in 4.483s

FAILED (failures=1)
[INFO] Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=1429331890 to reproduce.
[WARNING] *** test-level seed set: all "@with_seed()" tests run deterministically ***
test_operator_gpu.test_exponential_generator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
FAIL

======================================================================
FAIL: test_operator_gpu.test_exponential_generator
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/usr/lib/python2.7/dist-packages/nose/util.py", line 620, in newfunc
    return func(*arg, **kw)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/common.py", line 155, in test_new
    orig_test(*args, **kwargs)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/test_random.py", line 369, in test_exponential_generator
    verify_generator(generator=generator_mx, buckets=buckets, probs=probs)
  File "/home/ubuntu/incubator-mxnet/python/mxnet/test_utils.py", line 1891, in verify_generator
    str(buckets), str(probs)))
AssertionError: Generator test fails, Chi-square p=[0.027708004423388152, 0.048739405069752051, 0.049679278693325266, 0.093148605055595635, 0.0099893338307405818], obs_freq=[array([198823, 200824, 200335, 199968, 200050]), array([200671, 199850, 198978, 199885, 200616]), array([200488, 200237, 199846, 198872, 200557]), array([199234, 199570, 200093, 200233, 200870]), array([200578, 198743, 200487, 200589, 199603])], expected_freq=[array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32)].
buckets=[(0.0, 0.22314355131420976), (0.22314355131420976, 0.51082562376599072), (0.51082562376599072, 0.916290731874155), (0.916290731874155, 1.6094379124341005), (1.6094379124341005, inf)], probs=[0.2, 0.2, 0.2, 0.2, 0.2]
-------------------- >> begin captured stdout << ---------------------
ctx=gpu(0), dtype=float16, Scale=0.1:
ctx=gpu(0), dtype=float16, Scale=1:
ctx=gpu(0), dtype=float32, Scale=0.1:
ctx=gpu(0), dtype=float32, Scale=1:

--------------------- >> end captured stdout << ----------------------
-------------------- >> begin captured logging << --------------------
common: INFO: Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=1429331890 to reproduce.
common: WARNING: *** test-level seed set: all "@with_seed()" tests run deterministically ***
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 1 test in 3.407s

FAILED (failures=1)
[INFO] Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=1471819377 to reproduce.
[WARNING] *** test-level seed set: all "@with_seed()" tests run deterministically ***
test_operator_gpu.test_exponential_generator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
FAIL

======================================================================
FAIL: test_operator_gpu.test_exponential_generator
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/usr/lib/python2.7/dist-packages/nose/util.py", line 620, in newfunc
    return func(*arg, **kw)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/common.py", line 155, in test_new
    orig_test(*args, **kwargs)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/test_random.py", line 369, in test_exponential_generator
    verify_generator(generator=generator_mx, buckets=buckets, probs=probs)
  File "/home/ubuntu/incubator-mxnet/python/mxnet/test_utils.py", line 1891, in verify_generator
    str(buckets), str(probs)))
AssertionError: Generator test fails, Chi-square p=[0.027708004423388152, 0.048739405069752051, 0.049679278693325266, 0.093148605055595635, 0.0099893338307405818], obs_freq=[array([198823, 200824, 200335, 199968, 200050]), array([200671, 199850, 198978, 199885, 200616]), array([200488, 200237, 199846, 198872, 200557]), array([199234, 199570, 200093, 200233, 200870]), array([200578, 198743, 200487, 200589, 199603])], expected_freq=[array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32)].
buckets=[(0.0, 0.22314355131420976), (0.22314355131420976, 0.51082562376599072), (0.51082562376599072, 0.916290731874155), (0.916290731874155, 1.6094379124341005), (1.6094379124341005, inf)], probs=[0.2, 0.2, 0.2, 0.2, 0.2]
-------------------- >> begin captured stdout << ---------------------
ctx=gpu(0), dtype=float16, Scale=0.1:
ctx=gpu(0), dtype=float16, Scale=1:
ctx=gpu(0), dtype=float32, Scale=0.1:
ctx=gpu(0), dtype=float32, Scale=1:

--------------------- >> end captured stdout << ----------------------
-------------------- >> begin captured logging << --------------------
common: INFO: Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=1471819377 to reproduce.
common: WARNING: *** test-level seed set: all "@with_seed()" tests run deterministically ***
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 1 test in 3.576s

FAILED (failures=1)
[INFO] Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=454445406 to reproduce.
[WARNING] *** test-level seed set: all "@with_seed()" tests run deterministically ***
test_operator_gpu.test_exponential_generator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
FAIL

======================================================================
FAIL: test_operator_gpu.test_exponential_generator
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/usr/lib/python2.7/dist-packages/nose/util.py", line 620, in newfunc
    return func(*arg, **kw)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/common.py", line 155, in test_new
    orig_test(*args, **kwargs)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/test_random.py", line 369, in test_exponential_generator
    verify_generator(generator=generator_mx, buckets=buckets, probs=probs)
  File "/home/ubuntu/incubator-mxnet/python/mxnet/test_utils.py", line 1891, in verify_generator
    str(buckets), str(probs)))
AssertionError: Generator test fails, Chi-square p=[0.027708004423388152, 0.048739405069752051, 0.049679278693325266, 0.093148605055595635, 0.0099893338307405818], obs_freq=[array([198823, 200824, 200335, 199968, 200050]), array([200671, 199850, 198978, 199885, 200616]), array([200488, 200237, 199846, 198872, 200557]), array([199234, 199570, 200093, 200233, 200870]), array([200578, 198743, 200487, 200589, 199603])], expected_freq=[array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32)].
buckets=[(0.0, 0.22314355131420976), (0.22314355131420976, 0.51082562376599072), (0.51082562376599072, 0.916290731874155), (0.916290731874155, 1.6094379124341005), (1.6094379124341005, inf)], probs=[0.2, 0.2, 0.2, 0.2, 0.2]
-------------------- >> begin captured stdout << ---------------------
ctx=gpu(0), dtype=float16, Scale=0.1:
ctx=gpu(0), dtype=float16, Scale=1:
ctx=gpu(0), dtype=float32, Scale=0.1:
ctx=gpu(0), dtype=float32, Scale=1:

--------------------- >> end captured stdout << ----------------------
-------------------- >> begin captured logging << --------------------
common: INFO: Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=454445406 to reproduce.
common: WARNING: *** test-level seed set: all "@with_seed()" tests run deterministically ***
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 1 test in 3.255s

FAILED (failures=1)
[INFO] Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=53548602 to reproduce.
[WARNING] *** test-level seed set: all "@with_seed()" tests run deterministically ***
test_operator_gpu.test_exponential_generator ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
FAIL

======================================================================
FAIL: test_operator_gpu.test_exponential_generator
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/usr/lib/python2.7/dist-packages/nose/util.py", line 620, in newfunc
    return func(*arg, **kw)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/common.py", line 155, in test_new
    orig_test(*args, **kwargs)
  File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/test_random.py", line 369, in test_exponential_generator
    verify_generator(generator=generator_mx, buckets=buckets, probs=probs)
  File "/home/ubuntu/incubator-mxnet/python/mxnet/test_utils.py", line 1891, in verify_generator
    str(buckets), str(probs)))
AssertionError: Generator test fails, Chi-square p=[0.027708004423388152, 0.048739405069752051, 0.049679278693325266, 0.093148605055595635, 0.0099893338307405818], obs_freq=[array([198823, 200824, 200335, 199968, 200050]), array([200671, 199850, 198978, 199885, 200616]), array([200488, 200237, 199846, 198872, 200557]), array([199234, 199570, 200093, 200233, 200870]), array([200578, 198743, 200487, 200589, 199603])], expected_freq=[array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32), array([200000, 200000, 200000, 200000, 200000], dtype=int32)].
buckets=[(0.0, 0.22314355131420976), (0.22314355131420976, 0.51082562376599072), (0.51082562376599072, 0.916290731874155), (0.916290731874155, 1.6094379124341005), (1.6094379124341005, inf)], probs=[0.2, 0.2, 0.2, 0.2, 0.2]
-------------------- >> begin captured stdout << ---------------------
ctx=gpu(0), dtype=float16, Scale=0.1:
ctx=gpu(0), dtype=float16, Scale=1:
ctx=gpu(0), dtype=float32, Scale=0.1:
ctx=gpu(0), dtype=float32, Scale=1:

--------------------- >> end captured stdout << ----------------------
-------------------- >> begin captured logging << --------------------
common: INFO: Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=53548602 to reproduce.
common: WARNING: *** test-level seed set: all "@with_seed()" tests run deterministically ***
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=548415857 to reproduce.
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 1 test in 3.210s

FAILED (failures=1)

[sxjscience]

Should I revise the code to use a smaller threshold?

[azai91]

@marcoabreu the success rate has been lowered. do we know if this test is still causing us problems?

[marcoabreu]

Haven't seen any problems recently. Thanks for following up!

[azai91]

kk ran this test 500x as well and didn't see a problem

ubuntu@ip-172-31-11-93:~/incubator-mxnet-original$ for i in {1..500}; do nosetests tests/python/unittest/test_random.py:test_exponential_generator; if [ $? -ne 0 ]; then break; fi; done || exit 1