control flow operator tests failing on Linux and Windows

[larroy]

Note: Providing complete information in the most concise form is the best way to get help. This issue template serves as the checklist for essential information to most of the technical issues and bug reports. For non-technical issues and feature requests, feel free to present the information in what you believe is the best form.

For Q & A and discussion, please start a discussion thread at https://discuss.mxnet.io

Description

nosetests test_contrib_control_flow.py is failing in at least py3 in linux (debug mode) and py2 in windows

Environment info (Required)

----------Python Info----------
Version      : 3.5.2
Compiler     : GCC 5.4.0 20160609
Build        : ('default', 'Nov 23 2017 16:37:01')
Arch         : ('64bit', 'ELF')
------------Pip Info-----------
Version      : 18.0
Directory    : /home/piotr/devel/mxnet/mxnet/mxnet_py3/lib/python3.5/site-packages/pip
----------MXNet Info-----------
Version      : 1.3.0
Directory    : /home/piotr/devel/mxnet/mxnet/python/mxnet
Hashtag not found. Not installed from pre-built package.
----------System Info----------
Platform     : Linux-4.15.0-24-generic-x86_64-with-Ubuntu-16.04-xenial
system       : Linux
node         : oruga
release      : 4.15.0-24-generic
version      : #26~16.04.1-Ubuntu SMP Fri Jun 15 14:35:08 UTC 2018
----------Hardware Info----------
machine      : x86_64
processor    : x86_64
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                32
On-line CPU(s) list:   0-31
Thread(s) per core:    2
Core(s) per socket:    16
Socket(s):             1
NUMA node(s):          1
Vendor ID:             AuthenticAMD
CPU family:            23
Model:                 1
Model name:            AMD Ryzen Threadripper 1950X 16-Core Processor
Stepping:              1
CPU MHz:               1865.541
CPU max MHz:           3400.0000
CPU min MHz:           2200.0000
BogoMIPS:              6799.26
Virtualization:        AMD-V
L1d cache:             32K
L1i cache:             64K
L2 cache:              512K
L3 cache:              8192K
NUMA node0 CPU(s):     0-31
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid amd_dcm aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb hw_pstate sme ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 xsaves clzero irperf xsaveerptr arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif overflow_recov succor smca
----------Network Test----------
Setting timeout: 10
Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0267 sec, LOAD: 0.5596 sec.
Timing for Gluon Tutorial(en): http://gluon.mxnet.io, DNS: 0.0295 sec, LOAD: 0.2071 sec.
Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0384 sec, LOAD: 0.8882 sec.
Timing for FashionMNIST: https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, DNS: 0.0285 sec, LOAD: 1.1026 sec.
Timing for Conda: https://repo.continuum.io/pkgs/free/, DNS: 0.0265 sec, LOAD: 0.0963 sec.
Timing for Gluon Tutorial(cn): https://zh.gluon.ai, DNS: 0.1624 sec, LOAD: 0.8599 sec.

Package used (Python/R/Scala/Julia):
(I'm using Python)

Build info (Required if built from source)

Compiler (gcc/clang/mingw/visual studio):
gcc and vs 14

MXNet commit hash:
394e5cc

Build config:

cmake\
    -DCMAKE_CXX_COMPILER_LAUNCHER=ccache \
    -DCMAKE_C_COMPILER_LAUNCHER=ccache \
    -DUSE_CPP_PACKAGE=ON\
    -DUSE_CUDA=OFF\
    -DUSE_OPENMP=OFF\
    -DUSE_OPENCV=ON\
    -DCMAKE_BUILD_TYPE=Debug\
    -GNinja ..
ninja -v

Error Message:

.[INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=133245677 to reproduce.
F..[INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1862402888 to reproduce.
F[INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1371884833 to reproduce.
F[INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1164686348 to reproduce.
F.[INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=275941523 to reproduce.
F
======================================================================
FAIL: test_contrib_control_flow.test_while_loop_for_foreach
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/piotr/devel/mxnet/mxnet/mxnet_py3/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/piotr/devel/mxnet/mxnet/tests/python/unittest/common.py", line 172, in test_new
    orig_test(*args, **kwargs)
  File "/home/piotr/devel/mxnet/mxnet/tests/python/unittest/test_contrib_control_flow.py", line 618, in test_while_loop_for_foreach
    n_steps=5,
  File "/home/piotr/devel/mxnet/mxnet/tests/python/unittest/test_contrib_control_flow.py", line 322, in case_1
    **params
  File "/home/piotr/devel/mxnet/mxnet/tests/python/unittest/test_contrib_control_flow.py", line 240, in _verify_while_loop
    assert_almost_equal(imp_out, sym_out, rtol=1e-4, atol=1e-4)
  File "/home/piotr/devel/mxnet/mxnet/python/mxnet/test_utils.py", line 493, in assert_almost_equal
    raise AssertionError(msg)
AssertionError: 
Items are not equal:
Error 9993.736328 exceeds tolerance rtol=0.000100, atol=0.000100.  Location of maximum error:(3, 0), a=-2.064602, b=-4891.381348
 a: array([[ 0.96158886],
       [ 0.6512103 ],
       [ 1.7375352 ],...
 b: array([[  127.01133],
       [ -379.67126],
       [ 1419.797  ],...
-------------------- >> begin captured logging << --------------------
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=133245677 to reproduce.
--------------------- >> end captured logging << ---------------------

======================================================================
FAIL: test_contrib_control_flow.test_cond
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/piotr/devel/mxnet/mxnet/mxnet_py3/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/piotr/devel/mxnet/mxnet/tests/python/unittest/common.py", line 172, in test_new
    orig_test(*args, **kwargs)
  File "/home/piotr/devel/mxnet/mxnet/tests/python/unittest/test_contrib_control_flow.py", line 1129, in test_cond
    (2, 3),
  File "/home/piotr/devel/mxnet/mxnet/tests/python/unittest/test_contrib_control_flow.py", line 1087, in run_case
    **params
  File "/home/piotr/devel/mxnet/mxnet/tests/python/unittest/test_contrib_control_flow.py", line 1056, in _verify_cond
    assert_almost_equal(imp_out, sym_out, rtol=1e-5, atol=1e-5)
  File "/home/piotr/devel/mxnet/mxnet/python/mxnet/test_utils.py", line 493, in assert_almost_equal
    raise AssertionError(msg)
AssertionError: 
Items are not equal:
Error 30851.714844 exceeds tolerance rtol=0.000010, atol=0.000010.  Location of maximum error:(0, 0), a=2.513998, b=4.081830
 a: array([[ 2.513998  , -0.06135673, -0.42358735],
       [ 0.13876215, -0.403529  ,  1.133575  ]], dtype=float32)
 b: array([[ 4.0818295 , -0.19492993, -0.59344125],
       [ 0.35262597, -0.72699374,  1.6560557 ]], dtype=float32)
-------------------- >> begin captured logging << --------------------
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1862402888 to reproduce.
--------------------- >> end captured logging << ---------------------

======================================================================
FAIL: test_contrib_control_flow.test_contrib_rnn
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/piotr/devel/mxnet/mxnet/mxnet_py3/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/piotr/devel/mxnet/mxnet/tests/python/unittest/common.py", line 172, in test_new
    orig_test(*args, **kwargs)
  File "/home/piotr/devel/mxnet/mxnet/tests/python/unittest/test_contrib_control_flow.py", line 1193, in test_contrib_rnn
    check_contrib_rnn(cell_type, num_states)
  File "/home/piotr/devel/mxnet/mxnet/tests/python/unittest/test_contrib_control_flow.py", line 1177, in check_contrib_rnn
    assert_almost_equal(res1.asnumpy(), res2.asnumpy(), rtol=0.001, atol=0.0001)
  File "/home/piotr/devel/mxnet/mxnet/python/mxnet/test_utils.py", line 493, in assert_almost_equal
    raise AssertionError(msg)
AssertionError: 
Items are not equal:
Error 5429.805664 exceeds tolerance rtol=0.001000, atol=0.000100.  Location of maximum error:(4, 2, 29), a=-0.566625, b=-0.003677
 a: array([[[ 0.33987436, -0.12906979,  0.33146507, ...,  0.4379619 ,
         -0.40905568,  0.17931955],
        [-0.40162748,  0.07519079,  0.19347169, ..., -0.31540644,...
 b: array([[[ 1.        ,  1.        ,  0.9999986 , ...,  1.        ,
         -0.9180124 ,  1.        ],
        [-1.        , -0.99999005, -0.999999  , ..., -1.        ,...
-------------------- >> begin captured logging << --------------------
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1371884833 to reproduce.
--------------------- >> end captured logging << ---------------------

======================================================================
FAIL: test_contrib_control_flow.test_foreach
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/piotr/devel/mxnet/mxnet/mxnet_py3/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/piotr/devel/mxnet/mxnet/tests/python/unittest/common.py", line 172, in test_new
    orig_test(*args, **kwargs)
  File "/home/piotr/devel/mxnet/mxnet/tests/python/unittest/test_contrib_control_flow.py", line 1324, in test_foreach
    lambda frees : [frees[0] + frees[1]])
  File "/home/piotr/devel/mxnet/mxnet/tests/python/unittest/test_contrib_control_flow.py", line 1297, in verify_foreach
    rtol=0.001, atol=0.0001)
  File "/home/piotr/devel/mxnet/mxnet/python/mxnet/test_utils.py", line 493, in assert_almost_equal
    raise AssertionError(msg)
AssertionError: 
Items are not equal:
Error 17112.298828 exceeds tolerance rtol=0.001000, atol=0.000100.  Location of maximum error:(2, 1), a=1016.000000, b=56.000000
 a: array([[   2.,   18.],
       [  18.,   90.],
       [ 198., 1016.]], dtype=float32)
 b: array([[ 2., 12.],
       [12., 30.],
       [30., 56.]], dtype=float32)
-------------------- >> begin captured logging << --------------------
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1164686348 to reproduce.
--------------------- >> end captured logging << ---------------------

======================================================================
FAIL: test_contrib_control_flow.test_foreach_rnn
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/piotr/devel/mxnet/mxnet/mxnet_py3/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/piotr/devel/mxnet/mxnet/tests/python/unittest/common.py", line 172, in test_new
    orig_test(*args, **kwargs)
  File "/home/piotr/devel/mxnet/mxnet/tests/python/unittest/test_contrib_control_flow.py", line 1664, in test_foreach_rnn
    check_foreach_rnn(cell_type, num_states)
  File "/home/piotr/devel/mxnet/mxnet/tests/python/unittest/test_contrib_control_flow.py", line 1652, in check_foreach_rnn
    rtol=0.001, atol=0.0001)
  File "/home/piotr/devel/mxnet/mxnet/python/mxnet/test_utils.py", line 493, in assert_almost_equal
    raise AssertionError(msg)
AssertionError: 
Items are not equal:
Error 482.427795 exceeds tolerance rtol=0.001000, atol=0.000100.  Location of maximum error:(0, 0), a=14.919886, b=28.919886
 a: array([[14.919886, 16.264572, 18.878408, 20.891766],
       [14.76185 ,  8.976585, 15.503097, 27.326567]], dtype=float32)
 b: array([[28.919886, 18.264572, 19.878408, 23.891766],
       [18.76185 , 15.976585, 23.503098, 21.326567]], dtype=float32)
-------------------- >> begin captured logging << --------------------
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=275941523 to reproduce.
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 9 tests in 7.259s

FAILED (failures=5)

[junrushao]

CC @zheng-da seems that foreach, while_loop and cond all fail.

[junrushao]

Hi @larroy,

Thank you so much for the valuable feedback! It seems that it fails many many testcases in control flow operators. Could you help to provide more information:

1. Does it work when compiled with the default Makefile with default flags? Did you try to cleanup previous build cache?
2. Does it happen every time (deterministic failure) or from time to time (flaky test)

Thank you so much!

[zheng-da]

It seems I can't reproduce the bug. Can you double check if all submodules are updated?

[larroy]

This fails on windows: branch "windows" on my fork.
instance m5d.25xlarge

To reproduce do:

python ci\build_windows.py
ci/windows/test_py3_cpu.ps1

======================================================================
ERROR: test_contrib_control_flow.test_while_loop_simple_forward
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Anaconda3\envs\py3\lib\site-packages\nose\case.py", line 197, in runTest
    self.test(*self.arg)
  File "d:\mxnet\tests\python\unittest\common.py", line 172, in test_new
    orig_test(*args, **kwargs)
  File "d:\mxnet\tests\python\unittest\test_contrib_control_flow.py", line 58, in test_while_loop_simple_forward
    mx.nd.array([0], dtype="int64"), # s
  File "d:\mxnet\windows_package\python\mxnet\gluon\block.py", line 540, in __call__
    out = self.forward(*args)
  File "d:\mxnet\windows_package\python\mxnet\gluon\block.py", line 905, in forward
    return self._call_cached_op(x, *args)
  File "d:\mxnet\windows_package\python\mxnet\gluon\block.py", line 811, in _call_cached_op
    out = self._cached_op(*cargs)
  File "d:\mxnet\windows_package\python\mxnet\_ctypes\ndarray.py", line 150, in __call__
    ctypes.byref(out_stypes)))
  File "d:\mxnet\windows_package\python\mxnet\base.py", line 255, in check_call
    raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: Error in operator _testblock6__while_loop0: [11:39:51] D:\mxnet\src\operator\control_flow.cc:758: Check failed: input_nids.size() == subg_in.size() (1 vs. 2)
-------------------- >> begin captured logging << --------------------
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1828478084 to reproduce.
--------------------- >> end captured logging << ---------------------

======================================================================
ERROR: test_contrib_control_flow.test_while_loop_for_foreach
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Anaconda3\envs\py3\lib\site-packages\nose\case.py", line 197, in runTest
    self.test(*self.arg)
  File "d:\mxnet\tests\python\unittest\common.py", line 172, in test_new
    orig_test(*args, **kwargs)
  File "d:\mxnet\tests\python\unittest\test_contrib_control_flow.py", line 606, in test_while_loop_for_foreach
    case_0()
  File "d:\mxnet\tests\python\unittest\test_contrib_control_flow.py", line 280, in case_0
    n_steps=1,
  File "d:\mxnet\tests\python\unittest\test_contrib_control_flow.py", line 236, in _verify_while_loop
    sym_outs, sym_grads = _get_symbolic_result(out_grads, n_steps)
  File "d:\mxnet\tests\python\unittest\test_contrib_control_flow.py", line 218, in _get_symbolic_result
    args_grad=args_grad,
  File "d:\mxnet\windows_package\python\mxnet\symbol\symbol.py", line 1703, in bind
    ctypes.byref(handle)))
  File "d:\mxnet\windows_package\python\mxnet\base.py", line 255, in check_call
    raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: Error in operator _while_loop0: [11:39:51] D:\mxnet\src\operator\control_flow.cc:758: Check failed: input_nids.size() == subg_in.size() (1 vs. 2)
-------------------- >> begin captured logging << --------------------
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=528855989 to reproduce.
--------------------- >> end captured logging << ---------------------

======================================================================
ERROR: test_contrib_control_flow.test_while_loop_nested
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Anaconda3\envs\py3\lib\site-packages\nose\case.py", line 197, in runTest
    self.test(*self.arg)
  File "d:\mxnet\tests\python\unittest\common.py", line 172, in test_new
    orig_test(*args, **kwargs)
  File "d:\mxnet\tests\python\unittest\test_contrib_control_flow.py", line 887, in test_while_loop_nested
    sym_out, sym_grad = _get_sym_result(is_train=is_train, args=args, args_grad=args_grad, out_grad=out_grad)
  File "d:\mxnet\tests\python\unittest\test_contrib_control_flow.py", line 876, in _get_sym_result
    args_grad=args_grad,
  File "d:\mxnet\windows_package\python\mxnet\symbol\symbol.py", line 1703, in bind
    ctypes.byref(handle)))
  File "d:\mxnet\windows_package\python\mxnet\base.py", line 255, in check_call
    raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: Error in operator _while_loop1: [11:39:51] D:\mxnet\src\operator\control_flow.cc:758: Check failed: input_nids.size() == subg_in.size() (1 vs. 4)
-------------------- >> begin captured logging << --------------------
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1172039107 to reproduce.
--------------------- >> end captured logging << ---------------------

======================================================================
ERROR: test_contrib_control_flow.test_while_loop_rnn
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Anaconda3\envs\py3\lib\site-packages\nose\case.py", line 197, in runTest
    self.test(*self.arg)
  File "d:\mxnet\tests\python\unittest\common.py", line 172, in test_new
    orig_test(*args, **kwargs)
  File "d:\mxnet\tests\python\unittest\test_contrib_control_flow.py", line 934, in test_while_loop_rnn
    s_0=(batch_size, hidden_dim),
  File "d:\mxnet\windows_package\python\mxnet\symbol\symbol.py", line 996, in infer_shape
    res = self._infer_shape_impl(False, *args, **kwargs)
  File "d:\mxnet\windows_package\python\mxnet\symbol\symbol.py", line 1126, in _infer_shape_impl
    ctypes.byref(complete)))
  File "d:\mxnet\windows_package\python\mxnet\base.py", line 255, in check_call
    raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: Error in operator _while_loop3: [11:39:51] D:\mxnet\src\operator\control_flow.cc:758: Check failed: input_nids.size() == subg_in.size() (1 vs. 8)
-------------------- >> begin captured stdout << ---------------------
infer_shape error. Arguments:
  data: (3, 2, 4)
  s_0: (2, 3)

--------------------- >> end captured stdout << ----------------------
-------------------- >> begin captured logging << --------------------
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=714833391 to reproduce.
--------------------- >> end captured logging << ---------------------

======================================================================
ERROR: test_contrib_control_flow.test_cond
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Anaconda3\envs\py3\lib\site-packages\nose\case.py", line 197, in runTest
    self.test(*self.arg)
  File "d:\mxnet\tests\python\unittest\common.py", line 172, in test_new
    orig_test(*args, **kwargs)
  File "d:\mxnet\tests\python\unittest\test_contrib_control_flow.py", line 1129, in test_cond
    (2, 3),
  File "d:\mxnet\tests\python\unittest\test_contrib_control_flow.py", line 1087, in run_case
    **params
  File "d:\mxnet\tests\python\unittest\test_contrib_control_flow.py", line 1052, in _verify_cond
    sym_outs, sym_grads = _get_symbolic_result(out_grads)
  File "d:\mxnet\tests\python\unittest\test_contrib_control_flow.py", line 1040, in _get_symbolic_result
    {"FreeVar" + str(i): mx.nd.zeros(s) for i, s in enumerate(free_var_shapes)},
  File "d:\mxnet\windows_package\python\mxnet\symbol\symbol.py", line 1703, in bind
    ctypes.byref(handle)))
  File "d:\mxnet\windows_package\python\mxnet\base.py", line 255, in check_call
    raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: Error in operator _cond0: [11:39:51] D:\mxnet\src\operator\control_flow.cc:1065: Check failed: input_nids.size() == subg_in.size() (1 vs. 2)
-------------------- >> begin captured logging << --------------------
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1451113207 to reproduce.
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
XML: d:\mxnet\nosetests_unittest.xml
----------------------------------------------------------------------
Ran 590 tests in 565.787s

FAILED (SKIP=10, errors=5)

[larroy]

And in linux:

(mxnet_py3) piotr@oruga:0:~/devel/mxnet/mxnet/tests/python/unittest (flaky)+$ nosetests test_contrib_control_flow.py

.[INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=2139283002 to reproduce.
F.[INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1979070457 to reproduce.
F[INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=655250043 to reproduce.
F[INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=828861162 to reproduce.
F[INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1883457791 to reproduce.
F[INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=896726612 to reproduce.
F[INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1361494438 to reproduce.
F
======================================================================
FAIL: test_contrib_control_flow.test_while_loop_for_foreach
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/piotr/devel/mxnet/mxnet/mxnet_py3/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/piotr/devel/mxnet/mxnet/tests/python/unittest/common.py", line 172, in test_new
    orig_test(*args, **kwargs)
  File "/home/piotr/devel/mxnet/mxnet/tests/python/unittest/test_contrib_control_flow.py", line 618, in test_while_loop_for_foreach
    n_steps=5,
  File "/home/piotr/devel/mxnet/mxnet/tests/python/unittest/test_contrib_control_flow.py", line 322, in case_1
    **params
  File "/home/piotr/devel/mxnet/mxnet/tests/python/unittest/test_contrib_control_flow.py", line 240, in _verify_while_loop
    assert_almost_equal(imp_out, sym_out, rtol=1e-4, atol=1e-4)
  File "/home/piotr/devel/mxnet/mxnet/python/mxnet/test_utils.py", line 493, in assert_almost_equal
    raise AssertionError(msg)
AssertionError: 
Items are not equal:
Error 9950.742188 exceeds tolerance rtol=0.000100, atol=0.000100.  Location of maximum error:(4, 0), a=-56.551762, b=-11682.902344
 a: array([[ -0.38177693],
       [  1.3139207 ],
       [ -4.6210213 ],...
 b: array([[   -82.3224 ],
       [   266.25528],
       [  -961.9607 ],...
-------------------- >> begin captured logging << --------------------
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=2139283002 to reproduce.
--------------------- >> end captured logging << ---------------------

======================================================================
FAIL: test_contrib_control_flow.test_while_loop_rnn
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/piotr/devel/mxnet/mxnet/mxnet_py3/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/piotr/devel/mxnet/mxnet/tests/python/unittest/common.py", line 172, in test_new
    orig_test(*args, **kwargs)
  File "/home/piotr/devel/mxnet/mxnet/tests/python/unittest/test_contrib_control_flow.py", line 969, in test_while_loop_rnn
    assert_almost_equal(x, y, rtol=1e-4, atol=1e-4)
  File "/home/piotr/devel/mxnet/mxnet/python/mxnet/test_utils.py", line 493, in assert_almost_equal
    raise AssertionError(msg)
AssertionError: 
Items are not equal:
Error 3136.270508 exceeds tolerance rtol=0.000100, atol=0.000100.  Location of maximum error:(1, 0, 0), a=0.177647, b=0.715753
 a: array([[[ 0.17861612,  0.00000016, -0.00000008],
        [ 0.7398634 ,  0.00000016, -0.00000005]],
...
 b: array([[[ 0.49748358,  0.00000021, -0.00000008],
        [ 0.8218198 ,  0.00000021, -0.00000005]],
...
-------------------- >> begin captured logging << --------------------
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1979070457 to reproduce.
--------------------- >> end captured logging << ---------------------

======================================================================
FAIL: test_contrib_control_flow.test_cond
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/piotr/devel/mxnet/mxnet/mxnet_py3/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/piotr/devel/mxnet/mxnet/tests/python/unittest/common.py", line 172, in test_new
    orig_test(*args, **kwargs)
  File "/home/piotr/devel/mxnet/mxnet/tests/python/unittest/test_contrib_control_flow.py", line 1129, in test_cond
    (2, 3),
  File "/home/piotr/devel/mxnet/mxnet/tests/python/unittest/test_contrib_control_flow.py", line 1087, in run_case
    **params
  File "/home/piotr/devel/mxnet/mxnet/tests/python/unittest/test_contrib_control_flow.py", line 1056, in _verify_cond
    assert_almost_equal(imp_out, sym_out, rtol=1e-5, atol=1e-5)
  File "/home/piotr/devel/mxnet/mxnet/python/mxnet/test_utils.py", line 493, in assert_almost_equal
    raise AssertionError(msg)
AssertionError: 
Items are not equal:
Error 74184.750000 exceeds tolerance rtol=0.000010, atol=0.000010.  Location of maximum error:(0, 2), a=-3.311802, b=-15.702538
 a: array([[-1.3206271 , -0.30327547, -3.3118017 ],
       [-0.04058055, -1.1410502 , -0.38296768]], dtype=float32)
 b: array([[ -5.027631  ,  -1.7639736 , -15.7025385 ],
       [ -0.39918604,  -0.89145124,  -0.6611627 ]], dtype=float32)
-------------------- >> begin captured logging << --------------------
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=655250043 to reproduce.
--------------------- >> end captured logging << ---------------------

======================================================================
FAIL: test_contrib_control_flow.test_contrib_rnn
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/piotr/devel/mxnet/mxnet/mxnet_py3/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/piotr/devel/mxnet/mxnet/tests/python/unittest/common.py", line 172, in test_new
    orig_test(*args, **kwargs)
  File "/home/piotr/devel/mxnet/mxnet/tests/python/unittest/test_contrib_control_flow.py", line 1193, in test_contrib_rnn
    check_contrib_rnn(cell_type, num_states)
  File "/home/piotr/devel/mxnet/mxnet/tests/python/unittest/test_contrib_control_flow.py", line 1177, in check_contrib_rnn
    assert_almost_equal(res1.asnumpy(), res2.asnumpy(), rtol=0.001, atol=0.0001)
  File "/home/piotr/devel/mxnet/mxnet/python/mxnet/test_utils.py", line 493, in assert_almost_equal
    raise AssertionError(msg)
AssertionError: 
Items are not equal:
Error 4594.763672 exceeds tolerance rtol=0.001000, atol=0.000100.  Location of maximum error:(0, 2, 65), a=-0.484439, b=0.006944
 a: array([[[-0.20372412,  0.28804368, -0.16334954, ...,  0.85556924,
         -0.05496813,  0.14489251],
        [-0.09742647,  0.16537917, -0.07625283, ..., -0.50425553,...
 b: array([[[-0.9999997 ,  0.999737  ,  0.99994946, ...,  1.        ,
         -0.99980724, -0.9999999 ],
        [-0.9865516 ,  0.9999999 , -0.9432786 , ..., -1.        ,...
-------------------- >> begin captured logging << --------------------
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=828861162 to reproduce.
--------------------- >> end captured logging << ---------------------

======================================================================
FAIL: test_contrib_control_flow.test_foreach
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/piotr/devel/mxnet/mxnet/mxnet_py3/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/piotr/devel/mxnet/mxnet/tests/python/unittest/common.py", line 172, in test_new
    orig_test(*args, **kwargs)
  File "/home/piotr/devel/mxnet/mxnet/tests/python/unittest/test_contrib_control_flow.py", line 1324, in test_foreach
    lambda frees : [frees[0] + frees[1]])
  File "/home/piotr/devel/mxnet/mxnet/tests/python/unittest/test_contrib_control_flow.py", line 1297, in verify_foreach
    rtol=0.001, atol=0.0001)
  File "/home/piotr/devel/mxnet/mxnet/python/mxnet/test_utils.py", line 493, in assert_almost_equal
    raise AssertionError(msg)
AssertionError: 
Items are not equal:
Error 5882.352539 exceeds tolerance rtol=0.001000, atol=0.000100.  Location of maximum error:(2, 1), a=386.000000, b=56.000000
 a: array([[  2.,  18.],
       [ 16.,  90.],
       [ 82., 386.]], dtype=float32)
 b: array([[ 2., 12.],
       [12., 30.],
       [30., 56.]], dtype=float32)
-------------------- >> begin captured logging << --------------------
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1883457791 to reproduce.
--------------------- >> end captured logging << ---------------------

======================================================================
FAIL: test_contrib_control_flow.test_foreach_nested
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/piotr/devel/mxnet/mxnet/mxnet_py3/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/piotr/devel/mxnet/mxnet/tests/python/unittest/common.py", line 172, in test_new
    orig_test(*args, **kwargs)
  File "/home/piotr/devel/mxnet/mxnet/tests/python/unittest/test_contrib_control_flow.py", line 1562, in test_foreach_nested
    assert_almost_equal(res.asnumpy(), e.outputs[0].asnumpy(), rtol=0.001, atol=0.0001)
  File "/home/piotr/devel/mxnet/mxnet/python/mxnet/test_utils.py", line 493, in assert_almost_equal
    raise AssertionError(msg)
AssertionError: 
Items are not equal:
Error 362.811737 exceeds tolerance rtol=0.001000, atol=0.000100.  Location of maximum error:(1, 1, 0), a=28.000000, b=44.000000
 a: array([[[ 8., 22.],
        [12., 28.]],
...
 b: array([[[ 8., 25.],
        [12., 43.]],
...
-------------------- >> begin captured logging << --------------------
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=896726612 to reproduce.
--------------------- >> end captured logging << ---------------------

======================================================================
FAIL: test_contrib_control_flow.test_foreach_rnn
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/piotr/devel/mxnet/mxnet/mxnet_py3/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/piotr/devel/mxnet/mxnet/tests/python/unittest/common.py", line 172, in test_new
    orig_test(*args, **kwargs)
  File "/home/piotr/devel/mxnet/mxnet/tests/python/unittest/test_contrib_control_flow.py", line 1664, in test_foreach_rnn
    check_foreach_rnn(cell_type, num_states)
  File "/home/piotr/devel/mxnet/mxnet/tests/python/unittest/test_contrib_control_flow.py", line 1652, in check_foreach_rnn
    rtol=0.001, atol=0.0001)
  File "/home/piotr/devel/mxnet/mxnet/python/mxnet/test_utils.py", line 493, in assert_almost_equal
    raise AssertionError(msg)
AssertionError: 
Items are not equal:
Error 206.602814 exceeds tolerance rtol=0.001000, atol=0.000100.  Location of maximum error:(0, 0, 3), a=0.772737, b=1.000000
 a: array([[[0.91257566, 0.93461883, 0.95408654, 0.7727369 ],
        [0.85923904, 0.79833966, 0.913655  , 0.8811143 ]],
...
 b: array([[[1.       , 1.       , 1.       , 1.       ],
        [1.       , 1.       , 1.       , 0.9999943]],
...
-------------------- >> begin captured logging << --------------------
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1361494438 to reproduce.
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 9 tests in 3.555s

FAILED (failures=7)

[larroy]

Could it be outdated TVM / nnvm subrepo? I can't always reproduce

[larroy]

Tested on a clean repo, got the failure again.

tvm @ commit 290226e1c9adbb3e598f9ed9184018df1c12be33
Author: Jian Weng werefluke@gmail.com
Date: Fri Jun 22 13:47:45 2018 -0700

[FRONTEND] A Python hybrid frontend (#1251)

(mxnet_py3) piotr@oruga:0:~/tmp/mxnet/tests/python/unittest (master)+$ nosetests test_contrib_control_flow.py 
.[INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1683218039 to reproduce.
F.[INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=941599958 to reproduce.
F[INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=240299619 to reproduce.
F[INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1598500162 to reproduce.
F[INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1409147838 to reproduce.
F.[INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=2084233205 to reproduce.
F
======================================================================
FAIL: test_contrib_control_flow.test_while_loop_for_foreach
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/piotr/tmp/mxnet/mxnet_py3/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/piotr/tmp/mxnet/tests/python/unittest/common.py", line 172, in test_new
    orig_test(*args, **kwargs)
  File "/home/piotr/tmp/mxnet/tests/python/unittest/test_contrib_control_flow.py", line 618, in test_while_loop_for_foreach
    n_steps=5,
  File "/home/piotr/tmp/mxnet/tests/python/unittest/test_contrib_control_flow.py", line 322, in case_1
    **params
  File "/home/piotr/tmp/mxnet/tests/python/unittest/test_contrib_control_flow.py", line 240, in _verify_while_loop
    assert_almost_equal(imp_out, sym_out, rtol=1e-4, atol=1e-4)
  File "/home/piotr/tmp/mxnet/python/mxnet/test_utils.py", line 493, in assert_almost_equal
    raise AssertionError(msg)
AssertionError: 
Items are not equal:
Error 9419.442383 exceeds tolerance rtol=0.000100, atol=0.000100.  Location of maximum error:(3, 0), a=-278.021240, b=-4805.078613
 a: array([[   7.8534303],
       [ -21.46705  ],
       [  81.15463  ],...
 b: array([[  122.632 ],
       [ -384.9325],
       [ 1387.7172],...
-------------------- >> begin captured logging << --------------------
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1683218039 to reproduce.
--------------------- >> end captured logging << ---------------------

======================================================================
FAIL: test_contrib_control_flow.test_while_loop_rnn
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/piotr/tmp/mxnet/mxnet_py3/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/piotr/tmp/mxnet/tests/python/unittest/common.py", line 172, in test_new
    orig_test(*args, **kwargs)
  File "/home/piotr/tmp/mxnet/tests/python/unittest/test_contrib_control_flow.py", line 969, in test_while_loop_rnn
    assert_almost_equal(x, y, rtol=1e-4, atol=1e-4)
  File "/home/piotr/tmp/mxnet/python/mxnet/test_utils.py", line 493, in assert_almost_equal
    raise AssertionError(msg)
AssertionError: 
Items are not equal:
Error 1192.026733 exceeds tolerance rtol=0.000100, atol=0.000100.  Location of maximum error:(2, 0, 0), a=-0.761594, b=-1.000000
 a: array([[[-0.9640275 ,  0.00000083, -0.9615498 ],
        [-0.99496937,  0.00000054, -0.96949065]],
...
 b: array([[[-0.9999996 ,  0.00000125, -0.9999996 ],
        [-0.9999996 ,  0.00000131, -0.99998957]],
...
-------------------- >> begin captured logging << --------------------
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=941599958 to reproduce.
--------------------- >> end captured logging << ---------------------

======================================================================
FAIL: test_contrib_control_flow.test_cond
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/piotr/tmp/mxnet/mxnet_py3/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/piotr/tmp/mxnet/tests/python/unittest/common.py", line 172, in test_new
    orig_test(*args, **kwargs)
  File "/home/piotr/tmp/mxnet/tests/python/unittest/test_contrib_control_flow.py", line 1129, in test_cond
    (2, 3),
  File "/home/piotr/tmp/mxnet/tests/python/unittest/test_contrib_control_flow.py", line 1087, in run_case
    **params
  File "/home/piotr/tmp/mxnet/tests/python/unittest/test_contrib_control_flow.py", line 1056, in _verify_cond
    assert_almost_equal(imp_out, sym_out, rtol=1e-5, atol=1e-5)
  File "/home/piotr/tmp/mxnet/python/mxnet/test_utils.py", line 493, in assert_almost_equal
    raise AssertionError(msg)
AssertionError: 
Items are not equal:
Error 36462.175781 exceeds tolerance rtol=0.000010, atol=0.000010.  Location of maximum error:(1, 1), a=-0.928016, b=-2.034438
 a: array([[-1.4817346 , -0.76791453, -0.26404768],
       [-0.22492273, -0.92801595, -0.4418202 ]], dtype=float32)
 b: array([[-1.4817346 , -0.76791453, -0.26404768],
       [-0.22492273, -2.0344381 , -0.4418202 ]], dtype=float32)
-------------------- >> begin captured logging << --------------------
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=240299619 to reproduce.
--------------------- >> end captured logging << ---------------------

======================================================================
FAIL: test_contrib_control_flow.test_contrib_rnn
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/piotr/tmp/mxnet/mxnet_py3/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/piotr/tmp/mxnet/tests/python/unittest/common.py", line 172, in test_new
    orig_test(*args, **kwargs)
  File "/home/piotr/tmp/mxnet/tests/python/unittest/test_contrib_control_flow.py", line 1193, in test_contrib_rnn
    check_contrib_rnn(cell_type, num_states)
  File "/home/piotr/tmp/mxnet/tests/python/unittest/test_contrib_control_flow.py", line 1177, in check_contrib_rnn
    assert_almost_equal(res1.asnumpy(), res2.asnumpy(), rtol=0.001, atol=0.0001)
  File "/home/piotr/tmp/mxnet/python/mxnet/test_utils.py", line 493, in assert_almost_equal
    raise AssertionError(msg)
AssertionError: 
Items are not equal:
Error 5951.356934 exceeds tolerance rtol=0.001000, atol=0.000100.  Location of maximum error:(3, 6, 25), a=0.627485, b=-0.006533
 a: array([[[ 0.00909132, -0.26697633,  0.07916373, ...,  0.31235063,
         -0.15670209, -0.4814965 ],
        [ 0.68945557,  0.31152144, -0.1506751 , ...,  0.06814943,...
 b: array([[[-0.16027294, -0.97353685, -0.99997824, ...,  0.99858755,
         -1.        , -1.        ],
        [ 1.        ,  1.        ,  0.9863122 , ...,  0.9025492 ,...
-------------------- >> begin captured logging << --------------------
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1598500162 to reproduce.
--------------------- >> end captured logging << ---------------------

======================================================================
FAIL: test_contrib_control_flow.test_foreach
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/piotr/tmp/mxnet/mxnet_py3/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/piotr/tmp/mxnet/tests/python/unittest/common.py", line 172, in test_new
    orig_test(*args, **kwargs)
  File "/home/piotr/tmp/mxnet/tests/python/unittest/test_contrib_control_flow.py", line 1324, in test_foreach
    lambda frees : [frees[0] + frees[1]])
  File "/home/piotr/tmp/mxnet/tests/python/unittest/test_contrib_control_flow.py", line 1297, in verify_foreach
    rtol=0.001, atol=0.0001)
  File "/home/piotr/tmp/mxnet/python/mxnet/test_utils.py", line 493, in assert_almost_equal
    raise AssertionError(msg)
AssertionError: 
Items are not equal:
Error 11016.041992 exceeds tolerance rtol=0.001000, atol=0.000100.  Location of maximum error:(2, 1), a=674.000000, b=56.000000
 a: array([[  2.,  22.],
       [ 24., 216.],
       [ 90., 674.]], dtype=float32)
 b: array([[ 2., 12.],
       [12., 30.],
       [30., 56.]], dtype=float32)
-------------------- >> begin captured logging << --------------------
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1409147838 to reproduce.
--------------------- >> end captured logging << ---------------------

======================================================================
FAIL: test_contrib_control_flow.test_foreach_rnn
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/piotr/tmp/mxnet/mxnet_py3/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/piotr/tmp/mxnet/tests/python/unittest/common.py", line 172, in test_new
    orig_test(*args, **kwargs)
  File "/home/piotr/tmp/mxnet/tests/python/unittest/test_contrib_control_flow.py", line 1664, in test_foreach_rnn
    check_foreach_rnn(cell_type, num_states)
  File "/home/piotr/tmp/mxnet/tests/python/unittest/test_contrib_control_flow.py", line 1652, in check_foreach_rnn
    rtol=0.001, atol=0.0001)
  File "/home/piotr/tmp/mxnet/python/mxnet/test_utils.py", line 493, in assert_almost_equal
    raise AssertionError(msg)
AssertionError: 
Items are not equal:
Error 190.672119 exceeds tolerance rtol=0.001000, atol=0.000100.  Location of maximum error:(0, 1, 0), a=0.790261, b=1.000000
 a: array([[[0.93483055, 0.95413005, 0.9039076 , 0.8553966 ],
        [0.7902607 , 0.8560267 , 0.89933217, 0.92859614]],
...
 b: array([[[1., 1., 1., 1.],
        [1., 1., 1., 1.]],
...
-------------------- >> begin captured logging << --------------------
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=2084233205 to reproduce.
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 9 tests in 6.138s

FAILED (failures=6)
(mxnet_py3) piotr@oruga:1:~/tmp/mxnet/tests/python/unittest (master)+$ 

[larroy]

Built with:

#!/bin/bash
set -e
set -x

renice -n 19 -p $$

mkdir -p build && cd build
#cmake -DUSE_CPP_PACKAGE=ON -DUSE_CUDA=OFF -DUSE_OPENMP=OFF -DUSE_OPENCV=ON -DCMAKE_BUILD_TYPE=Debug ..
cmake\
    -DCMAKE_CXX_COMPILER_LAUNCHER=ccache \
    -DCMAKE_C_COMPILER_LAUNCHER=ccache \
    -DUSE_CPP_PACKAGE=ON\
    -DUSE_CUDA=OFF\
    -DUSE_OPENMP=OFF\
    -DUSE_OPENCV=ON\
    -DCMAKE_BUILD_TYPE=Debug\
    -GNinja ..
ninja -v
#cmake -DUSE_CPP_PACKAGE=ON -DUSE_CUDA=OFF -DUSE_OPENMP=OFF -DUSE_OPENCV=ON ..
#VERBOSE=1 make -j5
cd ..
if [ ! -d mxnet_py3 ]; then
    virtualenv -p `which python3` mxnet_py3
fi
source mxnet_py3/bin/activate
cd python
pip install -e .
cd ..
pip install opencv-python
pip install ipython
pip install matplotlib
pip install nose

(mxnet_py3) piotr@oruga:0:~/tmp/mxnet/3rdparty/tvm ((290226e...))$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.4 LTS
Release:	16.04
Codename:	xenial
(mxnet_py3) piotr@oruga:0:~/tmp/mxnet/3rdparty/tvm ((290226e...))$ dpkg -l | grep ninja
ii  ninja-build                                   1.5.1-0.1ubuntu1                             amd64        small build system closest in spirit to Make

[zheng-da]

I saw the same errors when the tvm submodule was out-of-date. After I updated tvm, it works fine. I'll try on a new instance and follow your exact instructions.

[Roshrini]

@sandeep-krishnamurthy Can you please add label: CI, Test

[zheng-da]

i created a new instance and followed your instructions again. I could reproduce the problem. However, after I tried compiling with Makefile, (which leads me to install atlas), I couldn't reproduce the problem any more. I'm wondering if the problem is related to the BLAS library. could you check what BLAS library do you use?

[larroy]

@zheng-da I had the same situation. But then I reproduced on a fresh repo after making sure subs are updated. I'm not sure what's going on.

[zheng-da]

it seems there are some conflicts with blas.