[CI]Suspect a bug in reduce ops, leading to test_script failure

[siju-samuel]

nnvm/tests/python/compiler/test_top_level1.py and nnvm/tests/python/compiler/test_top_level4.py test scripts are failing.

For some of the PR's this script is success in CI and for some this will fail.
In local environment also this is failing for me. Please help to check and report whether this script is successful for you all.

Script 1

======================================================================
ERROR: test_top_level1.test_check_function
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/nose/case.py", line 197, in runTestself.test(*self.arg)
  File "/workspace/nnvm/tests/python/compiler/test_top_level1.py", line 129, in test_check_function
    check_function(z, lambda x, y: [2*x + y, x + 2*y, x, y, np.sum(x)])
  File "/workspace/nnvm/python/nnvm/testing/check_computation.py", line 437, in check_function
    main_function = graph_to_function(main_graph, target, ctx)
  File "/workspace/nnvm/python/nnvm/testing/check_computation.py", line 192, in graph_to_function
    compute_graph, lib, params = nnvm.compiler.build(graph, target, shape=shape, dtype=dtype)
  File "/workspace/nnvm/python/nnvm/compiler/build_module.py", line 305, in build
    graph = graph.apply("GraphCompile")
  File "/workspace/nnvm/python/nnvm/graph.py", line 234, in apply
    check_call(_LIB.NNGraphApplyPasses(self.handle, npass, cpass, ctypes.byref(ghandle)))
  File "/workspace/nnvm/python/nnvm/_base.py", line 75, in check_call
    raise NNVMError(py_str(_LIB.NNGetLastError()))
NNVMError: [10:28:50] /workspace/nnvm/src/compiler/compile_engine.cc:212: Check failed: out[i].ndim() == out_info[i].ndim() (0 vs. 1) sum

Stack trace returned 10 entries:
[bt] (0) /workspace/build/libtvm.so(dmlc::StackTrace[abi:cxx11]()+0x1bc) [0x7efdd902998c]
[bt] (1) /workspace/build/libtvm.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x28) [0x7efdd902a568]
[bt] (2) /workspace/nnvm/python/nnvm/../../../build/libnnvm_compiler.so(nnvm::compiler::CompileEngine::GetScheduleArgs(nnvm::Graph, tvm::Array<tvm::Tensor, void> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, tvm::Array<tvm::Tensor, void>*)+0x179d) [0x7efdbe3bf38d]
[bt] (3) /workspace/nnvm/python/nnvm/../../../build/libnnvm_compiler.so(nnvm::compiler::CompileEngine::DoLower(nnvm::Graph, tvm::Array<tvm::Tensor, void> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)+0xfd) [0x7efdbe3c0d5d]
[bt] (4) /workspace/nnvm/python/nnvm/../../../build/libnnvm_compiler.so(nnvm::compiler::CompileEngine::Lower(nnvm::Graph, tvm::Array<tvm::Tensor, void> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)+0x1bf) [0x7efdbe3c1e3f]
[bt] (5) /workspace/nnvm/python/nnvm/../../../build/libnnvm_compiler.so(nnvm::compiler::GraphLower(nnvm::Graph, tvm::Array<tvm::Tensor, void> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)+0x5f) [0x7efdbe3b6e1f]
[bt] (6) /workspace/nnvm/python/nnvm/../../../build/libnnvm_compiler.so(nnvm::compiler::GraphCompile(nnvm::Graph const&)+0xd7b) [0x7efdbe3c4e2b]
[bt] (7) /workspace/nnvm/python/nnvm/../../../build/libnnvm_compiler.so(std::_Function_handler<nnvm::Graph (nnvm::Graph), nnvm::Graph (*)(nnvm::Graph const&)>::_M_invoke(std::_Any_data const&, nnvm::Graph&&)+0x20) [0x7efdbe362950]
[bt] (8) /workspace/nnvm/python/nnvm/../../../build/libnnvm_compiler.so(nnvm::ApplyPasses(nnvm::Graph, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&)+0x32b) [0x7efdbe35afcb]
[bt] (9) /workspace/nnvm/python/nnvm/../../../build/libnnvm_compiler.so(NNGraphApplyPasses+0x348) [0x7efdbe32c998]

Script2

======================================================================
ERROR: test_top_level4.test_reduce
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/workspace/nnvm/tests/python/compiler/test_top_level4.py", line 98, in test_reduce
    verify_reduce((4, 4, 3), np.sum, sym.sum)
  File "/workspace/nnvm/tests/python/compiler/test_top_level4.py", line 56, in verify_reduce
    verify_reduce_explicit(dshape, data, result, fsym, oshape=oshape, otype=otype, **kwargs)
  File "/workspace/nnvm/tests/python/compiler/test_top_level4.py", line 37, in verify_reduce_explicit
    graph, lib, _ = nnvm.compiler.build(y, target, {"x": dshape})
  File "/workspace/nnvm/python/nnvm/compiler/build_module.py", line 305, in build
    graph = graph.apply("GraphCompile")
  File "/workspace/nnvm/python/nnvm/graph.py", line 234, in apply
    check_call(_LIB.NNGraphApplyPasses(self.handle, npass, cpass, ctypes.byref(ghandle)))
  File "/workspace/nnvm/python/nnvm/_base.py", line 75, in check_call
    raise NNVMError(py_str(_LIB.NNGetLastError()))
NNVMError: [10:30:32] /workspace/nnvm/src/compiler/compile_engine.cc:212: Check failed: out[i].ndim() == out_info[i].ndim() (0 vs. 1) sum



Stack trace returned 10 entries:
[bt] (0) /workspace/build/libtvm.so(dmlc::StackTrace[abi:cxx11]()+0x1bc) [0x7efdd902998c]
[bt] (1) /workspace/build/libtvm.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x28) [0x7efdd902a568]
[bt] (2) /workspace/nnvm/python/nnvm/../../../build/libnnvm_compiler.so(nnvm::compiler::CompileEngine::GetScheduleArgs(nnvm::Graph, tvm::Array<tvm::Tensor, void> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, tvm::Array<tvm::Tensor, void>*)+0x179d) [0x7efdbe3bf38d]
[bt] (3) /workspace/nnvm/python/nnvm/../../../build/libnnvm_compiler.so(nnvm::compiler::CompileEngine::DoLower(nnvm::Graph, tvm::Array<tvm::Tensor, void> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)+0xfd) [0x7efdbe3c0d5d]
[bt] (4) /workspace/nnvm/python/nnvm/../../../build/libnnvm_compiler.so(nnvm::compiler::CompileEngine::Lower(nnvm::Graph, tvm::Array<tvm::Tensor, void> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)+0x1bf) [0x7efdbe3c1e3f]
[bt] (5) /workspace/nnvm/python/nnvm/../../../build/libnnvm_compiler.so(nnvm::compiler::GraphLower(nnvm::Graph, tvm::Array<tvm::Tensor, void> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)+0x5f) [0x7efdbe3b6e1f]
[bt] (6) /workspace/nnvm/python/nnvm/../../../build/libnnvm_compiler.so(nnvm::compiler::GraphCompile(nnvm::Graph const&)+0xd7b) [0x7efdbe3c4e2b]
[bt] (7) /workspace/nnvm/python/nnvm/../../../build/libnnvm_compiler.so(std::_Function_handler<nnvm::Graph (nnvm::Graph), nnvm::Graph (*)(nnvm::Graph const&)>::_M_invoke(std::_Any_data const&, nnvm::Graph&&)+0x20) [0x7efdbe362950]
[bt] (8) /workspace/nnvm/python/nnvm/../../../build/libnnvm_compiler.so(nnvm::ApplyPasses(nnvm::Graph, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&)+0x32b) [0x7efdbe35afcb]
[bt] (9) /workspace/nnvm/python/nnvm/../../../build/libnnvm_compiler.so(NNGraphApplyPasses+0x348) [0x7efdbe32c998]

[sgrechanik-h]

Since ac7b420 and 543c424 topi and nnvm have different reduction behaviors, and it seems they cannot be used together reliably. Related discussion: https://discuss.tvm.ai/t/nnvm-reduction-is-broken/857

[siju-samuel]

@sgrechanik-h Thanks for the info
#2091 CI is failing because of this issue.
nnvm/tests/python/compiler/test_top_level1.py and nnvm/tests/python/compiler/test_top_level4.py test scripts are failing.

[tqchen]

This should be fixed by #2147

[tqchen]

The major cause is that originally we redefine the behavior via macro, and this will generate two conflicting function symbols which get resolved to one

[siju-samuel]

Thanks @tqchen @sgrechanik-h
This issue is now resolved.