Flaky test: test_gluon.test_hybrid_static_memory_switching

[ThomasDelteil]

test_gluon.test_hybrid_static_memory_switching

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/PR-11024/6/pipeline/

t_gluon.test_hybrid_static_memory_switching ... /work/runtime_functions.sh: line 382:     7 Segmentation fault      (core dumped) nosetests-3.4 --verbose tests/python/unittest

build.py: 2018-06-06 13:18:39,834 Running of command in container failed (139): docker run --rm -t --shm-size=500m -v /home/jenkins_slave/workspace/ut-python3-mkldnn-cpu:/work/mxnet -v /home/jenkins_slave/workspace/ut-python3-mkldnn-cpu/build:/work/build -u 1001:1001 mxnetci/build.ubuntu_cpu /work/runtime_functions.sh unittest_ubuntu_python3_cpu_mkldnn

build.py: 2018-06-06 13:18:39,835 You can try to get into the container by using the following command: docker run --rm -t --shm-size=500m -v /home/jenkins_slave/workspace/ut-python3-mkldnn-cpu:/work/mxnet -v /home/jenkins_slave/workspace/ut-python3-mkldnn-cpu/build:/work/build -u 1001:1001 -ti --entrypoint /bin/bash mxnetci/build.ubuntu_cpu /work/runtime_functions.sh unittest_ubuntu_python3_cpu_mkldnn

into container: False

Traceback (most recent call last):

  File "ci/build.py", line 318, in <module>

    sys.exit(main())

  File "ci/build.py", line 253, in main

    command=command, docker_registry=docker_registry)

  File "ci/build.py", line 155, in container_run

    raise subprocess.CalledProcessError(ret, cmd)

subprocess.CalledProcessError: Command 'docker run --rm -t --shm-size=500m -v /home/jenkins_slave/workspace/ut-python3-mkldnn-cpu:/work/mxnet -v /home/jenkins_slave/workspace/ut-python3-mkldnn-cpu/build:/work/build -u 1001:1001 mxnetci/build.ubuntu_cpu /work/runtime_functions.sh unittest_ubuntu_python3_cpu_mkldnn' returned non-zero exit status 139

script returned exit code 1

[kalyc]

Thanks for submitting this issue @ThomasDelteil

[ThomasDelteil]

@piiswrong you introduced this test in this commit

[WIP] Do Not Merge. Static memory allocation for cached_op (#10817) 2dbd143

and it seems to be flaky. I have seen it failing a few times in recent builds. Can you take a look? e.g
http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/master/1002/pipeline/

Given all the talk about quality contributions going on the dev mailing list, I am a little unsettled by the fact this PR went undocumented (no design docs or explanation in the PR), unreviewed (1 question ignored), the optimization wasn't tested or benchmarked (it was actually making slower), and the code was self-merged.

Should we enforce that each PR , especially the ones that introduce a significant number of changes be properly documented and reviewed before merging?

@szha @marcoabreu

[zheng-da]

I ran the test thousands of times. It doesn't appear frequently. I checked core dumps. It seems they fail all exactly in the same place.

#0  0x00007ffff1c5c510 in typeinfo for mkldnn::impl::cpu::cpu_primitive_t () from /home/ubuntu/incubator-mxnet/lib/libmkldnn.so.0
#1  0x00007ffff15c1372 in mkldnn::impl::cpu::jit_uni_reorder_t::execute(mkldnn::impl::event_t*) ()
   from /home/ubuntu/incubator-mxnet/lib/libmkldnn.so.0
#2  0x00007ffff16f8293 in mkldnn::impl::cpu::cpu_engine_t::submit(mkldnn_primitive*, mkldnn::impl::event_t*, mkldnn::impl::nstl::vector<mkldnn::impl::event_t*>&) () from /home/ubuntu/incubator-mxnet/lib/libmkldnn.so.0
#3  0x00007ffff15741c6 in mkldnn::impl::stream_eager_t::submit_impl(unsigned long, unsigned long, mkldnn_primitive**) ()
   from /home/ubuntu/incubator-mxnet/lib/libmkldnn.so.0
#4  0x00007ffff15735b1 in mkldnn_stream::submit(mkldnn::impl::nstl::vector<mkldnn_primitive*> const&, mkldnn_primitive**) ()
   from /home/ubuntu/incubator-mxnet/lib/libmkldnn.so.0
#5  0x00007ffff15737c8 in mkldnn_stream_submit () from /home/ubuntu/incubator-mxnet/lib/libmkldnn.so.0
#6  0x00007fffcac89608 in mkldnn::stream::submit(std::vector<mkldnn::primitive, std::allocator<mkldnn::primitive> >) ()
   from /home/ubuntu/incubator-mxnet/lib/libmxnet.so
#7  0x00007fffcac8cfc2 in mxnet::MKLDNNMemory::ReorderTo(mkldnn::memory*) const ()
   from /home/ubuntu/incubator-mxnet/lib/libmxnet.so
#8  0x00007fffcac791da in mxnet::NDArray::Reorder2Default() const () from /home/ubuntu/incubator-mxnet/lib/libmxnet.so
#9  0x00007fffc847911a in mxnet::FallBackCompute(std::function<void (nnvm::NodeAttrs const&, mxnet::OpContext const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&)>, nnvm::NodeAttrs const&, mxnet::OpContext const&, std::vector<mxnet::NDArray, std::allocator<mxnet::NDArray> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&, std::vector<mxnet::NDArray, std::allocator<mxnet::NDArray> > const&) () from /home/ubuntu/incubator-mxnet/lib/libmxnet.so
#10 0x00007fffca8b8696 in mxnet::op::PoolingComputeExCPU(nnvm::NodeAttrs const&, mxnet::OpContext const&, std::vector<mxnet::NDArray, std::allocator<mxnet::NDArray> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&, std::vector<mxnet::NDArray, std::allocator<mxnet::NDArray> > const&) () from /home/ubuntu/incubator-mxnet/lib/libmxnet.so
#11 0x00007fffcabf34c5 in std::_Function_handler<void (mxnet::RunContext), mxnet::imperative::PushFComputeEx(std::function<void (nnvm::NodeAttrs const&, mxnet::OpContext const&, std::vector<mxnet::NDArray, std::allocator<mxnet::NDArray> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&, std::vector<mxnet::NDArray, std::allocator<mxnet::NDArray> > const&)> const&, nnvm::Op const*, nnvm::NodeAttrs const&, mxnet::Context const&, std::vector<mxnet::engine::Var*, std::allocator<mxnet::engine::Var*> > const&, std::vector<mxnet::engine::Var*, std::allocator<mxnet::engine::Var*> > const&, std::vector<mxnet::Resource, std::allocator<mxnet::Resource> > const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&)::{la---Type <return> to continue, or q <return> to quit---
mbda(mxnet::RunContext)#1}>::_M_invoke(std::_Any_data const&, mxnet::RunContext&&) ()
   from /home/ubuntu/incubator-mxnet/lib/libmxnet.so
#12 0x00007fffcb0d6d88 in std::_Function_handler<void (mxnet::RunContext), mxnet::engine::ThreadedEngine::BulkAppend(std::function<void (mxnet::RunContext)>, mxnet::Context, std::vector<mxnet::engine::Var*, std::allocator<mxnet::engine::Var*> > const&, std::vector<mxnet::engine::Var*, std::allocator<mxnet::engine::Var*> > const&)::{lambda(mxnet::RunContext)#1}>::_M_invoke(std::_Any_data const&, mxnet::RunContext&&) () from /home/ubuntu/incubator-mxnet/lib/libmxnet.so
#13 0x00007fffcb0d6d67 in std::_Function_handler<void (mxnet::RunContext), mxnet::engine::ThreadedEngine::BulkAppend(std::function<void (mxnet::RunContext)>, mxnet::Context, std::vector<mxnet::engine::Var*, std::allocator<mxnet::engine::Var*> > const&, std::vector<mxnet::engine::Var*, std::allocator<mxnet::engine::Var*> > const&)::{lambda(mxnet::RunContext)#1}>::_M_invoke(std::_Any_data const&, mxnet::RunContext&&) () from /home/ubuntu/incubator-mxnet/lib/libmxnet.so
#14 0x00007fffcb0cb48b in std::_Function_handler<void (mxnet::RunContext, mxnet::engine::CallbackOnComplete), mxnet::engine::ThreadedEngine::BulkFlush()::{lambda(mxnet::RunContext, mxnet::engine::CallbackOnComplete)#1}>::_M_invoke(std::_Any_data const&, mxnet::RunContext&&, mxnet::engine::CallbackOnComplete&&) () from /home/ubuntu/incubator-mxnet/lib/libmxnet.so
#15 0x00007fffcb0ccd45 in mxnet::engine::ThreadedEngine::ExecuteOprBlock(mxnet::RunContext, mxnet::engine::OprBlock*) ()
   from /home/ubuntu/incubator-mxnet/lib/libmxnet.so
#16 0x00007fffcb0df459 in std::_Function_handler<void (std::shared_ptr<dmlc::ManualEvent>), mxnet::engine::ThreadedEnginePerDevice::PushToExecute(mxnet::engine::OprBlock*, bool)::{lambda()#1}::operator()() const::{lambda(std::shared_ptr<dmlc::ManualEvent>)#1}>::_M_invoke(std::_Any_data const&, std::shared_ptr<dmlc::ManualEvent>&&) () from /home/ubuntu/incubator-mxnet/lib/libmxnet.so
#17 0x00007fffcb0cc34a in std::thread::_Impl<std::_Bind_simple<std::function<void (std::shared_ptr<dmlc::ManualEvent>)> (std::shared_ptr<dmlc::ManualEvent>)> >::_M_run() () from /home/ubuntu/incubator-mxnet/lib/libmxnet.so
#18 0x00007fffe7f4dc80 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#19 0x00007ffff7bc16ba in start_thread (arg=0x7fff96154700) at pthread_create.c:333
#20 0x00007ffff78f741d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

[zheng-da]

@TaoLv @pengzhao-intel @ashokei @azai91 Could you help check why it always fails in the same place?

[ThomasDelteil]

@zheng-da which hybridize kwargs trigger the failure?

[zheng-da]

it seems the error only happens in static_alloc=True, static_shape=True.

[marcoabreu]

Does it only happen with MKLDNN or is it unrelated?

[pengzhao-intel]

If it's only related with MKLDNN, we will take over the issue. @marcoabreu @zheng-da

[piiswrong]

@pengzhao-intel Looks like it only fails when using MKLDNN. Could you help take a look?