`batch_dot` operator crash

[matteosal]

batch_dot seems completely broken.

import mxnet as mx

sym = mx.sym.batch_dot(mx.sym.Variable('in1'), mx.sym.Variable('in2'))

ex = sym._bind(
	mx.cpu(), 
	{'in1': mx.nd.ones((2, 3, 4)), 'in2': mx.nd.ones((2, 4, 5))}
)
ex.forward()

Running this script produces:

[14:24:17] /home/matteo/Git/mxnet-build/Build/Linux-x86-64/MKL/mxnet/src/storage/storage.cc:199: Using Pooled (Naive) StorageManager for CPU

Fatal Error: Segmentation fault

Fatal Error: Segmentation fault

Fatal Error: Segmentation fault

Fatal Error: Segmentation fault

Fatal Error: Segmentation fault

Fatal Error: Segmentation fault

Fatal Error: Segmentation fault
Stack trace:
Stack trace:
Segmentation fault (core dumped)

Maybe the problem is in my build. I'm building master from source with these settings (linking to MKL 2019.4):

 `# GENERAL FLAGS` \
 -DCMAKE_INSTALL_PREFIX=$output_dir \
 -DCMAKE_BUILD_TYPE=Release \
 -DCMAKE_SKIP_BUILD_RPATH=On \
 -DUSE_OPENCV=OFF \
 -DUSE_F16C=Off `# float16 support`\
 -DUSE_INT64_TENSOR_SIZE=OFF \
 -DCMAKE_C_FLAGS_RELEASE="-DNDEBUG" \
 -DCMAKE_CXX_FLAGS_RELEASE="-DNDEBUG" \
 `# MATH BACKENDS` \
 -DBLAS=MKL \
 -DUSE_LAPACK=OFF \
 -DUSE_ONEDNN=OFF \
 -DBLA_VENDOR="Intel10_64ilp" \
 -DBLA_STATIC=OFF \
 -DMKL_USE_SINGLE_DYNAMIC_LIBRARY=OFF \
 -DMKL_INCLUDE_DIR=$mkl_dir \
 -DBLAS_LIBRARIES="$mkl_dir/libmkl_def.so;$mkl_dir/libmkl_intel_ilp64.so;$mkl_dir/libmkl_core.so;$mkl_dir/libmkl_intel_thread.so;$mkl_dir/libiomp5.so" \
 `# OPENMP` \
 -DUSE_OPENMP=ON \
 -DOpenMP_C_FLAGS="-I$mkl_dir" \
 -DOpenMP_C_LIB_NAMES="libiomp5" \
 -DOpenMP_CXX_FLAGS="-I$mkl_dir" \
 -DOpenMP_CXX_LIB_NAMES="libiomp5" \
 -DOpenMP_libiomp5_LIBRARY="$mkl_dir/libiomp5.so" \
 `# CUDA` \
 -DUSE_CUDA=OFF \

[anko-intel]

@bartekkuncer please take a look

[matteosal]

Any news on this?

[bartekkuncer]

Hello @matteosal , sorry for the late response. I tried to reproduce the issue on master branch but had no success. Which branch are you working on?

[matteosal]

I have updated to the latest master and still see the crash. Are you using the same build settings I reported?

[bartekkuncer]

@matteosal almost. The only difference is I used newer version of mkl. Will try with yours.

[bartekkuncer]

@matteosal I tried to reproduce your bug using your exact build config but was unable to. I tried your version of MKL and also a newer and older one - all worked without any problems :(

Please try running your test with MKL_VERBOSE flag set e.g. MKL_VERBOSE=1 test.py and paste the output in the comment. Please also provide me with the information about what exact OS are you using, wherefrom/how you got your mkl library and which compiler are you using.

[matteosal]

Setting MKL_VERBOSE=1 doesn't change anything.

I have also tried to build with mkl 2020.1 and got the same crash + some symbol lookup issue:

[16:43:33] /home/matteo/Git/mxnet-build/Build/Linux-x86-64/MKL/mxnet/src/storage/storage.cc:199: Using Pooled (Naive) StorageManager for CPU

Fatal Error: Segmentation fault

Fatal Error: Segmentation fault

Fatal Error: Segmentation fault

Fatal Error: Segmentation fault

Fatal Error: Segmentation fault

Fatal Error: Segmentation fault

Fatal Error: Segmentation fault
INTEL MKL ERROR: /home/matteo/Git/mxnet-build/Build/Linux-x86-64/MKL/mkl/libmkl_avx2.so: undefined symbol: mkl_sparse_optimize_bsr_trsm_i8.
Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.
Stack trace:
Segmentation fault (core dumped)

Which makes me think that I'm doing something wrong for this one.

But: since MKL_VERBOSE=1 doesn't change anything and my symbol lookup messages show up after the seg faults, maybe the problem is encountered before reaching MKL?

[bartekkuncer]

What OS and compiler are you using?

[matteosal]

Linux/gcc

[bgawrych]

@matteosal Have you tried using

export LD_PRELOAD=$mkl_dir/libmkl_def.so;$mkl_dir/libmkl_intel_ilp64.so;$mkl_dir/libmkl_core.so;$mkl_dir/libmkl_intel_thread.so;$mkl_dir/libiomp5.so

?
(Use proper $mkl_dir path)

Also can you point exact OS version?

[matteosal]

@bgawrych I rebuilt everything from scratch with both mkl 2019.4 and 2020.1 and now I see the same exact behaviour (segfault messages without symbol lookup errors)

Full versions are Ubuntu 20.04 + gcc 9.3.0

[bgawrych]

@matteosal Are you sure that your $mkl_dir path is proper one? In my environment I have
-DMKL_INCLUDE_DIR=/home/bg/miniconda3/envs/mxnet/include
-DBLAS_LIBRARIES="/home/bg/miniconda3/envs/mxnet/lib/libmkl_def.so;
    /home/bg/miniconda3/envs/mxnet/lib/libmkl_intel_ilp64.so;
    /home/bg/miniconda3/envs/mxnet/lib/libmkl_core.so;
    /home/bg/miniconda3/envs/mxnet/lib/libmkl_intel_thread.so;
    /home/bg/miniconda3/envs/mxnet/lib/libiomp5.so"

Notice that include_dir have different path than BLAS_LIBRARIES lib vs include

[matteosal]

@matteosal Are you sure that your $mkl_dir path is proper one? In my environment I have
-DMKL_INCLUDE_DIR=/home/bg/miniconda3/envs/mxnet/include
-DBLAS_LIBRARIES="/home/bg/miniconda3/envs/mxnet/lib/libmkl_def.so;
    /home/bg/miniconda3/envs/mxnet/lib/libmkl_intel_ilp64.so;
    /home/bg/miniconda3/envs/mxnet/lib/libmkl_core.so;
    /home/bg/miniconda3/envs/mxnet/lib/libmkl_intel_thread.so;
    /home/bg/miniconda3/envs/mxnet/lib/libiomp5.so"

Notice that include_dir have different path than BLAS_LIBRARIES lib vs include

Yes that's because for some reason our reference internal MKL checkout has a non-standard file layout, all includes and libraries are dumped into the same folder ($mkl_dir in my script).
If there was a problem with this it would have failed at build time, and we have been building and intensively using mxnet with this setup for years. Also I don't see any symbol lookup errors anymore, so I'd rule out that something is wrong with the MKL path.

@bgawrych can you suggest another python which uses MKL primitives that I can try out? It should give us more information