[Python] Segfault when importing TensorFlow after Pyarrow

[asfimport]

This issue is similar to https://jira.apache.org/jira/browse/ARROW-2657 which was fixed in v0.10.0.

When we import TensorFlow after Pyarrow in Linux Debian Jessie, we get a segfault.  To reproduce:

import pyarrow 
import tensorflow

Here's the backtrace from gdb:

Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0000000000000000 in ?? ()
(gdb) bt
#0 0x0000000000000000 in ?? ()
#1 0x00007f529ee04410 in pthread_once () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_once.S:103
#2 0x00007f5229a74efa in void std::call_once<void (&)()>(std::once_flag&, void (&)()) () from /usr/local/lib/python2.7/dist-packages/tensorflow/python/../libtensorflow_framework.so
#3 0x00007f5229a74f3e in tensorflow::port::TestCPUFeature(tensorflow::port::CPUFeature) () from /usr/local/lib/python2.7/dist-packages/tensorflow/python/../libtensorflow_framework.so
#4 0x00007f522978b561 in tensorflow::port::(anonymous namespace)::CheckFeatureOrDie(tensorflow::port::CPUFeature, std::string const&) ()
from /usr/local/lib/python2.7/dist-packages/tensorflow/python/../libtensorflow_framework.so
#5 0x00007f522978b5b4 in _GLOBAL__sub_I_cpu_feature_guard.cc () from /usr/local/lib/python2.7/dist-packages/tensorflow/python/../libtensorflow_framework.so
#6 0x00007f529f224bea in call_init (l=<optimized out>, argc=argc@entry=9, argv=argv@entry=0x7ffc6d8c1488, env=env@entry=0x294c0c0) at dl-init.c:78
#7 0x00007f529f224cd3 in call_init (env=0x294c0c0, argv=0x7ffc6d8c1488, argc=9, l=<optimized out>) at dl-init.c:36
#8 _dl_init (main_map=main_map@entry=0x2e4aff0, argc=9, argv=0x7ffc6d8c1488, env=0x294c0c0) at dl-init.c:126
#9 0x00007f529f228e38 in dl_open_worker (a=a@entry=0x7ffc6d8bebb8) at dl-open.c:577
#10 0x00007f529f224aa4 in _dl_catch_error (objname=objname@entry=0x7ffc6d8beba8, errstring=errstring@entry=0x7ffc6d8bebb0, mallocedp=mallocedp@entry=0x7ffc6d8beba7,
operate=operate@entry=0x7f529f228b60 <dl_open_worker>, args=args@entry=0x7ffc6d8bebb8) at dl-error.c:187
#11 0x00007f529f22862b in _dl_open (file=0x7f5248178b54 "/usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow_internal.so", mode=-2147483646, caller_dlopen=<optimized out>,
nsid=-2, argc=9, argv=0x7ffc6d8c1488, env=0x294c0c0) at dl-open.c:661
#12 0x00007f529ebf402b in dlopen_doit (a=a@entry=0x7ffc6d8bedd0) at dlopen.c:66
#13 0x00007f529f224aa4 in _dl_catch_error (objname=0x2950fc0, errstring=0x2950fc8, mallocedp=0x2950fb8, operate=0x7f529ebf3fd0 <dlopen_doit>, args=0x7ffc6d8bedd0) at dl-error.c:187
#14 0x00007f529ebf45dd in _dlerror_run (operate=operate@entry=0x7f529ebf3fd0 <dlopen_doit>, args=args@entry=0x7ffc6d8bedd0) at dlerror.c:163
#15 0x00007f529ebf40c1 in __dlopen (file=<optimized out>, mode=<optimized out>) at dlopen.c:87
#16 0x0000000000540859 in _PyImport_GetDynLoadFunc ()
#17 0x000000000054024c in _PyImport_LoadDynamicModule ()
#18 0x00000000005f2bcb in ?? ()
#19 0x00000000004ca235 in PyEval_EvalFrameEx ()
#20 0x00000000004ca9c2 in PyEval_EvalFrameEx ()
#21 0x00000000004c8c39 in PyEval_EvalCodeEx ()
#22 0x00000000004c84e6 in PyEval_EvalCode ()
#23 0x00000000004c6e5c in PyImport_ExecCodeModuleEx ()
#24 0x00000000004c3272 in ?? ()
#25 0x00000000004b19e2 in ?? ()
#26 0x00000000004b13d7 in ?? ()
#27 0x00000000004b42f6 in ?? ()
#28 0x00000000004d1aab in PyEval_CallObjectWithKeywords ()
#29 0x00000000004ccdb3 in PyEval_EvalFrameEx ()
#30 0x00000000004c8c39 in PyEval_EvalCodeEx ()
#31 0x00000000004c84e6 in PyEval_EvalCode ()
#32 0x00000000004c6e5c in PyImport_ExecCodeModuleEx ()
#33 0x00000000004c3272 in ?? ()
#34 0x00000000004b1d3f in ?? ()
#35 0x00000000004b6b2b in ?? ()
#36 0x00000000004b0d82 in ?? ()
#37 0x00000000004b42f6 in ?? ()
#38 0x00000000004d1aab in PyEval_CallObjectWithKeywords ()
#39 0x00000000004ccdb3 in PyEval_EvalFrameEx ()

It looks like the code changes that fixed the previous issue was recently removed in b766bff#diff-16806bcebc1df2fae432db426905b9f0.

Reporter: Travis Addair
Assignee: Alexander Sergeev

PRs and other links:

• GitHub Pull Request #4232

_{Note: This issue was originally created as ARROW-5130. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
The recommendation for using the libraries together is to use conda to install. You can also try importing tensorflow first. We are awaiting TensorFlow and others developing a compatible wheel standard we can all use. TensorFlow's wheels are non-conforming to the manylinux1 standard

[asfimport]

Alexander Sergeev:
One workaround we found is to LD_PRELOAD /usr/lib/x86_64-linux-gnu/libstdc++.so.6.

 

Wes, is there a reason PyArrow re-exports a bunch of C++ std library symbols?

[asfimport]

Wes McKinney / @wesm:
We aren't doing so on purpose

[asfimport]

Alexander Sergeev:
Wes, would you take a PR that cleans these things up?

 

# for f in $(ls -1 /usr/local/lib/python2.7/dist-packages/pyarrow/*.so*); do echo $f; nm -D $f | c++filt | grep std::_Hash_bytes; done
/usr/local/lib/python2.7/dist-packages/pyarrow/_csv.so
U std::_Hash_bytes(void const*, unsigned long, unsigned long)
/usr/local/lib/python2.7/dist-packages/pyarrow/libarrow_boost_filesystem.so
/usr/local/lib/python2.7/dist-packages/pyarrow/libarrow_boost_filesystem.so.1.66.0
/usr/local/lib/python2.7/dist-packages/pyarrow/libarrow_boost_regex.so
/usr/local/lib/python2.7/dist-packages/pyarrow/libarrow_boost_regex.so.1.66.0
/usr/local/lib/python2.7/dist-packages/pyarrow/libarrow_boost_system.so
/usr/local/lib/python2.7/dist-packages/pyarrow/libarrow_boost_system.so.1.66.0
/usr/local/lib/python2.7/dist-packages/pyarrow/libarrow_python.so
00000000000e2250 T std::_Hash_bytes(void const*, unsigned long, unsigned long)
/usr/local/lib/python2.7/dist-packages/pyarrow/libarrow_python.so.13
00000000000e2250 T std::_Hash_bytes(void const*, unsigned long, unsigned long)
/usr/local/lib/python2.7/dist-packages/pyarrow/libarrow.so
/usr/local/lib/python2.7/dist-packages/pyarrow/libarrow.so.13
/usr/local/lib/python2.7/dist-packages/pyarrow/libparquet.so
00000000001ce380 T std::_Hash_bytes(void const*, unsigned long, unsigned long)
/usr/local/lib/python2.7/dist-packages/pyarrow/libparquet.so.13
00000000001ce380 T std::_Hash_bytes(void const*, unsigned long, unsigned long)
/usr/local/lib/python2.7/dist-packages/pyarrow/libplasma.so
/usr/local/lib/python2.7/dist-packages/pyarrow/libplasma.so.13
/usr/local/lib/python2.7/dist-packages/pyarrow/lib.so
U std::_Hash_bytes(void const*, unsigned long, unsigned long)
/usr/local/lib/python2.7/dist-packages/pyarrow/libz-7f57503f.so.1.2.11
/usr/local/lib/python2.7/dist-packages/pyarrow/_orc.so
/usr/local/lib/python2.7/dist-packages/pyarrow/_parquet.so
U std::_Hash_bytes(void const*, unsigned long, unsigned long)
/usr/local/lib/python2.7/dist-packages/pyarrow/_plasma.so

 

[asfimport]

Wes McKinney / @wesm:
defer to @pitrou who worked on the symbols files most recently

[asfimport]

Antoine Pitrou / @pitrou:
"_Hash_bytes" isn't exported, it's an undefined symbol ("U") that's referred to by libarrow.so. I'm not sure what the problem is.

By the way, I suggest you try with Arrow master, since the exported symbols were cleaned up recently there.

[asfimport]

Alexander Sergeev:
@antoine, you're right that the symbol I mentioned is not exported by libarrow.so, but it's exported by libparquet.so and libarrow_python.so.

I'll try the master to check if there are any improvements.

[asfimport]

Francois Saint-Jacques / @fsaintjacques:
This is not fixed with in master, I fetched the latest wheel from crossbow and can trigger the segfault. I think that propagating the fix in ARROW-2796 to all libraries (parquet, plasma, gandiva) will do the trick.

[asfimport]

Francois Saint-Jacques / @fsaintjacques:
Also note that I can't trigger this from a local build, I suspect that this is related to how we build/package the wheel.

[asfimport]

Antoine Pitrou / @pitrou:
@fsaintjacques The reproducer only involved PyArrow and Tensorflow. Not Parquet or Gandiva. You may want to try your suggestion but I doubt it will fix anything.

[asfimport]

Antoine Pitrou / @pitrou:
Yes, it's an ABI issue. Put simply Tensorflow are publishing invalid wheels that don't comply to the specification (but tend to "work" in situations simple enough), because they rely on the wrong compiler and runtime.

[asfimport]

Alexander Sergeev:
@fsaintjacques is the build process for wheels that end up in PyPI documented somewhere, so I could reproduce the issue locally with containers & spread the #2096 around?

[asfimport]

Francois Saint-Jacques / @fsaintjacques:
It's a component called crossbow, the gist of what you need is here

[asfimport]

Francois Saint-Jacques / @fsaintjacques:
You'll have to replicate https://github.com/apache/arrow/blob/master/dev/tasks/python-wheels/travis.linux.yml

[asfimport]

Wes McKinney / @wesm:
See https://github.com/apache/arrow/tree/master/python/manylinux1