[Python] manylinux2010 wheels have shared library dependency on liblz4

[asfimport]

I am using pyarrow in my project. It works well for version 0.13.0

However, it seems recently there is a release for 0.14.0. After upgrading to the latest, I got this error.
AttributeError: module 'pyarrow' has no attribute 'compat'

Stacktrace:
2019-07-06 09:08:21 Traceback (most recent call last):
2019-07-06 09:08:21 File "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/src/PerfTestRunner.py", line 12, in
2019-07-06 09:08:21 import snowflake.connector
2019-07-06 09:08:21 File "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/init.py", line 21, in
2019-07-06 09:08:21 from .connection import SnowflakeConnection
2019-07-06 09:08:21 File "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/connection.py", line 42, in
2019-07-06 09:08:21 from .cursor import SnowflakeCursor, LOG_MAX_QUERY_LENGTH
2019-07-06 09:08:21 File "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/snowflake/connector/cursor.py", line 35, in
2019-07-06 09:08:21 from pyarrow.ipc import open_stream
2019-07-06 09:08:21 File "/home/jenkins/workspace/CLIENTS_PERF/Tests/ClientsPerf/PythonConnectorPerf/pythonconnector-perf/lib/python3.5/site-packages/pyarrow/init.py", line 47, in
2019-07-06 09:08:21 import pyarrow.compat as compat

I can provide more detail if requested.

Reporter: Haowei Yu / @sfc-gh-hyu
Assignee: Wes McKinney / @wesm

PRs and other links:

• GitHub Pull Request #4828

_{Note: This issue was originally created as ARROW-5868. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
It looks as if import pyarrow is failing somehow. Can you try python -c "import pyarrow" and let us know if that works?

[asfimport]

Haowei Yu / @sfc-gh-hyu:
It failed with following:

 

(py36env) [hyu@c6x64-hyu-newuser-final-clone trunk]$ python -c "import pyarrow" 
Traceback (most recent call last): 
File "<string>", line 1, in <module> 
File "/home/hyu/py36env/lib/python3.6/site-packages/pyarrow/__init__.py", line 49, in <module> 
from pyarrow.lib import cpu_count, set_cpu_count 
ImportError: liblz4.so.1: cannot open shared object file: No such file or directory

[asfimport]

Wes McKinney / @wesm:
How did you install pyarrow, with pip or conda (it looks like with conda)?

[asfimport]

Haowei Yu / @sfc-gh-hyu:
I use pip, not using conda. I have a virtual env on python3.6. Is liblz4.so.1 bundled with pyarrow package? 

[asfimport]

Wes McKinney / @wesm:
No, it should be statically linked in the wheels, no runtime dependency

[asfimport]

Haowei Yu / @sfc-gh-hyu:
Then it is weird. I am not familiar with this but .so seems to be a dynamic linked library. Python somehow tries to import a dynamic library? Or maybe that is a secondary dependency?

[asfimport]

Wes McKinney / @wesm:
Something seems to be wrong with the C++ build, someone will have to take a closer look at the wheels themselves and the build to understand what is going on

[asfimport]

Wes McKinney / @wesm:
Confirmed that the manylinux2010 wheel at least has a shared library dependency on liblz4

$ ldd ~/miniconda/envs/pyarrow-0.14-wheel-3.6/lib/python3.6/site-packages/pyarrow/libarrow.so.14 
	linux-vdso.so.1 (0x00007ffe6974d000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f1de9c68000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f1de9c5d000)
	liblz4.so.1 => /lib/x86_64-linux-gnu/liblz4.so.1 (0x00007f1de9c2e000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f1de9c12000)
	libarrow_boost_system.so.1.68.0 => /home/wesm/miniconda/envs/pyarrow-0.14-wheel-3.6/lib/python3.6/site-packages/pyarrow/libarrow_boost_system.so.1.68.0 (0x00007f1de99fc000)
	libarrow_boost_filesystem.so.1.68.0 => /home/wesm/miniconda/envs/pyarrow-0.14-wheel-3.6/lib/python3.6/site-packages/pyarrow/libarrow_boost_filesystem.so.1.68.0 (0x00007f1de97df000)
	libarrow_boost_regex.so.1.68.0 => /home/wesm/miniconda/envs/pyarrow-0.14-wheel-3.6/lib/python3.6/site-packages/pyarrow/libarrow_boost_regex.so.1.68.0 (0x00007f1de94e3000)
	libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f1de9301000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f1de91b3000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f1de9199000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1de9178000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1de8f8b000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f1deab1e000)

Checking manylinux1, then will look at the Docker image to see what's going on

[asfimport]

Wes McKinney / @wesm:
manylinux1 wheels are OK. I'm going to check some other things, but I'm probably going to revoke the manylinux2010 wheels

[asfimport]

Wes McKinney / @wesm:
manylinux2010 wheels have been pulled from PyPI. If you reinstall and get the manylinux1 wheel it should work for you

[asfimport]

Haowei Yu / @sfc-gh-hyu:
Ok I tried and it seems working. Thanks for the quick response.

[asfimport]

Wes McKinney / @wesm:
Issue resolved by pull request 4828
#4828