[Python] Python C++ libraries no longer in pyarrow/lib directory

[wjones127]

Describe the bug, including details regarding any error messages, version, and platform.

Prior to #14925, libraries such as arrow_python.dylib were installed in both python/pyarrow and python/pyarrow/lib. Afterward, they are only present in python/pyarrow, and when I build in place and run import pyarrow I get:

ImportError: dlopen(/Users/willjones/Documents/arrows/arrow/python/pyarrow/lib.cpython-310-darwin.so, 0x0002): Library not loaded: '/Users/willjones/Documents/arrows/arrow/python/pyarrow/lib/libarrow_python.1100.dylib'

It seems my dev environment relied on that lib directory and I suspect I am not alone.

Component(s)

Python

[wjones127]

> otool pyarrow/lib.cpython-310-darwin.so -L
pyarrow/lib.cpython-310-darwin.so:
        /Users/willjones/Documents/arrows/arrow/python/pyarrow/lib/libarrow_python.1100.dylib (compatibility version 1100.0.0, current version 1100.0.0)
        /Users/willjones/mambaforge/envs/arrow-dev/lib/libarrow.1100.dylib (compatibility version 1100.0.0, current version 1100.0.0)
        @rpath/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.100.3)

[wjones127]

Workaround for now (on MacOS): after building,

mkdir -p pyarrow/lib && cp pyarrow/*.dylib pyarrow/lib/

[kou]

Related:

• ARROW-17838: [Python] Unify CMakeLists.txt in python/ #14925 (comment)
• [Benchmarking][Python] Benchmark builds on test-mac-arm are failing #15122

I think that we should always use @rpath even when Apache Arrow C++ is built with -DARROW_INSTALL_NAME_RPATH=NO.