libparquet.1.0.0.dylib does not exist

[mrandrewandrade]

I recently got the build for source working using the instructions from here: https://arrow.apache.org/docs/python/development.html#development

I am able to compile the first time, but when I try and recompile I get an error that libparquet.1.0.0.dylib doesn't exist (there is libparquet.1.dylib and a later version). I also reproed the recompile error on CentOS with both PIP and conda installs. I am able to solve the issue using a symlink. Do you think this is an issue with my system/setup or is an issue I should create on JIRA?

(pyarrow) aandrade54-mac:python aandrade$ python setup.py build_ext --build-type=$ARROW_BUILD_TYPE \
>        --with-parquet --with-plasma --bundle-arrow-cpp bdist_wheel
running build_ext
creating build
creating build/temp.macosx-10.6-x86_64-2.7
-- Runnning cmake for pyarrow
cmake  -DPYTHON_EXECUTABLE=/Users/aandrade/arrow-dev/repos/pyarrow/bin/python  -DPYARROW_BUILD_PARQUET=on -DPYARROW_BUILD_PLASMA=on -DPYARROW_BUNDLE_ARROW_CPP=ON -DCMAKE_BUILD_TYPE=release /Users/aandrade/arrow-dev/repos/arrow/python
-- The C compiler identification is AppleClang 9.0.0.9000038
-- The CXX compiler identification is AppleClang 9.0.0.9000038
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
INFOCompiler command: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++
INFOCompiler version: Apple LLVM version 9.0.0 (clang-900.0.38)
Target: x86_64-apple-darwin16.7.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

INFOCompiler id: Clang
Selected compiler clang 4.0.0svn
-- Performing Test CXX_SUPPORTS_SSE3
-- Performing Test CXX_SUPPORTS_SSE3 - Success
-- Performing Test CXX_SUPPORTS_ALTIVEC
-- Performing Test CXX_SUPPORTS_ALTIVEC - Failed
Configured for RELEASE build (set with cmake -DCMAKE_BUILD_TYPE={release,debug,...})
-- Build Type: RELEASE
-- Build output directory: /Users/aandrade/arrow-dev/repos/arrow/python/build/temp.macosx-10.6-x86_64-2.7/release/
-- Found PythonInterp: /Users/aandrade/arrow-dev/repos/pyarrow/bin/python (found version "2.7.14")
-- Found PythonLibs: /Users/aandrade/anaconda2/lib/libpython2.7.dylib
-- Found NumPy: version "1.13.3" /Users/aandrade/arrow-dev/repos/pyarrow/lib/python2.7/site-packages/numpy/core/include
-- Found Cython: /Users/aandrade/arrow-dev/repos/pyarrow/bin/cython
-- Found PkgConfig: /usr/local/bin/pkg-config (found version "0.29.2")
-- Checking for module 'arrow'
--   Found arrow, version 0.8.0-SNAPSHOT
-- Arrow ABI version: 0.0.0
-- Arrow SO version: 0
-- Found the Arrow core library: /Users/aandrade/arrow-dev/repos/dist/lib/libarrow.dylib
-- Found the Arrow Python library: /Users/aandrade/arrow-dev/repos/dist/lib/libarrow_python.dylib
Added shared library dependency arrow: /Users/aandrade/arrow-dev/repos/dist/lib/libarrow.dylib
Added shared library dependency arrow_python: /Users/aandrade/arrow-dev/repos/dist/lib/libarrow_python.dylib
-- Found the Parquet library: /Users/aandrade/arrow-dev/repos/dist/lib/libparquet.dylib
CMake Error: File /Users/aandrade/arrow-dev/repos/dist/lib/libparquet.1.0.0.dylib does not exist.
CMake Error at CMakeLists.txt:207 (configure_file):
  configure_file Problem configuring file
Call Stack (most recent call first):
  CMakeLists.txt:290 (bundle_arrow_lib)


Added shared library dependency parquet: /Users/aandrade/arrow-dev/repos/dist/lib/libparquet.dylib
-- Checking for module 'plasma'
--   Found plasma, version
-- Plasma ABI version: 0.0.0
-- Plasma SO version: 0
-- Found the Plasma core library: /Users/aandrade/arrow-dev/repos/dist/lib/libplasma.dylib
-- Found Plasma executable: /Users/aandrade/arrow-dev/repos/dist/bin/plasma_store
Added shared library dependency libplasma: /Users/aandrade/arrow-dev/repos/dist/lib/libplasma.dylib
-- Configuring incomplete, errors occurred!
See also "/Users/aandrade/arrow-dev/repos/arrow/python/build/temp.macosx-10.6-x86_64-2.7/CMakeFiles/CMakeOutput.log".
See also "/Users/aandrade/arrow-dev/repos/arrow/python/build/temp.macosx-10.6-x86_64-2.7/CMakeFiles/CMakeError.log".
error: command 'cmake' failed with exit status 1

The solution here was to create a sym link:

ln -s /Users/aandrade/arrow-dev/repos/dist/lib/libparquet.1.dylib /Users/aandrade/arrow-dev/repos/dist/lib/libparquet.1.0.0.dylib

[xhochy]

The problem is that we have hard-coded the version in the case one sets PARQUET_HOME.

We normally use the option --bundle-arrow-cpp only for builds that discover Arrow C++ and Parquet C++ via PKG_CONFIG_PATH and not using the *_HOME variables. If you want to build wheels that can be passed around, I would advise you to use the pkg-config based discovery. Otherwise, if you only want a local development version, omit --bundle-arrow-cpp.

[xhochy]

Opened https://issues.apache.org/jira/browse/ARROW-1856 about adding auto-detection for the ABI version in the case that PARQUET_HOME is set.

[mrandrewandrade]

This is helpful! Should we create a PR to explain --bundle-arrow-cpp and the other configs? I don't see anywhere in the documentation on what the configs mean (but I only did a quick search)

[xhochy]

Yes, we should somewhere document these options, a PR is welcome. --bundle-arrow-cpp will include all libraries (Arrow C++, Parquet C++) into the wheel so it is self-contained.

[mrandrewandrade]

Cool, I have an open PR with documentation fixes, I'll include this.

[mrandrewandrade]

#1356

[wesm]

Is this fixed by https://issues.apache.org/jira/browse/ARROW-1856?

[xhochy]

This is fixed with https://issues.apache.org/jira/browse/ARROW-1856