[C++][Python][Docs] Provide instructions for "fixing" build environment issues

[asfimport]

Due to bigger changes in the build workflow for Arrow C++ coming up in the 10.0.0 release, failures when building the libraries are quite common. The errors we bump into are similar to:

    CMake Error at build/dist/lib/cmake/ArrowPython/ArrowPythonConfig.cmake:61 (arrow_keep_backward_compatibility):
      Unknown CMake command "arrow_keep_backward_compatibility".
    Call Stack (most recent call first):
      CMakeLists.txt:240 (find_package)

or

-- Found Python3Alt: /Users/alenkafrim/repos/pyarrow-dev-9/bin/python  
CMake Error at /opt/homebrew/Cellar/cmake/3.24.1/share/cmake/Modules/CMakeFindDependencyMacro.cmake:47 (find_package):
  By not providing "FindArrow.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "Arrow", but
  CMake did not find one.


  Could not find a package configuration file provided by "Arrow" with any of
  the following names:


    ArrowConfig.cmake
    arrow-config.cmake


  Add the installation prefix of "Arrow" to CMAKE_PREFIX_PATH or set
  "Arrow_DIR" to a directory containing one of the above files.  If "Arrow"
  provides a separate development package or SDK, be sure it has been
  installed.
Call Stack (most recent call first):
  build/dist/lib/cmake/ArrowPython/ArrowPythonConfig.cmake:54 (find_dependency)
  CMakeLists.txt:240 (find_package)

Connected issues:

• [C++][Python] CMake cannot find Arrow/Arrow Python when building PyArrow #32826
• [C++][Docs] Update build document to follow new CMake package #32824

Reporter: Alenka Frim / @AlenkaF
Assignee: Anja Boskovic / @anjakefala

Related issues:

• [C++][Python] CMake cannot find Arrow/Arrow Python when building PyArrow (is related to)
• [C++][Docs] Update build document to follow new CMake package (is related to)

PRs and other links:

• GitHub Pull Request #14260

_{Note: This issue was originally created as ARROW-17753. Please see the migration documentation for further details.}

[asfimport]

Alenka Frim / @AlenkaF:
The fix for this issues is:

• Delete stale Arrow C++ and PyArrow build artifacts before rebuilding: navigate to the arrow folder and run git clean -Xfd ..
• Delete lib,  include and share directories if the Arrow libraries are located on a custom path and not in the arrow  folder.
• If Arrow libraries are installed on a custom path instead of a common system one you will have to add the path to CMAKE_PREFIX_PATH. See: https://arrow.apache.org/docs/dev/cpp/build_system.html

[asfimport]

Anja Boskovic / @anjakefala:
How do we feel about adding these notes to the Python developer documentation: https://arrow.apache.org/docs/developers/python.html

? That is where people will look if they run into build problems.

[asfimport]

Alenka Frim / @AlenkaF:
I am not sure. There is already info about:

• running rm -rf arrow/cpp/build in case of errors in Arrow C++ build process

• running git clean -Xfd . to delete stale PyArrow build artifacts

  all towards the end of this section: https://arrow.apache.org/docs/developers/python.html#build-and-test.

  And after https://issues.apache.org/jira/browse/ARROW-17575 the documentation about CMAKE_PREFIX_PATH is quite clear.

  I do agree that the page for Python dev https://arrow.apache.org/docs/developers/python.html needs to be reorganised and that is something I would like to do when I get a chance. It is tracked here: https://issues.apache.org/jira/browse/ARROW-15751.

  This is just my view of the topic and more ideas would be very welcome.
  I know that more I use the docs, less things I feel needs change =)

  So any PR to make the docs better is always welcome!

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
For people using conda (something we do advertise in our python development docs), those cleaning steps are unfortunately not sufficient with the recent changes. You also need to potentially remove $ARROW_HOME/lib/cmake/Arrow* (with ARROW_HOME set to CONDA_PREFIX) or some of the actual libraries or include files, before rebuilding Arrow C++.

I think it might be useful to mention something about it, but I am not sure how to do this robustly and succinctly for all platforms ..

[asfimport]

Alenka Frim / @AlenkaF:
That is true, yes!

Also the info I listed connected to the cleaning of old libs is mentioned only under the Building on Linux and MacOS section of the documentation. There should maybe be a separate item for these frequent tasks that is connected to a specific OS.

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
We could maybe have some checklist of things to try or ensure when your build is failing.

[asfimport]

Alenka Frim / @AlenkaF:
+1

[asfimport]

Anja Boskovic / @anjakefala:
Here are the stepsr that resolved the build issues for pyarrow on a conda env:

Delete stale Arrow C++ build artifacts

rm -rf arrow/cpp/build

Delete stale PyArrow build artifacts

cd python
python3 setup.py clean

Clean up misc build artifacts.

git clean -Xfd .

Remove, clean, and recreate the conda environment.
(Or create a fresh conda environment under a different name, if you wish to avoid removing the previously created one.)

conda deactivate
conda remove -n pyarrow-dev
# On Linux and macOS
conda create -y -n pyarrow-dev -c conda-forge \
       --file arrow/ci/conda_env_unix.txt \
       --file arrow/ci/conda_env_cpp.txt \
       --file arrow/ci/conda_env_python.txt \
       --file arrow/ci/conda_env_gandiva.txt \
       compilers \
       python=3.9 \
       pandas

Activate the environment

conda activate pyarrow-dev
export ARROW_HOME=$CONDA_PREFIX

Adding to environment variables:

export CMAKE_PREFIX_PATH=${ARROW_HOME}

[asfimport]

Anja Boskovic / @anjakefala:
Oh apologies! The code styling did not transfer well at all, and I cannot edit the comment. In particular, the conda environment (which was the most amalgamated) recreation step was pulled from:

https://arrow.apache.org/docs/developers/python.html#using-conda

[asfimport]

Anja Boskovic / @anjakefala:
If I was pointed to a good place to put this, I would happily add this checklist to the documentation!

[asfimport]

Alenka Frim / @AlenkaF:
The Python dev page needs improvement in general, but for now, I would say the best place would be before the "Installing Nightly Packages" subsection on this page:

https://arrow.apache.org/docs/dev/developers/python.html

as a separate subsection by itself.

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
(re-opened the issue so comments can be edited ... :))

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
and repurposed the issue to better document this

[asfimport]

Anja Boskovic / @anjakefala:
Omg, bless you Joris.

Thank you! =) I will open a PR.

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
Issue resolved by pull request 14260
#14260