BUG: test_get_include failing in conda builds on unix

[h-vetinari]

Describe the bug, including details regarding any error messages, version, and platform.

#14832 switched on the test suite within the conda-builds, which itself synced with conda-forge/arrow-cpp-feedstock#875

In the process of doing so test_get_include was skipped,

arrow/dev/tasks/conda-recipes/arrow-cpp/meta.yaml

Lines 382 to 383 in 2c768a1

	# test is broken; header is in $PREFIX, not $SP_DIR
	{% set tests_to_skip = tests_to_skip + " or (test_misc and test_get_include)" %} # [unix]

I believe that is broken now that the C/C++ lib is separate. In particular, the test is looking for include/arrow/api.h (rather than, say, include/arrow/pyarrow/api.h) in the (python-specific) site-packages directory, but the C/C++ headers are now in $PREFIX/include/arrow/.... Interestingly, this passes on windows, because the windows build just seems to copy the C/C++ headers to the %SP_DIR% as well (in addition to the ones correctly put into %LIBRARY_INC%). My outsider's impression is that the test should be adapted (and that windows shouldn't copy the non-python headers to %SP_DIR%).

To fix this, the line quoted above should be removed, and the conda tests still pass.

Component(s)

Continuous Integration, Python

[kou]

test_get_include() is failed because pyarrow package doesn't bundle Apache Arrow C++ headers (

arrow/dev/tasks/conda-recipes/arrow-cpp/build-pyarrow.sh

Line 11 in 2c768a1

export PYARROW_BUNDLE_ARROW_CPP_HEADERS=0

) and get_include() only returns pyarrow's include directory.

I think that test_get_include() should check arrow/python/api.h not arrow/api.h.

I also think that we need one more related API: pyarrow.get_include_dirs()
It returns all include directories that pyarrow C++ API users need. It means that pyarrow.get_include_dirs() may return multiple paths that are for Apache Arrow C++ and pyarrow:

diff --git a/python/pyarrow/tests/test_misc.py b/python/pyarrow/tests/test_misc.py
index 457d7d396a..3725ddf4ac 100644
--- a/python/pyarrow/tests/test_misc.py
+++ b/python/pyarrow/tests/test_misc.py
@@ -29,6 +29,16 @@ def test_get_include():
     assert os.path.exists(os.path.join(include_dir, 'arrow', 'api.h'))
 
 
+def test_get_include_dirs():
+    include_dirs = pa.get_include_dirs()
+    assert any([os.path.exists(os.path.join(include_dir, 'arrow', 'api.h'))
+                for include_dir
+                in include_dirs])
+    assert any([os.path.exists(os.path.join(include_dir, 'arrow', 'python', 'api.h'))
+                for include_dir
+                in include_dirs])
+
+
 @pytest.mark.skipif('sys.platform != "win32"')
 def test_get_library_dirs_win32():
     assert any(os.path.exists(os.path.join(directory, 'arrow.lib'))