[Packaging][Release][Python] Binary verification of wheels on conda is segfaulting on ORC test_example_using_json

[raulcd]

Describe the bug, including details regarding any error messages, version, and platform.

The verify-rc-binaries-wheels-linux-conda-latest-amd64 segfaults on test_example_using_json when checking the file:

 Fatal Python error: Aborted

Current thread 0x00007f48f1c1a740 (most recent call first):
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/pyarrow/orc.py", line 187 in read
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/pyarrow/tests/test_orc.py", line 96 in check_example_file
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/pyarrow/tests/test_orc.py", line 140 in test_example_using_json
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/_pytest/python.py", line 162 in pytest_pyfunc_call
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/_pytest/python.py", line 1632 in runtest
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/_pytest/runner.py", line 173 in pytest_runtest_call
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/_pytest/runner.py", line 241 in <lambda>
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/_pytest/runner.py", line 341 in from_call
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/_pytest/runner.py", line 240 in call_and_report
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/_pytest/runner.py", line 135 in runtestprotocol
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/_pytest/runner.py", line 116 in pytest_runtest_protocol
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/_pytest/main.py", line 364 in pytest_runtestloop
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/_pytest/main.py", line 339 in _main
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/_pytest/main.py", line 285 in wrap_session
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/_pytest/main.py", line 332 in pytest_cmdline_main
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/_pytest/config/__init__.py", line 178 in main
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/_pytest/config/__init__.py", line 206 in console_main
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/pytest/__main__.py", line 7 in <module>
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/runpy.py", line 87 in _run_code
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/runpy.py", line 194 in _run_module_as_main
/arrow/ci/scripts/python_wheel_unix_test.sh: line 97:  3967 Aborted                 (core dumped) python -m pytest -r s --pyargs pyarrow
tests/test_orc.py .Failed to verify release candidate. See /tmp/arrow-17.0.0.cgU0S for details.

This can be reproduced locally with archery docker run -e VERIFY_VERSION="17.0.0" -e VERIFY_RC="2" -e TEST_DEFAULT=0 -e TEST_WHEELS=1 conda-verify-rc. This is the only test failing as I can have a successful verification with the following local change:

diff --git a/ci/scripts/python_wheel_unix_test.sh b/ci/scripts/python_wheel_unix_test.sh
index a25e5c5..8445dce 100755
--- a/ci/scripts/python_wheel_unix_test.sh
+++ b/ci/scripts/python_wheel_unix_test.sh
@@ -93,5 +93,5 @@ if [ "${CHECK_UNITTESTS}" == "ON" ]; then
 
   # Execute unittest, test dependencies must be installed
   python -c 'import pyarrow; pyarrow.create_library_symlinks()'
-  python -m pytest -r s --pyargs pyarrow
+  python -m pytest -r s --pyargs pyarrow -k "not test_example_using_json"
 fi

Component(s)

Packaging, Python, Release

[raulcd]

@kou @pitrou @wgtmac any idea what might cause this? This test is passing on non-conda verification tasks, see the rest of binary verifications here: #43220

[raulcd]

This is a single test on a single job failing for the release. I am also not sure if we want to block the release for this one, @jorisvandenbossche @AlenkaF FYI

[kou]

Can we get a backtrace by gdb?

[wgtmac]

What's the version of ORC? I'm not sure if it is related to apache/orc#1882

@h-vetinari Does it mean we have to install tzdata in the conda env after this change?

[raulcd]

What's the version of ORC? I'm not sure if it is related to https://github.com/apache/orc/pull/1882/files

The wheel was built with:

-- ARROW_ORC_BUILD_VERSION: 2.0.1
-- ARROW_ORC_BUILD_SHA256_CHECKSUM: 1ffac0228aa83f04a1b1cf2788a3af5953e82587ae3a77c41900e99f2557132d

[wgtmac]

Yes, the commit in doubt is included in the 2.0.1 release.

[kou]

Could you try this?

diff --git a/dev/release/verify-release-candidate.sh b/dev/release/verify-release-candidate.sh
index fcaaa423a4..bd136054e4 100755
--- a/dev/release/verify-release-candidate.sh
+++ b/dev/release/verify-release-candidate.sh
@@ -1153,7 +1153,7 @@ test_linux_wheels() {
     local pyver=${python/m}
     for platform in ${platform_tags}; do
       show_header "Testing Python ${pyver} wheel for platform ${platform}"
-      CONDA_ENV=wheel-${pyver}-${platform} PYTHON_VERSION=${pyver} maybe_setup_conda
+      CONDA_ENV=wheel-${pyver}-${platform} PYTHON_VERSION=${pyver} maybe_setup_conda tzdata
       if ! VENV_ENV=wheel-${pyver}-${platform} PYTHON_VERSION=${pyver} maybe_setup_virtualenv; then
         continue
       fi

[raulcd]

Thanks @kou ! That fixes the issue