[CI][Python] wheel-manylinux2014-* sometimes crashed on pytest exit

[kou]

Describe the bug, including details regarding any error messages, version, and platform.

wheel-manylinux2014-cp38-amd64 2022-12-20 nighty:

https://github.com/ursacomputing/crossbow/actions/runs/3738813751/jobs/6345300403#step:9:775

= 4294 passed, 382 skipped, 18 xfailed, 2 xpassed, 4 warnings in 522.05s (0:08:42) =
/arrow/ci/scripts/python_wheel_unix_test.sh: line 95:    79 Segmentation fault      (core dumped) python -m pytest -r s --pyargs pyarrow

wheel-manylinux2014-cp39-arm64 2022-12-20 nighty:

https://app.travis-ci.com/github/ursacomputing/crossbow/builds/259026563#L5708

= 4324 passed, 352 skipped, 18 xfailed, 2 xpassed, 1 warning in 141.20s (0:02:21) =
/arrow/ci/scripts/python_wheel_unix_test.sh: line 95:    15 Segmentation fault      (core dumped) python -m pytest -r s --pyargs pyarrow

Component(s)

Continuous Integration, Packaging, Python

[raulcd]

This is still happening, core dump on those wheel jobs yesterday:
wheel-manylinux2014-cp310-arm64
wheel-manylinux2014-cp311-amd64
wheel-manylinux2014-cp37-amd64
wheel-manylinux2014-cp39-amd64
wheel-manylinux2014-cp39-arm64
I am marking this as a release blocker.

[lidavidm]

I can sort of reproduce; you have to go through the release verification process. Still trying to get GDB inside this process so I can try to backtrace (the test that SIGINTs itself is giving me a bit of trouble). I can't yet reproduce with a 22.04 container directly; trying to make sure the setup is exactly the same.

....and it doesn't crash under gdb, hmm.

[lidavidm]

And now I can no longer get the crash to reproduce either, so I'm not sure I can make much headway on this.

[assignUser]

This is now the last non-archery blocker, with the difficulty in reproducing this should it still stay a blocker?

[raulcd]

It does seem to have happened on the latest test-fedora-35-python-3 on Azure too: https://dev.azure.com/ursacomputing/crossbow/_build/results?buildId=42753&view=logs&j=0da5d1d9-276d-5173-c4c4-9d4d4ed14fdb&t=d9b15392-e4ce-5e4c-0c8c-b69645229181&l=5623

===== 6762 passed, 356 skipped, 18 xfailed, 2 xpassed in 295.11s (0:04:55) =====
/arrow/ci/scripts/python_test.sh: line 60: 22828 Segmentation fault      (core dumped) pytest -r s ${PYTEST_ARGS} --pyargs pyarrow
139

I am not entirely sure this is a blocker for the release at this point. It's quite difficult to reproduce and flaky (not always happens) and it only seems to happen on test exit (as all tests are successful). I would probably remove it from being a blocker on the release and keep investigating around it. @kou @lidavidm what are your thoughts about moving it out of the release?

[lidavidm]

I am OK with this, though we will have to note it at verification time. (And actually, I think it will make binary verification quite painful...)

Can we get some more systematic approaches to solving this? Would it be possible, for instance, to backtrace failing tests automatically, or upload a core dump? In #33720 I'm going to try enabling both PYTHONFAULTHANDLER and catchsegv to see if we can get any more info on what's happening.

[lidavidm]

I think it's an AWS SDK issue again!

I enabled catchsegv here: https://github.com/ursacomputing/crossbow/actions/runs/3939847223/jobs/6740188523

The crashing instruction is at 0x7fed49842bb7:

Backtrace:
/usr/local/lib/python3.9/site-packages/pyarrow/libarrow.so.1100(+0x17b5bb7)[0x7fed49842bb7]

libarrow.so is mapped to 0x7fed4808d000:

7fed4808d000-7fed484d7000 r--p 00000000 00:2c 2138445 /usr/local/lib/python3.9/site-packages/pyarrow/libarrow.so.1100

Subtracting the two, we're looking for 0x17b5bb7 in libarrow.so:

$ objdump --demangle -d -T libarrow.so.1100
...
00000000017b5b90 <Aws::Http::CurlHandleContainer::~CurlHandleContainer()>:
 17b5b90:	41 57                	push   %r15
 17b5b92:	49 89 ff             	mov    %rdi,%r15
 17b5b95:	41 56                	push   %r14
 17b5b97:	41 55                	push   %r13
 17b5b99:	41 54                	push   %r12
 17b5b9b:	55                   	push   %rbp
 17b5b9c:	53                   	push   %rbx
 17b5b9d:	48 81 ec 08 02 00 00 	sub    $0x208,%rsp
 17b5ba4:	e8 37 b3 f9 ff       	callq  1750ee0 <Aws::Utils::Logging::GetLogSystem()>
 17b5ba9:	48 85 c0             	test   %rax,%rax
 17b5bac:	74 15                	je     17b5bc3 <Aws::Http::CurlHandleContainer::~CurlHandleContainer()+0x33>
 17b5bae:	48 89 c5             	mov    %rax,%rbp
 17b5bb1:	48 8b 00             	mov    (%rax),%rax
 17b5bb4:	48 89 ef             	mov    %rbp,%rdi
 17b5bb7:	ff 50 10             	callq  *0x10(%rax)
...

[lidavidm]

Reconstructed backtrace:

Backtrace:
libarrow.so.1100(+0x17b5bb7)[0x7fed49842bb7] Aws::Http::CurlHandleContainer::~CurlHandleContainer()
libarrow.so.1100(+0x17a1d83)[0x7fed4982ed83] Aws::Http::CurlHttpClient::~CurlHttpClient()
libarrow.so.1100(+0x591c6a)[0x7fed4861ec6a]  std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()
libarrow.so.1100(+0x16d882f)[0x7fed4976582f] Aws::S3::S3Client::~S3Client()
libarrow.so.1100(+0x591c6a)[0x7fed4861ec6a]  std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()
libarrow.so.1100(+0x12d5a94)[0x7fed49362a94] std::_Sp_counted_ptr<arrow::fs::(anonymous namespace)::RegionResolver*, (__gnu_cxx::_Lock_policy)2>::_M_dispose()
libarrow.so.1100(+0x591c6a)[0x7fed4861ec6a]  std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()

…maybe we could just leak the S3 client as a workaround?