[Python] AWS SDK 1.11 support in pyarrow wheels?

[johnkerl]

Describe the bug, including details regarding any error messages, version, and platform.

With regard to #40262, is there a plan to update pyarrow's AWS SDK dependency from 1.10 to 1.11? We believe from

arrow/cpp/thirdparty/versions.txt

Line 54 in fe4d04f

ARROW_AWSSDK_BUILD_VERSION=1.10.55

that pyarrow is currently using 1.10:

ARROW_AWSSDK_BUILD_VERSION=1.10.55

It appears that a mitigation for #40262 is in AWS SDK 1.11:
aws/aws-sdk-cpp#2710

(There's significant backstory on single-cell-data/TileDB-SOMA#2692 and on TileDB-Inc/tiledbsoma-feedstock#171, if backstory is desired. A repro is here: single-cell-data/TileDB-SOMA#2692 (comment).)

cc @pitrou

Component(s)

Python

[johnkerl]

Also cc @ihnorton @ivirshup @h-vetinari

[h-vetinari]

Conda-forge has been building against aws 1.11 for a long time already, and this also got synched back to the conda tests in arrow itself (which have bitrotted in the meantime, but there are efforts to revive them):

arrow/dev/tasks/conda-recipes/.ci_support/linux_64_cuda_compiler_version11.2.yaml

Lines 3 to 4 in 801de2f

	aws_sdk_cpp:
	- 1.11.156

In any case, we run the full test suite on the python side (not the C++ side yet, c.f. #35587), in every feedstock build, and it passes on osx. So I don't see the immediate incompatibility, which I assume is restricted to some corner cases.

You should provide a stacktrace (or ideally, a reproducer) of what fails.

PS. In the past there was once something that kept arrow stuck on aws 1.8 for a long time (which might help for context): aws/aws-sdk-cpp#1809

[h-vetinari]

Looks like TileDB-Inc/TileDB-Py#1990 is relevant, but again, you should really provide an example where arrow crashes or does something wrong, not another downstream project. The fact that the import order seems to matter is already ground for suspecting that there's something else going on here.

[ihnorton]

The specific question here is if/when will arrow wheels update to AWS SDK 1.11? The reason for the question is to understand whether the mitigation for the issue described below will be available "soon", or we need to work around it (rename symbols, further patch the AWS SDK, etc?).

For more background on the issue:

• When pyarrow and tiledb-py are installed from PyPI, and imported in the same process, making S3 requests (which happens via the AWS SDK) causes an abort on some platforms.
• Standalone repro: https://gist.github.com/ihnorton/790b575944a5d09674e86a10700f1dab
  • this is a macOS-arm64 system
  • note that conda is used only for python interpreter installation
  • the repro works in both directions: import tiledb first -> pyarrow s3 request crashes --- or --- import pyarrow first -> tiledb s3 request crashes
• The top of the stack trace is:

1   libarrow.1601.dylib                 0x000000010e08b8d0 aws_fatal_assert + 80
2   libarrow.1601.dylib                 0x000000010e08ab98 aws_mem_acquire + 64
3   libarrow.1601.dylib                 0x000000010e09dd68 aws_string_new_from_cursor + 76
4   libarrow.1601.dylib                 0x000000010e0975e4 aws_json_value_get_from_object + 44
5   libarrow.1601.dylib                 0x000000010e083970 aws_endpoints_ruleset_new_from_string + 120
6   libarrow.1601.dylib                 0x000000010e01d5a4 _ZN3Aws3Crt9Endpoints10RuleEngineC2ERK15aws_byte_cursorS5_P13aws_allocator + 48
7   libarrow.1601.dylib                 0x000000010ddf7180 _ZN3Aws8Endpoint23DefaultEndpointProviderINS_2S321S3ClientConfigurationENS2_8Endpoint19S3BuiltInParametersENS4_25S3ClientContextParametersEEC2EPKcm + 116
8   libtiledb.dylib                     0x0000000162bf367c _ZN3Aws2S38S3ClientC2ERKNS_6Client19ClientConfigurationENS2_15AWSAuthV4Signer20PayloadSigningPolicyEbNS0_34US_EAST_1_REGIONAL_ENDPOINT_OPTIONE + 980
9   libtiledb.dylib                     0x000000016241a4bc _ZN6tiledb6common11make_sharedINS_2sm14TileDBS3ClientELi66EJRKNS2_12S3ParametersERN3Aws6Client19ClientConfigurationENS8_15AWSAuthV4Signer20PayloadSigningPolicyERKbEEENSt3__110shared_ptrIT_EERAT0__KcDpOT1_ + 92
10  libtiledb.dylib                     0x0000000162403624 _ZNK6tiledb2sm2S311init_clientEv + 3428

• This pre-existing arrow issue describes the underlying cause of the stack trace above: [C++] Avoid double initialisation / double finalize of Aws SDK #40262
• There is extensive prior discussion elsewhere on the AWS SDK repro, including Aws::InitAPI(aws_options_) in two independent libraries may crash on OSX aws/aws-sdk-cpp#2699
• This is unrelated to conda, where -- as @h-vetinari notes -- the AWS SDK version is up-to-date; additionally, there is only a single copy of the SDK in the process image when both are installed from conda-forge.

Summarizing: AWS has released a mitigation for the abort, implemented here: aws/aws-sdk-cpp#2710. The mitigation is available in AWS SDK 1.11. TileDB wheels have updated to AWS SDK 1.11, but AFAICT all packages need to be updated for the mitigation to work.

This issue will likely impact any other library that bundles the AWS SDK in a wheel and is loaded at the same time as pyarrow.

[kou]

PyArrow wheels don't use bundled AWS SDK for C++. It uses vcpkg's one:

https://github.com/ursacomputing/crossbow/actions/runs/9544500563/job/26303310398#step:7:559

-- Found AWS SDK for C++, Version: 1.11.201, Install Root:/opt/vcpkg/installed/amd64-linux-static-release, Platform Prefix:, Platform Dependent Libraries: pthread;crypto;ssl;z;curl

[jorisvandenbossche]

The VCPKG version is currently pinned at 2023.11.20:

arrow/.env

Line 92 in eec6f17

VCPKG="a42af01b72c28a8e1d7b48107b33e4f286a55ef6" # 2023.11.20 Release

(last updated in #39622)

This could certainly use another update to a more recent vcpkg state (EDIT: this is currently being done in #42171), but so that release (as @kou also showed from the logs) already included AWS SDK 1.11 (https://github.com/microsoft/vcpkg/releases/tag/2023.11.20, it updated it from 1.11.169#2 to 1.11.201)