[Python] Stop initializing s3 upon import

[vyasr]

Describe the enhancement requested

Currently pyarrow initializes the s3 filesystem when pyarrow.fs is imported. This leads to AWS consuming resources on startup that may never be used if the user is not actually taking advantage of that support. Ideally the s3fs would instead be delayed to first use to avoid AWS spinning up unnecessary threads/doing work on pyarrow import.

Making this change would also allow sidestepping a bug present in newer versions of the aws-sdk-cpp that occasionally leads to segfaults simply by using the AWS APIs, at least for the majority of users who are not using the s3fs by default.

Component(s)

C++, Python

[vyasr]

I believe @pentschev has put together a patch demonstrating how this could be done.

[felipecrv]

@westonpace has worked on the problem of S3 initialization in the past and it seems like we can either init eagerly and avoid some issues or init lazily and suffer bugs. Any solution here needs to consider the issues that lead to the eager init solution.

[felipecrv]

Reference #33858

[jakirkham]

If Weston is able to review PR ( #38375 ), we would appreciate hearing his insights 🙂

[pitrou]

@pentschev Can you post a message on this issue, so that you can assign it to you?

[pentschev]

Sure, please assign the issue to myself, @pitrou .