GH-41493: [C++][S3] Add a new option to check existence before CreateDir

[HaochengLIU]

Rationale for this change

I have a use case that thousands of jobs are writing hive partitioned parquet files daily to the same bucket via S3FS filesystem. The gist here is a lot of keys are being created at the same time hense jobs hits AWS Error SLOW_DOWN. during Put Object operation: The object exceeded the rate limit for object mutation operations(create, update, and delete). Please reduce your rate request error. frequently throughout the day since the code is creating directories pessimistically.

What changes are included in this PR?

Add a new S3Option to check the existence of the directory before creation in CreateDir. It's disabled by default.

When it's enabled, the CreateDir function will check the existence of the directory first before creation. It ensures that the create operation is only acted when necessary. Though there are more I/O calls, but it avoids hitting the cloud vendor put object limit.

Are these changes tested?

Add test cases when the flag is set to true. Right on top of the mind i donno how to ensure it's working in these tests. But in our production environment, we have very similar code and it worked well.

Are there any user-facing changes?

• GitHub Issue: Avoid throttling: S3FileSystem::CreateDir should check the existence first rather than creating the parent path all the time #41493

[github-actions]

⚠️ GitHub issue #41493 has been automatically assigned in GitHub to PR creator.

[HaochengLIU]

@pitrou @westonpace Could you pls review when you have time?

[pitrou]

I'm curious, don't you want to do the checks by walking up instead of walking down?
For example, when CreateDir("a/b/c") is called, it would seem more logical to first check for a/b/c's existence, then a/b if it doesn't exist, etc.

[HaochengLIU]

I just reflect on it and my 2cents is to keep the walking down logic. I'm doing the checking inside the creation step. I can only build a wall from the bottom up otherwise the world is upside down :) So I must check a then a/b and a/b/c.

when CreateDir("a/b/c") is called, a/b/c's existence will be checked immediately in non recursive mode

[pitrou]

What I'm proposing is (example in the case of a/b/c):

1. check which ancestors exist by walking up the directory tree: first a/b/c then a/b... until you find the first existing ancestor
2. create the missing descendents by walking down the directory tree: for example, if you just found that a exists, create a/b then a/b/c

The idea is that, most of the time, almost the entire directory chain will exist, especially if your workload is writing into a deeply partitioned dataset. So doing the directory checks from leaf to root should issue less requests and have less latency.

More formally, let's call n the depth of the path given to CreateDir, and m the number of directories missing along that path.

• with the current approach, we're calling HeadObject O(n) times and PutObject O(m) times;
• with my proposal, we're calling HeadObject O(m) times and PutObject O(m) times.

Assuming that m is on average much smaller than n (m would be 0 or 1 most of the time), the approach I'm proposing should be much more efficient, and it would never be less efficient anyway.

Does that make sense or am I missing something?

[HaochengLIU]

@pitrou Could you please review again?
In this round:

• I added the above logic.
• Ensure my tests are working as I foget to call MakeFileSystem()
• Add missing tests for non recursive CreateDir scenario.

[pitrou]

Thanks for submitting this @HaochengLIU ! Here are some comments.

[pitrou]

Thanks for the update @HaochengLIU . This looks good to me, bar some minor comments.

[pitrou]

@bkietz Will we have to support this new option in S3 URIs as well?

[HaochengLIU]

Whoa this is insane
TIL in github you can modify a code review in place as a reviewer!
Thanks for the reviewing and guidance @pitrou here

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit a44b537.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 3 possible false positives for unstable benchmarks that are known to sometimes produce them.