ARROW-18113: [C++] Use RandomAccessFile::ReadManyAsync for cloud file systems #14747
Conversation
…w::io::CacheOptions to arrow::io::CoalesceOptions
| std::shared_ptr<S3RetryStrategy> retry_strategy; | ||
|
|
||
| /// Coalesce options for requesting multiple reads at once | ||
| io::CoalesceOptions coalesceOptions = io::CoalesceOptions::Defaults(); |
There was a problem hiding this comment.
nit: follow the naming conventions (snake_case)
There was a problem hiding this comment.
thanks, btw I'll use here io::CoalesceOptions::MakeFromNetworkMetrics(5, 500) as default value (i.e. hole_size_limit = 2.5 MiB and range_size_limit = 22.5 MiB)
| int64_t hole_size_limit = 1; | ||
| int64_t range_size_limit = 100; |
There was a problem hiding this comment.
Why do we need these separately?
There was a problem hiding this comment.
typo, I forgot to delete those fields
| ctx, io::internal::CoalesceReadRanges(std::move(ranges), | ||
| coalesceOptions.hole_size_limit, | ||
| coalesceOptions.range_size_limit)); |
There was a problem hiding this comment.
Since coalescing may merge ranges, you'll need to post-process things so that the returned futures map 1:1 with the original ranges. (May need to refactor things so you don't have to do the coalescing step twice?)
There was a problem hiding this comment.
Got it, thanks David, I'm writing a test for verifying these cases.
|
Thank you for your contribution. Unfortunately, this pull request has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this PR will be closed in 14 days. Feel free to re-open this if it has been closed in error. If you do not have repository permissions to reopen the PR, please tag a maintainer. |
github-actions
bot
added
the
Status: stale-warning
2 participants
This PR is a follow up from #14723:
arrow::io::CacheOptionstoarrow::io::CoalesceOptions(deprecateCacheOptions)RandomAccessFile::ReadManyAsyncfor S3 file system.RandomAccessFile::ReadManyAsyncfor Google Cloud Storage file system.Related Jira ticket:
https://issues.apache.org/jira/browse/ARROW-18113