CopyStream in arrow/filesystem/util_internal.h does this, but we should expose it, multithread it (can read in one thread while the other thread writes), and further see if there are filesystem-specific optimizations (e.g. S3 multipart uploading/downloading). We may also want a version that takes a FileSelector or vector of paths and parallelizes the operations on them.
Reporter: Neal Richardson / @nealrichardson
Assignee: Ben Kietzman / @bkietz
Related issues:
PRs and other links:
Note: This issue was originally created as ARROW-9868. Please see the migration documentation for further details.
CopyStreamin arrow/filesystem/util_internal.h does this, but we should expose it, multithread it (can read in one thread while the other thread writes), and further see if there are filesystem-specific optimizations (e.g. S3 multipart uploading/downloading). We may also want a version that takes a FileSelector or vector of paths and parallelizes the operations on them.Reporter: Neal Richardson / @nealrichardson
Assignee: Ben Kietzman / @bkietz
Related issues:
PRs and other links:
Note: This issue was originally created as ARROW-9868. Please see the migration documentation for further details.