Leverage object store bulk copy to efficiently deep_clone dataset

[majin1102]

Lance format now support deep_clone dataset: #5250

But the underlying is using single file copy interface. As we know all cloud storage have batch copy, we should ideally leverage that, but that requires upstream support.

This issue is to track this progress

[Xuanwo]

How batch copy works? Can you share a link to this API?

[majin1102]

https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-ops-examples-copy.html

I'm afraid we need to provide some kind of job level API if we want to do this.

[Xuanwo]

https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-ops-examples-copy.html

I'm afraid we need to provide some kind of job level API if we want to do this.

I don't think that's a suitable API we can provide. The batch API itself doesn't guarantee that they'll finish in the time we want.

[jackye1995]

The batch API itself doesn't guarantee that they'll finish in the time we want.

I think it has an API to check progress, but I agree it is not super friendly for an IO library to expose.

I mentioned this because I know there are some perf benefits for doing this in Amazon S3, and GCS and Azure also has similar APIs that people can use. But this will require arrow-rs-object-store or opendal to support that first, and then we can use it in lance-io, so it's a bit far away at this point.