Skip to content

feat(oio): Add MultipartUploadWrite to easier the work for Writer#2699

Merged
Xuanwo merged 3 commits intomainfrom
multipart-writer
Jul 24, 2023
Merged

feat(oio): Add MultipartUploadWrite to easier the work for Writer#2699
Xuanwo merged 3 commits intomainfrom
multipart-writer

Conversation

@Xuanwo
Copy link
Copy Markdown
Member

@Xuanwo Xuanwo commented Jul 24, 2023

This PR will add MultipartUploadWrite to easier the work of implementing Writer.

It's an internal API that users can't use directly. We add this to simplify the work to implement Writer for services like s3, cos and oss. And align the behavior of buffering and available options.

This PR just migrated the implement of s3 and we will start a migrate issue for other services.

@Xuanwo Xuanwo requested review from PsiACE and suyanhanx July 24, 2023 07:51
@github-actions github-actions Bot added the releases-note/feat The PR implements a new feature or has a title that begins with "feat" label Jul 24, 2023
Signed-off-by: Xuanwo <github@xuanwo.io>
Copy link
Copy Markdown
Member

@suyanhanx suyanhanx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly LGTM.

But for some services like Google Drive, they need position to indicate where to continue to write. We could count and record the sizes we uploaded, then we can also get the position.

Comment thread core/src/raw/oio/write/multipart_upload_write.rs Outdated
@Xuanwo
Copy link
Copy Markdown
Member Author

Xuanwo commented Jul 24, 2023

But for some services like Google Drive, they need position to indicate where to continue to write. We could count and record the sizes we uploaded, then we can also get the position.

This concern is invalid because only services such as S3 and OSS, which have a multipart upload API, can implement this feature. Other services like fs, google drive are not covered (maybe will have their own abstraction).

Co-authored-by: Suyan <suyanhanx@gmail.com>
@suyanhanx
Copy link
Copy Markdown
Member

But for some services like Google Drive, they need position to indicate where to continue to write. We could count and record the sizes we uploaded, then we can also get the position.

This concern is invalid because only services such as S3 and OSS, which have a multipart upload API, can implement this feature. Other services like fs, google drive are not covered (maybe will have their own abstraction).

😯 Ok.

Copy link
Copy Markdown
Member

@suyanhanx suyanhanx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@Xuanwo Xuanwo merged commit a16dbec into main Jul 24, 2023
@Xuanwo Xuanwo deleted the multipart-writer branch July 24, 2023 08:21
@oowl oowl mentioned this pull request Jul 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

releases-note/feat The PR implements a new feature or has a title that begins with "feat"

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants