Improve S3 upload speeds using aws transfer manager#17674
Improve S3 upload speeds using aws transfer manager#17674gianm merged 5 commits intoapache:masterfrom
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| private MockAmazonS3Client() | ||
| { | ||
| super(new AmazonS3Client(), new NoopServerSideEncryption()); | ||
| super(new AmazonS3Client(), new NoopServerSideEncryption(), new S3TransferConfig()); |
Check notice
Code scanning / CodeQL
Deprecated method or constructor invocation
| // https://github.com/aws/aws-sdk-java/issues/2285 | ||
| return true; | ||
| } else if (e instanceof InterruptedException) { | ||
| Thread.interrupted(); // Clear interrupted state and not retry |
There was a problem hiding this comment.
Why do we need to clear the interrupted flag.
There was a problem hiding this comment.
The intention is to gracefully stop the retry operations.
Although, The flag would already be cleared when the InterruptedException is thrown.
Let me know if your suggestion is to modify it.
gianm
left a comment
gianm
left a comment
There was a problem hiding this comment.
LGTM. Thank you for your contribution!
This change will need a release note, since the usage of the transfer manager by default involves multipart uploads, so people may need to set up a lifecycle rule to clean up unused parts: https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpu-abort-incomplete-mpu-lifecycle-config.html
Follow-up to apache#17674.
|
@suraj-goel Since the feature is enabled by default, could you please update the description ? |
|
@suraj-goel Adjusted the PR description. Please have a look. |
|
@suraj-goel For your benchmarking, I've got some questions if you don't mind -
Troubleshooting some s3 throttling and wondering if the default of 20MB parts for big enough files can flood a cold s3 path too quickly. |
6 participants
Fixes #17669.
Description
Release note
Improved S3 Upload Performance in Druid
common.runtime.propertiesThis PR has: