Skip to content

Comments

Use the newer recommended S3 ListObjectsV2 API#1048

Closed
giuli007 wants to merge 1 commit intoaptly-dev:masterfrom
Yelp:faster_s3_listbojectsv2
Closed

Use the newer recommended S3 ListObjectsV2 API#1048
giuli007 wants to merge 1 commit intoaptly-dev:masterfrom
Yelp:faster_s3_listbojectsv2

Conversation

@giuli007
Copy link

@giuli007 giuli007 commented Mar 22, 2022

This change is to address a situation we have encountered running in AWS EKS where each single (1000-paginated) request using the older ListObjects API via s3.ListObjectsPages takes seconds to receive a response, resulting in tens of minutes long publishing times when the bucket contains >100k packages (which is the cases for repos mirrored from ubuntu repos and republished in-house).
This doesn't look like normal throttling as no explicit throttled responses are seen when enabling debug in the https://www.aptly.info/doc/configuration/. The reasons why the old API is so slow only when running in EKS are still unknown.

Description of the Change

Uses the more modern recommended ListObjectsV2 to list objects on the S3 bucket when publishing to it.
https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html

This is faster, especially when running in EKS.

Checklist

  • unit-test added (if change is algorithm)
  • functional test added/updated (if change is functional)
  • man page updated (if applicable)
  • bash completion updated (if applicable)
  • documentation updated
  • author name in AUTHORS

@giuli007 giuli007 force-pushed the faster_s3_listbojectsv2 branch from f3590eb to 4142c9b Compare March 23, 2022 12:12
@giuli007 giuli007 force-pushed the faster_s3_listbojectsv2 branch from 4142c9b to 4b4408c Compare March 23, 2022 12:41
@neolynx
Copy link
Member

neolynx commented Jan 14, 2024

this made it into master already..

thanks !

@neolynx neolynx closed this Jan 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants