Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Improves the
s3Cleanuptool so that subsequent runs don't re-process objects that were already tagged as orphans, and ensures non-asset prefixes (CloudFront logs, admin manifests, fonts) are never considered orphan candidates.Changes
tools/s3Cleanup.ts
--tagrun, the script uploads a manifest of all successfully-tagged keys tos3://assets.pubpub.org/_orphanAdmin/tagged-<timestamp>.txt. On subsequent runs, these manifests are downloaded and their keys are skipped during the S3 listing phase. This replaces the previous local-file approach and survives redeploys._prefixed folders: Any top-level key starting with_is now ignored (covers_testing/,_cflogs/,_orphanAdmin/,_fonts/, and any future underscore-prefixed paths).newOrphans.txt: Phase 2 now writes only newly-discovered orphans totmp/newOrphans.txt, making it clear these are incremental finds not already tracked.GetObjectCommand,PutObjectCommandadded for manifest download/upload.scripts/upload-fonts-to-s3.sh
_fonts/prefix (already covered by the_skip rule in s3Cleanup).server/Html.tsx & workers/tasks/export/html.tsx
_fonts/<hash>/fonts.csspath.How it works
_orphanAdmin/*.txtmanifests in S3_*andfonts/prefixes, skip previously-tagged keys → write only new orphans totmp/newOrphans.txt--tag): Tag new orphans, then upload a manifest of successfully-tagged keys to_orphanAdmin/Safety
--untag_orphanAdmin/prefix returns no manifests)