Place all versions of a single package into the same shard #1164
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
andrewbranch
approved these changes
Apr 2, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This has bugged me for a while; when a package breaks (e.g. react right now), we're effectively guaranteed to place all of that package's versions into different shards, so CI will have failures spread across multiple logs. That's pretty annoying to gauge how broken something is.
This PR modifies our shading algorithm to group packages and then shard on those groups, placing
react/react/v18, ornode/node/v20/node/v18, etc, all in the same shard to fail together.Probably it'd also be worth it to place something like
reactwithreact-dom, since those are also pretty much guaranteed to be in separate CI jobs.Another fix would be to not round robin shard the packages, instead group them directly alphabetically, however I think that might have the negative effect of making CI run longer as we'd end up in a situation like "all react packages are located on the same few shards" and take more time to run than other jobs.
This grouping thing I think is the "least bad" to at least get the same package into the same log.