Open
Conversation
The default --partition M/N divides invocations into contiguous chunks in workspace-member order, which leaves heavy members biased toward the same partition across CI runs. --partition-seed <SEED> hashes the seed string with FNV-1a and uses it to seed a SplitMix64-driven Fisher-Yates shuffle of the assignment, balancing load across partitions while keeping each partition exactly the same size as the unseeded version.
partition_seeded verifies that the same seed produces identical partition decisions across runs and that the union of "running" commands across all M/N partitions covers every invocation exactly once with the unseeded chunk sizes (6, 6, 5). partition_bad gains a case for using --partition-seed without --partition.
- Reformat with nightly rustfmt: uppercase hex literals in fnv1a_64 (hex_literal_case = "Upper") and collapse a few multi-line blocks (use_small_heuristics = "Max"). - Add PRNG to project-dictionary.txt; it appears in a SplitMix64 doc comment.
Author
|
I did some proper measurements of the last 41 cargo-hack runs with our 155 feature-powerset targets:
I've done 3 additional test runs using this --partition-seed change:
It's clear the load balancing is still not perfectly uniform, but I'm pretty confident that it has improved the status quo. |
Author
|
One thing I have noticed: this implementation still runs through all the tests in order, so this does not solve the issue for #265. I'm happy to adjust the code to account for that if desired (and to make this feature work without needing |
Author
|
Ok. I have a PoC locally for |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An unsolicited idea I had, turns out to be an alternative to #266 and a potential solution for #265 and #255 when given a random seed.
I noticed that in our pretty heavy workspace, of our 4 partitions, 2 of them would finish within 5 minutes while the other 2 would take 20 minutes to complete.
This adds a
--partition-seed <random string>flag that shuffles the partition assignment in an attempt at getting an even amount of load across partitions. The seed (rather than pure random) is for consistency within re-runs and across partitions. When the flag is not provided, the partitions are left unshuffled, which preserved the existing behaviour.Current notes:
--partitionand will error without (style choice, can change this)The code was written with the assistance of Claude Opus 4.7 Extra High Effort. Idea and design is entirely my own.