Return paths in PackageSourceMap deterministically #2357
Merged
+1
−1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
in v1.220.0, specifically 28ea1a0, package source pathing was switched over to a helper utility. the `PackageSourceMap::paths` method looks very similar to the logic before, but has a very subtle difference: in order to avoid duplicate paths, it collects paths from each source map's source files into a `HashSet` and then returns an iterator over than (in contrast to before, which just presented a flat-mapped iterator over all source files directly). unfortunately, this means that iteration order is now random, which can produce non-deterministic compilation when wit-parser is used as part of the compilation macro, say for proc-macros like wasmtime-component macro. it's... difficult to notice this in ordinary cargo compilation, but it causes _extremely_ strange behavior in deterministic/caching build tooling like buck2: given a series of deps `outer -> inner -> [something that uses wasmtime-component-macro, like wasmtime-wasi]`, compiling `outer` will cause rustc to reject `wasmtime-wasi` on a hash mismatch, which will then surface as "unable to find crate "inner"` [^1] fixing this is relatively easy: since our input iteration order is deterministic, simply collecting into an `IndexMap` (already used elsewhere in the file for this purpose) instead of a `HashMap` gives us deterministic iteration order, and thus deterministic compilation. [^1]: i know this happens when compiling `outer` as a static-pic for library; i have not yet tested if it occurs in other compilation modes like binaries or shared libs.
Contributor
Author
|
this maybe also merits a rustc error fix as well -- needa check if the error is better on nightly, but it's extremely unclear on a number of stable rusts |
alexcrichton
approved these changes
Oct 21, 2025
Member
alexcrichton
left a comment
alexcrichton
left a comment
There was a problem hiding this comment.
Oh definitely a mistake, thanks for the PR!
Merged
via the queue into
bytecodealliance:main
with commit Oct 21, 2025
6f96797
64 of 66 checks passed
This was referenced Nov 14, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
in v1.220.0, specifically 28ea1a0, package source pathing was switched over to a helper utility. the
PackageSourceMap::pathsmethod looks very similar to the logic before, but has a very subtle difference:in order to avoid duplicate paths, it collects paths from each source map's source files into a
HashSetand then returns an iterator over that (in contrast to before, which just presented a flat-mapped iterator over all source files directly).unfortunately, this means that iteration order is now random, which can produce non-deterministic compilation when wit-parser is used as part of the compilation macro, say for proc-macros like wasmtime-component macro.
it's... difficult to notice this in ordinary cargo compilation, but it causes extremely strange behavior in deterministic/caching build tooling like buck2: given a series of deps
outer -> inner -> [something that uses wasmtime-component-macro, like wasmtime-wasi], compilingouterwill cause rustc to rejectwasmtime-wasion a hash mismatch, which will then surface as "unable to find crateinner" in yourrustcoutput 1fixing this is relatively easy: since our input iteration order is deterministic, simply collecting into an
IndexSet(already used elsewhere in the file for this purpose) instead of aHashSetgives us deterministic iteration order, and thus deterministic compilation.Footnotes
i know this happens when compiling
outeras a static-pic for library; i have not yet tested if it occurs in other compilation modes like binaries or shared libs. ↩