Flaky conflict installing `rust-src` on GitHub actions runners

[davidhewitt]

On the PyO3 CI we're hitting a flaky issue installing rust-src as part of setup steps on a GitHub actions runner. (Via dtolnay/rust-toolchain action). E.g. https://github.com/PyO3/pyo3/actions/runs/6829368772/job/18575387359#step:5:111

error: failed to install component: 'rust-src', detected conflict: 'lib/rustlib/src/rust/Cargo.lock'

I think we've been encountering this for a little while at a very low probability, but since yesterday this was failing at a probably of maybe 2-3%. Still low, but because we want the whole build matrix to succeed, just one job failing will fail the merge. Restarting the CI doesn't help us much, because we get a different job failing with the same error.

See PyO3/pyo3#3570 for a repeated chain of failed merges hitting this.

Any insight you can offer to help resolve this would be greatly appreciated. Given the flakiness, it feels like a cache issue, but at the point of failing install I don't think we've restored anything from cache.

[rami3l]

@davidhewitt Thanks for filing this issue!

The current workaround seems to be uninstalling the toolchain and installing it again: rust-lang/rls#1587. I'm still not sure what caused it though.
(Maybe removing the minimal profile will also work since you're on the official ubuntu image with rust already installed and this workflow will do an upgrade-downgrade to a newer version with a smaller profile, not quite sure about this...)

@rbtcollins do you happen to have any more context on this one?

[davidhewitt]

(Maybe removing the minimal profile will also work since you're on the official ubuntu image with rust already installed and this workflow will do an upgrade-downgrade to a newer version with a smaller profile, not quite sure about this...)

Interesting, should this upgrade / downgrade be visible in the logs? I don't see any mention of "downgrade" in the logs from the failing CI job linked in the OP, for example. (At the time of writing this the ubuntu-latest image is up to date on Rust 1.73, so the toolchain version isn't changing.)

It looks like the GitHub Actions ubuntu-latest image is the minimal profile plus rustfmt and clippy components installed by rustup component add rustfmt clippy. Does that mean that if I were to add rustfmt and clippy components I would be able avoid the "downgrade"?

https://github.com/actions/runner-images/blob/e5b8919eebf4da1abfd013d080a4d75e6db21e34/images/linux/scripts/installers/rust.sh#L14

Thanks for the help 🙏

[rami3l]

@davidhewitt Thanks for following up! I was probably wrong with the downgrade thing ¹. It's just that I don't see why Cargo.toml is there already (I believe 'lib/rustlib/src/rust/Cargo.lock' is part of rust-src as the message suggests, which means we already have rust-src on that runner)... Normally this is what that conflict error message is for 🤔

Probably the status of the rust toolchains on the runner is not what it seems to be.

But anyway as a temporary workaround, you can uninstall stable before running dtolnay/rust-toolchain.

---

Update: Oops, it's a known issue! Looks like #2601 is still not completely fixed. Claim.

Footnotes

1. I'm quite new here so I'm also learning the different parts of the project on the fly, sorry about that 🙇 ↩

[rami3l]

I can reproduce the same error message on my machine by:

1. rustup toolchain install stable --component=rust-src --profile=minimal;
2. Back up lib/rustlib/src to somewhere else (!!);
3. rustup component remove rust-src;
4. Restore the backup.

Now we have:

> rustup toolchain install stable --component=rust-src --profile=minimal
info: syncing channel updates for 'stable-aarch64-apple-darwin'
info: latest update on 2023-10-05, rust version 1.73.0 (cc66ad468 2023-10-03)
info: downloading component 'rust-src'
info: installing component 'rust-src'
info: rolling back changes
error: failed to install component: 'rust-src', detected conflict: 'lib/rustlib/src/rust/Cargo.lock'

But I'm sure if this is the same scenario. We can avoid this check, or possibly add something like --force, like what brew install does when it has detected an existing file 🤔

[rami3l]

@davidhewitt Would you mind helping me record the output of rustup component list --installed and the status of lib/rustlib/src before setting up Rust in your failed CI? I suspect there is an incoherence problem. Thanks!

[davidhewitt]

It looks like lib/rustlib/src is indeed populated when the job fails, but not always, which is why we see the flaky behaviour.

https://github.com/PyO3/pyo3/actions/runs/6861561741/job/18676110914?pr=3571

[davidhewitt]

It also seems that our rust-toolchain.toml was part of the problem. It seemed to be triggering rustup to check if rust-src was installed multiple times over.

Removing it helped us get to a green CI run again: PyO3/pyo3#3575

[rami3l]

@davidhewitt Thanks! I'll try to reproduce this issue again with this new info when I have time :)

[messense]

Looks like it's not just rust-src, other components can also fail with the same conflict error, see PyO3/maturin#1856

error: failed to install component: 'rustc-x86_64-unknown-linux-gnu', detected conflict: 'bin/rust-gdb'

and

error: failed to install component: 'clippy-preview-x86_64-unknown-linux-gnu', detected conflict: 'bin/cargo-clippy'

[rami3l]

Looks like it's not just rust-src, other components can also fail with the same conflict error, see PyO3/maturin#1856

error: failed to install component: 'rustc-x86_64-unknown-linux-gnu', detected conflict: 'bin/rust-gdb'

and

error: failed to install component: 'clippy-preview-x86_64-unknown-linux-gnu', detected conflict: 'bin/cargo-clippy'

@messense Thanks for reporting! In your logs I have spotted something like:

info: removing previous version of component 'clippy'
warning: during uninstall component clippy was not found

... so I'm even more convinced that this is an inconsistency problem. Could it be that somehow certain (but not all) GHA runners pretend to have clippy or rust-src or whatever installed?

I'm still not sure if I have correctly reproduced this error message locally... So far it hasn't been possible to generate that error message on my machine without manually manipulating what's in the folder of installation.

But there is indeed a point of improvement anyway on our side:

Maybe rustup could have a toolchain verify subcommand that checks the health of and fixes corruption in the toolchain and components? Of course it'd be better if the corruption could be avoided in the first place but it might be good to have some obvious method of recovery for when the worst does happen.

Bonus points if it could diagnose the problem and ask the user to send a bug report with that information, but that's maybe a bit extra 🙂.

Originally posted by @ChrisDenton in #2704 (comment)

[Xuanwo]

Could it be that somehow certain (but not all) GHA runners pretend to have clippy or rust-src or whatever installed?

Yes.

Github Runner has it's own rust setup by default: https://github.com/actions/runner-images/blob/cd2cabc7ab676a9c4603ffd680963ddfc5220270/images/ubuntu/Ubuntu2204-Readme.md?plain=1#L142-L154