Skip to content

Conversation

@narpfel
Copy link
Member

@narpfel narpfel commented May 23, 2025

@narpfel narpfel force-pushed the rust-install-miri branch from 1f7def9 to 8e3ee2b Compare May 23, 2025 20:11
@mattgodbolt
Copy link
Member

@mattgodbolt
Copy link
Member

The miri build failed:

2025-05-27 18:19:40,703 lib.installation_context INFO     Symlinking mrustc-master-20250507 to /opt/compiler-explorer/mrustc-master
2025-05-27 18:19:40,707 compilers/rust/newer/nightly/mrustc master WARNING  Not running on admin node - not saving compiler version info to AWS
2025-05-27 18:19:40,708 lib.ce_install  INFO     compilers/rust/newer/nightly/mrustc master installed OK
2025-05-27 18:19:40,940 lib.installation_context INFO     Fetching https://s3.amazonaws.com/compiler-explorer/opt/rustc-cg-gcc-master-20250522.tar.xz (193118464 bytes)
2025-05-27 18:19:46,190 lib.installation_context INFO     73.8% of https://s3.amazonaws.com/compiler-explorer/opt/rustc-cg-gcc-master-20250522.tar.xz...
2025-05-27 18:19:48,523 lib.installation_context INFO     100% of https://s3.amazonaws.com/compiler-explorer/opt/rustc-cg-gcc-master-20250522.tar.xz
2025-05-27 18:19:48,523 lib.installation_context INFO     Piping to tar Jxf -
2025-05-27 18:20:02,867 lib.installation_context INFO     Moving from staging (/opt/compiler-explorer/staging/61a17e30-86d2-4b03-a02d-e629c87af820/rustc-cg-gcc-master-20250522) to final destination (/opt/compiler-explorer/rustc-cg-gcc-master-20250522)
2025-05-27 18:20:02,905 lib.installation_context INFO     Symlinking rustc-cg-gcc-master-20250522 to /opt/compiler-explorer/rustc-cg-gcc-master
2025-05-27 18:20:02,918 compilers/rust/newer/nightly/rustc-cg-gcc master WARNING  Not running on admin node - not saving compiler version info to AWS
2025-05-27 18:20:02,918 lib.ce_install  INFO     compilers/rust/newer/nightly/rustc-cg-gcc master installed OK
Installing compilers/rust/newer/nightly beta
Installing compilers/rust/newer/nightly nightly
Installing compilers/rust/newer/nightly/gcc gccrs-master
Installing compilers/rust/newer/nightly/mrustc master
Installing compilers/rust/newer/nightly/rustc-cg-gcc master
5 packages installed OK, 0 skipped, and 0 failed installation
+ RUST=/opt/compiler-explorer/rust-miri-nightly
+ mv /opt/compiler-explorer/rust-nightly /opt/compiler-explorer/rust-miri-nightly
+ source .cargo/env
++ case ":${PATH}:" in
++ export PATH=/root/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ PATH=/root/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ rustup toolchain link miri /opt/compiler-explorer/rust-miri-nightly
+ rustup default miri
info: default toolchain set to 'miri'
+ export MIRI_SYSROOT=/opt/compiler-explorer/rust-miri-nightly/miri-sysroot
+ MIRI_SYSROOT=/opt/compiler-explorer/rust-miri-nightly/miri-sysroot
+ for manifest_path in ${RUST}/lib/rustlib/manifest-rust-std-*
+ cargo miri setup --target=aarch64-apple-darwin --verbose
Running `"rustup" "component" "add" "rust-src"` to install the `rust-src` component for the selected toolchain.
error: toolchain 'miri' does not support components
fatal error: failed to install the `rust-src` component for the selected toolchain
Error: Process completed with exit code 1.
Run ./post-run.sh "miri" "" "" "1748368570"
+ COMPILER=miri
+ STATUS=

@mattgodbolt
Copy link
Member

@narpfel let me know if I can help with this: I don't really know what's going on there and the error seems to be in rustup

@narpfel
Copy link
Member Author

narpfel commented May 27, 2025

Running `"rustup" "component" "add" "rust-src"` to install the `rust-src` component for the selected toolchain.
error: toolchain 'miri' does not support components
fatal error: failed to install the `rust-src` component for the selected toolchain

Building the Miri sysroot needs the standard library sources, which it tries to install, but this doesn’t work because CE’s rustc is not installed via rustup. I filed #1630 to install rust-src via ./bin/ce_install.

If you’d prefer that rust-src is only installed during build, I can make the necessary changes to https://github.com/compiler-explorer/misc-builder/blob/main/miri/build.sh.

@mattgodbolt
Copy link
Member

This will need to be installed in the docker container. The error is in the docker - the docker image doesn't see anything other than the things we place in it. Maybe I'm misunderstanding this: do you mean for the infra package to install something ecxtra just so it appears in the docker container here?

@narpfel
Copy link
Member Author

narpfel commented May 27, 2025

@mattgodbolt
Copy link
Member

Now I get it: I think we should only install it in the docker container if that's OK: it already takes hours to install rust things on our slow network drive, adding a ton more on the nightly install (unless it's needed at runtime anyway) just to have it available in the docker seems unfortunate.

@narpfel
Copy link
Member Author

narpfel commented May 27, 2025

I think we should only install it in the docker container if that's OK

That’s also okay, as far as I know the sources are only needed during build time.

It’s not a lot of space, just a lot of small files, so I imagine installing them on a network drive would be rather slow.

@mattgodbolt
Copy link
Member

Perfect: thanks @narpfel

@narpfel
Copy link
Member Author

narpfel commented May 27, 2025

PR here: compiler-explorer/misc-builder#113

@mattgodbolt
Copy link
Member

@mattgodbolt
Copy link
Member

Still failing: https://github.com/compiler-explorer/compiler-workflows/actions/runs/15289645148

can you please run and confirm the docker image runs locally if possible? It will save a lot of long round-trips and is partly why we use docker; it should be fairly straightforward (if computiationally expensive I know) to test locally.

@mattgodbolt
Copy link
Member

The log file is large- here's the important part:

2025-05-28T02:24:02.7962729Z     Checking test v0.0.0 (/opt/compiler-explorer/rust-miri-nightly/lib/rustlib/src/rust/library/test)
2025-05-28T02:24:03.1781499Z     Checking sysroot v0.0.0 (/opt/compiler-explorer/rust-miri-nightly/lib/rustlib/src/rust/library/sysroot)
2025-05-28T02:24:03.2067780Z     Checking custom-local-sysroot v0.0.0 (/tmp/.tmppRIxeC)
2025-05-28T02:24:03.2348192Z     Finished `custom_sysroot` profile [optimized] target(s) in 16.00s
2025-05-28T02:24:03.3466196Z A sysroot for Miri is now available in `/opt/compiler-explorer/rust-miri-nightly/miri-sysroot`.
2025-05-28T02:24:03.3470149Z + for manifest_path in ${RUST}/lib/rustlib/manifest-rust-std-*
2025-05-28T02:24:03.3470610Z + cargo miri setup --target=aarch64-apple-ios-sim --verbose
2025-05-28T02:24:03.4521479Z Preparing a sysroot for Miri (target: aarch64-apple-ios-sim)...
2025-05-28T02:24:03.5687515Z    Compiling compiler_builtins v0.1.159
2025-05-28T02:24:03.5688095Z     Checking core v0.0.0 (/opt/compiler-explorer/rust-miri-nightly/lib/rustlib/src/rust/library/core)
2025-05-28T02:24:03.5688795Z    Compiling libc v0.2.172
2025-05-28T02:24:03.5689199Z    Compiling object v0.36.7
2025-05-28T02:24:03.5698825Z    Compiling std v0.0.0 (/opt/compiler-explorer/rust-miri-nightly/lib/rustlib/src/rust/library/std)
2025-05-28T02:24:12.1317582Z     Checking rustc-std-workspace-core v1.99.0 (/opt/compiler-explorer/rust-miri-nightly/lib/rustlib/src/rust/library/rustc-std-workspace-core)
2025-05-28T02:24:12.9727998Z     Checking alloc v0.0.0 (/opt/compiler-explorer/rust-miri-nightly/lib/rustlib/src/rust/library/alloc)
2025-05-28T02:24:12.9728707Z     Checking cfg-if v1.0.0
2025-05-28T02:24:12.9729063Z     Checking memchr v2.7.4
2025-05-28T02:24:12.9729318Z     Checking adler2 v2.0.0
2025-05-28T02:24:12.9729581Z     Checking rustc-demangle v0.1.24
2025-05-28T02:24:12.9755959Z     Checking panic_abort v0.0.0 (/opt/compiler-explorer/rust-miri-nightly/lib/rustlib/src/rust/library/panic_abort)
2025-05-28T02:24:13.0004436Z     Checking unwind v0.0.0 (/opt/compiler-explorer/rust-miri-nightly/lib/rustlib/src/rust/library/unwind)
2025-05-28T02:24:14.4266509Z     Checking rustc-std-workspace-alloc v1.99.0 (/opt/compiler-explorer/rust-miri-nightly/lib/rustlib/src/rust/library/rustc-std-workspace-alloc)
2025-05-28T02:24:14.4267511Z     Checking panic_unwind v0.0.0 (/opt/compiler-explorer/rust-miri-nightly/lib/rustlib/src/rust/library/panic_unwind)
2025-05-28T02:24:14.4530217Z     Checking gimli v0.31.1
2025-05-28T02:24:14.4530514Z     Checking hashbrown v0.15.3
2025-05-28T02:24:14.4531778Z     Checking std_detect v0.1.5 (/opt/compiler-explorer/rust-miri-nightly/lib/rustlib/src/rust/library/stdarch/crates/std_detect)
2025-05-28T02:24:14.4532397Z     Checking miniz_oxide v0.8.8
2025-05-28T02:24:15.7888318Z     Checking addr2line v0.24.2
2025-05-28T02:24:18.7435953Z     Checking rustc-std-workspace-std v1.99.0 (/opt/compiler-explorer/rust-miri-nightly/lib/rustlib/src/rust/library/rustc-std-workspace-std)
2025-05-28T02:24:18.7727598Z     Checking unicode-width v0.1.14
2025-05-28T02:24:18.7727936Z     Checking rustc-literal-escaper v0.0.2
2025-05-28T02:24:18.8332686Z     Checking proc_macro v0.0.0 (/opt/compiler-explorer/rust-miri-nightly/lib/rustlib/src/rust/library/proc_macro)
2025-05-28T02:24:18.9263422Z     Checking getopts v0.2.21
2025-05-28T02:24:19.0275194Z     Checking test v0.0.0 (/opt/compiler-explorer/rust-miri-nightly/lib/rustlib/src/rust/library/test)
2025-05-28T02:24:19.4116776Z     Checking sysroot v0.0.0 (/opt/compiler-explorer/rust-miri-nightly/lib/rustlib/src/rust/library/sysroot)
2025-05-28T02:24:19.4399052Z     Checking custom-local-sysroot v0.0.0 (/tmp/.tmpXvoAuy)
2025-05-28T02:24:19.4681763Z     Finished `custom_sysroot` profile [optimized] target(s) in 16.01s
2025-05-28T02:24:19.5826870Z A sysroot for Miri is now available in `/opt/compiler-explorer/rust-miri-nightly/miri-sysroot`.
2025-05-28T02:24:19.5830075Z + for manifest_path in ${RUST}/lib/rustlib/manifest-rust-std-*
2025-05-28T02:24:19.5830584Z + cargo miri setup --target=aarch64-fuchsia --verbose
2025-05-28T02:24:19.6865348Z Preparing a sysroot for Miri (target: aarch64-fuchsia)...
2025-05-28T02:24:19.7483810Z error: failed to run `rustc` to learn about target-specific information
2025-05-28T02:24:19.7484131Z 
2025-05-28T02:24:19.7484225Z Caused by:
2025-05-28T02:24:19.7486141Z   process didn't exit successfully: `/opt/compiler-explorer/rust-miri-nightly/bin/cargo-miri - --crate-name ___ --print=file-names -Zforce-unstable-if-unmarked -Aunexpected_cfgs -Cdebug-assertions=off -Coverflow-checks=on --target aarch64-fuchsia --crate-type bin --crate-type rlib --crate-type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro --print=sysroot --print=split-debuginfo --print=crate-name --print=cfg -Wwarnings` (exit status: 1)
2025-05-28T02:24:19.7487970Z   --- stderr
2025-05-28T02:24:19.7488431Z   error: error loading target specification: could not find specification for target "aarch64-fuchsia"
2025-05-28T02:24:19.7488922Z     |
2025-05-28T02:24:19.7489228Z     = help: run `rustc --print target-list` for a list of built-in targets
2025-05-28T02:24:19.7489518Z 
2025-05-28T02:24:19.7504377Z fatal error: failed to build sysroot: sysroot build failed
2025-05-28T02:24:27.7136177Z ##[error]Process completed with exit code 1.
2025-05-28T02:24:27.7182498Z ##[group]Run ./post-run.sh "miri" "" "" "1748397209"
2025-05-28T02:24:27.7182877Z �[36;1m./post-run.sh "miri" "" "" "1748397209"�[0m

@narpfel
Copy link
Member Author

narpfel commented May 28, 2025

can you please run and confirm the docker image runs locally if possible?

I was trying to avoid that (I tested with just a few common (tier 1) targets), but in the end it wasn’t that bad. Builds successfully now. The “fix” was to not abort the whole build if one target fails to build.

See compiler-explorer/misc-builder#114.

@mattgodbolt
Copy link
Member

thanks @narpfel ! sorry this is so painful

@mattgodbolt
Copy link
Member

@mattgodbolt
Copy link
Member

Success!!

@mattgodbolt mattgodbolt merged commit 02b398d into compiler-explorer:main May 29, 2025
5 checks passed
@mattgodbolt
Copy link
Member

Now to work out why it's not finding the built stuff...

@mattgodbolt
Copy link
Member

  File "/home/ubuntu/infra/bin/lib/installable/archives.py", line 109, in __init__
    raise RuntimeError(f"Unable to find nightlies for {compiler_name}")
RuntimeError: Unable to find nightlies for rust-miri-nightly

@mattgodbolt
Copy link
Member

image
it's called "miri" not 'rust-miri' - will fix

@narpfel narpfel deleted the rust-install-miri branch May 29, 2025 18:04
@narpfel
Copy link
Member Author

narpfel commented May 29, 2025

it's called "miri" not 'rust-miri' - will fix

Thanks! I wasn’t sure if it’s miri (like the build script) or rust-miri (like the resulting tarball).

@mattgodbolt
Copy link
Member

Installing compilers/rust/newer/miri nightly
2025-05-29 17:53:20,434 lib.installation_context INFO     Fetching https://s3.amazonaws.com/compiler-explorer/opt/miri-nightly-20250529.tar.xz (1999229504 bytes)
2025-05-29 17:53:25,540 lib.installation_context INFO     10.1% of https://s3.amazonaws.com/compiler-explorer/opt/miri-nightly-20250529.tar.xz...
2025-05-29 17:53:30,545 lib.installation_context INFO     21.2% of https://s3.amazonaws.com/compiler-explorer/opt/miri-nightly-20250529.tar.xz...
2025-05-29 17:53:35,744 lib.installation_context INFO     32.3% of https://s3.amazonaws.com/compiler-explorer/opt/miri-nightly-20250529.tar.xz...
2025-05-29 17:53:40,794 lib.installation_context INFO     42.8% of https://s3.amazonaws.com/compiler-explorer/opt/miri-nightly-20250529.tar.xz...
2025-05-29 17:53:46,019 lib.installation_context INFO     52.0% of https://s3.amazonaws.com/compiler-explorer/opt/miri-nightly-20250529.tar.xz...
2025-05-29 17:53:51,098 lib.installation_context INFO     61.7% of https://s3.amazonaws.com/compiler-explorer/opt/miri-nightly-20250529.tar.xz...
2025-05-29 17:53:56,261 lib.installation_context INFO     72.2% of https://s3.amazonaws.com/compiler-explorer/opt/miri-nightly-20250529.tar.xz...
2025-05-29 17:54:01,445 lib.installation_context INFO     82.2% of https://s3.amazonaws.com/compiler-explorer/opt/miri-nightly-20250529.tar.xz...
2025-05-29 17:54:06,561 lib.installation_context INFO     92.7% of https://s3.amazonaws.com/compiler-explorer/opt/miri-nightly-20250529.tar.xz...
2025-05-29 17:54:10,273 lib.installation_context INFO     100% of https://s3.amazonaws.com/compiler-explorer/opt/miri-nightly-20250529.tar.xz
2025-05-29 17:54:10,277 lib.installation_context INFO     Piping to tar Jxf -
2025-05-29 18:02:05,562 lib.installation_context INFO     Moving from staging (/opt/compiler-explorer/staging/3d43463c-edb0-4e92-9ff6-e15802044a0d/miri-nightly-20250529) to final destination (/opt/compiler-explorer/miri-nightly-20250529)
2025-05-29 18:02:05,583 lib.installation_context INFO     Directory listing of staging:
total 4
drwxrwxr-x 7 ubuntu ubuntu 6144 May 29 18:01 rust-miri-nightly

2025-05-29 18:02:59,597 lib.ce_install  INFO     compilers/rust/newer/miri nightly failed to install: Missing source '/opt/compiler-explorer/staging/3d43463c-edb0-4e92-9ff6-e15802044a0d/miri-nightly-20250529'
Traceback (most recent call last):
  File "/home/ubuntu/infra/bin/lib/ce_install.py", line 433, in install
    installable.install()
  File "/home/ubuntu/infra/bin/lib/installable/archives.py", line 185, in install
    self.install_context.move_from_staging(staging, self.local_path, self.install_path)
  File "/home/ubuntu/infra/bin/lib/installation_context.py", line 262, in move_from_staging
    raise RuntimeError(f"Missing source '{source}'")
RuntimeError: Missing source '/opt/compiler-explorer/staging/3d43463c-edb0-4e92-9ff6-e15802044a0d/miri-nightly-20250529'

0 packages installed OK, 0 skipped, and 1 failed installation
Failed:
  compilers/rust/newer/miri nightly

well, that didn't work unfortunatlely. I'll fix. the name of the container needs to match the exported name :) It'll be a quick fix

@mattgodbolt
Copy link
Member

Sorry this is taking so long, a number of subtle issues I didn't spot ... I'll get this in the next day or two for sure though

@mattgodbolt
Copy link
Member

Finally installed!

mattgodbolt pushed a commit to compiler-explorer/compiler-explorer that referenced this pull request May 30, 2025
@narpfel
Copy link
Member Author

narpfel commented May 30, 2025

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants