Path to registry index is not cross-platform stable (i.e. hash part of `registry/index/index.crates.io-<hash>`)

[uncomfyhalomacro]

Hello. I have an issue with cargo-fetch behaving differently than cargo-vendor. In a part of the help section of cargo fetch, it says for the flag --target

Fetch for the given architecture. The default is all architectures. The
general format of the triple is <arch><sub>-<vendor>-<sys>-<abi>.
Run rustc --print target-list for a list of supported targets. This flag may
be specified multiple times.

However, I believe this is untrue since in some architectures, they cannot find their dependencies. I haven't tried the --target flag yet as a workaround but given that some target architectures cannot find their dependencies is weird to me, since it says that the default is to use "all" target architectures.

Here is a comparison between cargo vendor vs cargo fetch.

• Vendor https://build.opensuse.org/package/show/home:uncomfyhalomacro:branches:editors:vendor/kak-lsp
• Fetch https://build.opensuse.org/package/show/home:uncomfyhalomacro:branches:editors:registry/kak-lsp

You can see here that in https://build.opensuse.org/package/show/home:uncomfyhalomacro:branches:editors:registry/kak-lsp, only aarch64, x86_64, and ppc64le are able to find their dependencies and compile (whether it fails or not). i586, armv7l, and s390x are the architectures that were not able to find their dependencies.

Everything in https://build.opensuse.org/package/show/home:uncomfyhalomacro:branches:editors:vendor/kak-lsp are able to find their dependencies and are able to compile (whether it fails or not).

I would like to know if there is an inaccuracy with the documentation or not.

For context, I've been helping out to build a tool to ease the process of vendoring dependencies in openSUSE/SUSE. https://github.com/openSUSE-Rust/obs-service-cargo.

[weihanglo]

Thanks for the bug report. Could you attach some additional information, like

• The version of Cargo (the output of cargo -vV) was observed with this behavior.
• If this is a regression, the last successful version of Cargo.
• The exact Cargo commands or scripts ran in that service.
• Any other [source] replacement or proxies configured in the server or not.

cargo vendor and cargo fetch has no difference in the sense of dependency resolution (except cargo vendor accepts multiple workspaces). However, there are slightly different when filtering platforms:

cargo fetch fetches only packages in the dependency resolution (also seen as packages recorded in Cargo.lock) that could be reached from workspace members.

cargo/src/cargo/ops/cargo_fetch.rs

Lines 35 to 64 in cb088dc

	while let Some(id) = deps_to_fetch.pop() {
	if !fetched_packages.insert(id) {
	continue;
	}
	to_download.push(id);
	let deps = resolve
	.deps(id)
	.filter(|&(_id, deps)| {
	deps.iter().any(|d| {
	// If no target was specified then all dependencies are
	// fetched.
	if options.targets.is_empty() {
	return true;
	}
	// Otherwise we only download this dependency if any of the
	// requested platforms would match this dependency. Note
	// that this is a bit lossy because not all dependencies are
	// always compiled for all platforms, but it should be
	// "close enough" for now.
	build_config
	.requested_kinds
	.iter()
	.any(|kind| data.dep_platform_activated(d, *kind))
	})
	})
	.map(|(id, _deps)| id);
	deps_to_fetch.extend(deps);
	}

cargo vendor, in contrast, fetches everything in the dependency resolution. And there is no way for cargo vendor to filter packages by platforms at this moment.

cargo/src/cargo/ops/vendor.rs

Line 119 in cb088dc

.get_many(resolve.iter())

Bugs might be there if they exist.

[uncomfyhalomacro]

Hi @weihanglo . This the version of cargo running on my container installed through zypper.

cargo 1.82.0 (8f40fc59f 2024-08-21)
release: 1.82.0
commit-hash: 8f40fc59fb0c8df91c97405785197f3c630304ea
commit-date: 2024-08-21
host: x86_64-unknown-linux-gnu
libgit2: 1.8.1 (sys:0.19.0 vendored)
libcurl: 8.10.1 (sys:0.4.74+curl-8.9.0 system ssl:OpenSSL/3.2.3)
os: openSUSE 20241106.0.0 [64-bit]

The commands ran in that service is as follows.

1. cargo-fetch with flags --manifest-path. If my service is set with respect-lockfile to true, a --locked flag will be added.
2. cargo-vendor with flags --manifest-path. Additional manifest paths will be passed with --sync. If my service is set with respect-lockfile to true, a --locked flag will be added.

If this is a regression, the last successful version of Cargo.

I wouldn't know of a regression as this is the first time that I noticed this and my first bug report about it.

Any other [source] replacement or proxies configured in the server or not.

None. I just use whatever is the default.

cargo fetch fetches only packages in the dependency resolution (also seen as packages recorded in Cargo.lock) that could be reached from workspace members.

So shouldn't the help section of cargo-fetch be updated?

If --target is not specified, then all target dependencies are fetched

This seems to confuse me more as this was the only assumption I have from the help section but the behaviour for what I expected was different than what was described. Should fetch at least have the same behaviour as vendor that they both fetch everything in the dependency resolution by default?

[uncomfyhalomacro]

Hi. I tried using the --target flag in this commit openSUSE-Rust/obs-service-cargo@a34992c for my project but it seems it's still not budging in https://build.opensuse.org/package/show/home:uncomfyhalomacro:branches:editors:registry/kak-lsp

[weihanglo]

So shouldn't the help section of cargo-fetch be updated?

Either this or make the behavior of both commands aligned. Let me try to get a minimal reproduction first.

[weihanglo]

Let me try to get a minimal reproduction first.

I've re-read cargo-fetch logic. While it may fail to fetch packages that have unstable features like forced-tarrget or artifact depenedencies, for normal circumstance it should just work.

It is surprising that in your build log it's crossbeam-channel missing

[   66s] + /usr/bin/cargo auditable build -j8 --offline --release --all-features
[   66s] error: no matching package named `crossbeam-channel` found
[   66s] location searched: registry `crates-io`
[   66s] required by package `kak-lsp v18.0.3 (/home/abuild/rpmbuild/BUILD/kakoune-lsp-18.0.3)`
[   66s] As a reminder, you're using offline mode (--offline) which can sometimes cause surprising resolution failures, if this error is too confusing you may wish to retry without the offline flag.
[   66s] error: Bad exit status from /var/tmp/rpm-tmp.8OUc67 (%build)

If I read it correctly, crossbeam-channel is not even a platform-specific dependency. I wonder what is inside the registry.tar.zst tarball. Maybe it is corrupted or missing something when packing?

[uncomfyhalomacro]

It's not corrupted or missing since doing the following

raw -t registry.tar.zst -d registry/
cd registry/.cargo/registry/cache/index.crates.io-6f17d22bba15001f
fd crossbeam

gives me this output

crossbeam-channel-0.5.13.crate
crossbeam-utils-0.8.20.crate

The other variant which uses the vendor method produces a file vendor.tar.zst

Extracting it and running fd crossbeam to it will give me this output

crossbeam-channel-0.5.13/
crossbeam-channel-0.5.13/benches/crossbeam.rs
crossbeam-utils-0.8.20/

So I am not really sure how cargo-fetch and cargo-vendor is any different when they both have crossbeam.

[weihanglo]

registry/.cargo/registry/index/index.crates.io-6f17d22bba15001f

Do we have the correct index files under this location? Perhaps you can upload a tarball that failed to build and we can investigate together.

[uncomfyhalomacro]

registry/.cargo/registry/index/index.crates.io-6f17d22bba15001f

Do we have the correct index files under this location? Perhaps you can upload a tarball that failed to build and we can investigate together.

Hi. It seems I am not able to upload a tarball. One thing you can do to perform a reproduction is by doing the following

cargo install --git https://github.com/openSUSE-Rust/obs-service-cargo
git clone --depth 1 --branch v18.0.3 https://github.com/kakoune-lsp/kakoune-lsp/
cd kakoune-lsp
cargo_vendor --src $PWD --outdir /tmp
cd /tmp/
mkdir vendored-sources/
tar xf vendor.tar.zst -C vendored-sources/
cd vendored-sources/

by now you can check the inside. It should contain a Cargo.lock and the .cargo directory which contains the registry stuff.

[weihanglo]

@uncomfyhalomacro the current obs-service-cargo openSUSE-Rust/obs-service-cargo@bde0300 generates only vendored source. There is no registry index file or tarball at all. I am not sure which commit I should use to reproduce.

BTW, you could also run fd -hidden crossbeam under registry/index/index.crates.io-6f17d22bba15001f/.cache and check if there is any index file called crossbeam-channel. If no, there you need to pack it.