Refactor test-programs to build modules and components

[pchickey]

This is task 2 in #6370

On the tails of #6374, this adds support for test-programs to build components out of preview 1 modules.

But thats really just the side story for this PR, which rewrites the entire way that test-programs works. Over in preview2-prototyping, I got tired of doing a bunch of code generation of all the #[test] functions, which makes it tests to read and hard to modify.

So, I trimmed code generation in test-programs/build.rs down to the bare minimum: it calls out to cargo to build the test program crates, and then we have a function module_rs() which writes a source file {test-crate}_modules.rs into OUT_DIR. This source file defines a function get_module(&str) -> Module which gives the user the module corresponding to the binary name.

For test program crates which can be built into components, you can invoke component_rs() which creates {test-crate}_components.rs in OUT_DIR, and you use get_component(&str) -> wasmtime::component::Component. One extra argument is required: a built adapter, which is the command build of wasi-preview1-component-adapter from this tree.

As part of this rewrite, we generate the components for the test crates, but we don't actually execute them yet, because we need a host implementation of preview 2 to do so. Sit tight - that is coming in the very next PR!

[pchickey]

I am applying a temporary exception to deny.toml for duplicate dependencies on wasm-encoder, wasmparser, and wit-parser. The wasmtime project versions have fallen behind the latest wit-bindgen, and @itsrainy is going to update those shortly with their core dumps work.

I would typically just wait for that sort of work to finish up to land my PR, but I want to move along quickly so we can change the canonical home of the preview 2 work to this repository, which will unblock several other people.

[alexcrichton]

Oh? I've not seen this dependency before, what does it do?

[pchickey]

It is provides a wrapper over #[test] (or e.g. #[tokio::test] if you pass it in as an argument) which takes care of installing tracing-subscriber with an env filter before starting your test. It gets rid of a Once cell we used to do that manually.

[alexcrichton]

Could this diff be audited to avoid adding a new exemption? (assuming that this diff is a relatively small one hopefully)

[pchickey]

I could, I just figured that if we have an exemption I'd keep bumping it. I've been thinking about our exemptions of the tokio/hyper family (which this is a part of) is a wildcard audit for carl and sean.

[alexcrichton]

Ok sounds reasonable 👍

[alexcrichton]

Could you check to see if 0.2.13, which is exempted, is a small or big diff from 0.3.5? If small mind adding an audit for that? If big though it's a bit of a bummer but we can probably live with an exemption.

[pchickey]

I didn't want to do any sort of audit on this because it is the redox syscall layer, an OS I don't know anything about and don't think we care about in the context of this project. It just happens to be a transitive dep in some crates.

[alexcrichton]

Ok sounds reasonable 👍

[pchickey]

#6385 (comment) This change was backed out as part of merging with main after #6394 landed. Thank you @itsrainy.

Unfortunately, even with those fixes, there were still multiple versions of wasm-tools crates in the workspace, via the transitive dependencies of wit-bindgen. Alex and I decided that, since wit-bindgen is only used to build component tests, we will add a deny exception for its dependency tree.