HTTP streaming client, and trailer support

[sunfishcode]

Add a wstd API for creating proxy applications, wrapping the WASI proxy world trait and macro.

Compiling the example with RUSTFLAGS="-Clink-arg=--wasi-adapter=proxy" produces a program that runs in wasmtime serve.

Marking as a draft for now, as it's still experimental. Feedback on the approach or "is this even a reasonable idea" welcome 😄.

[yoshuawuyts]

Thank you for submitting this! In general, I'm a big fan of making people's lives easier, and I believe that the general thrust of what you're filing here is the right thing to do. So, yes - I believe this is reasonable!

On a first pass, this is what stood out to me:

1. I quite enjoy the pattern Rust has for main being the main entry point of the application. While I know Components blur the boundary between libraries and binaries since everything is virtualizable, I feel like it would be especially nice here if fn main for HTTP proxies could Just Work? How would you feel about changing that?
2. There is a fair bit of mapping code inside of the macro happening. Could we turn those into conversion trait impls on the types instead? That keeps as much code out of the macros as possible.
3. Is fn main(req) -> res the right pattern we want to implement? I believe that might lead to some trouble handling HTTP trailers?

[sunfishcode]

1. I quite enjoy the pattern Rust has for `main` being the main entry point of the application. While I know Components blur the boundary between libraries and binaries since everything is virtualizable, I feel like it would be especially nice here if `fn main` for HTTP proxies could Just Work? How would you feel about changing that?

Do you still like calling it main if it can be called multiple times on the same instance?

2. There is a fair bit of mapping code inside of the macro happening. Could we turn those into conversion trait impls on the types instead? That keeps as much code out of the macros as possible.

Yeah, I'll work on factoring that out.

3. Is `fn main(req) -> res` the right pattern we want to implement? I believe that might lead to some trouble handling HTTP trailers?

I think it'll lead to some trouble handling bodies too :-}. I need to revamp the Response/Request types, and maybe even have separate types for the incoming-request side of things.

[pchickey]

I was delayed a bit by starting a new job, but I've rewritten the reactor and streams abstractions in #35 #37. Hopefully that makes some aspects of this a bit simpler. I also added a bunch of common code to translate between the wasi-http types and the http crate types, so a bunch of the boilerplate in here can hopefully go away.

I'd like to avoid forcing users to ever use/know about RUSTFLAGS to compile applications. We can produce a HTTP Proxy world application with an ordinary cargo build --target wasm32-wasip2 of a bin with a wasi::http::proxy::export!(Wrapper) long as there is a "dummy" fn main() {} for the linker to export as the cli world run, so lets make #[wstd::proxy] async fn main(_: Request) -> Response {...} work like that. I think in this framework we should err on the side of letting the common case be straightforward - the user wants to write a proxy, and doesn't care that it also happens to impl the CLI world's export in a vacuous way - rather than enabling sophisticated behavior, like the user that has both a CLI world run export and a proxy export. Sophisticated users can use other methods to get sophisticated behavior, such as using the wasi crate directly and passing RUSTFLAGS if they want.

[sunfishcode]

I've now overhauled the API here.

• It now uses fn main syntax for the entrypoint.
• All the conversion code is now factored out of the macro.
• I switched from the response return type to a Resonder argument, which allows it to support trailers as well as streaming bodies.
• I added a placeholder fn main to make the crate compile with wasm32-wasip2 with no special RUSTFLAGS.

Here's a simple hello-world example:

#[wstd::proxy]
async fn main(request: Request<IncomingBody>, responder: Responder) -> Finished {
    responder
        .respond(Response::new(b"Hello, wasi:http/proxy world!\n"))
        .await
}

For more interesting examples, including a streaming body, see the examples/http_server.rs in the PR.

think in this framework we should err on the side of letting the common case be straightforward - the user wants to write a proxy, and doesn't care that it also happens to impl the CLI world's export in a vacuous way - rather than enabling sophisticated behavior, like the user that has both a CLI world run export and a proxy export.

I've now implemented this. That said, a downside is that it's not just the export, it's also the imports, and it means that the resulting proxy components don't run in plain wasmtime serve; they need wasmtime serve -S cli.

[sunfishcode]

With other features factored out into separate PRs that have now landed, this PR now contains a series of patches adding output-streaming client and client trailers support, and a nice example.

[pchickey]

I think this can be https://doc.rust-lang.org/std/future/trait.IntoFuture.html ?

[sunfishcode]

I've now updated it to use impl Future.