HttpProtocols.Http1AndHttp2 over HTTP without HTTPS/ALPN

[hagen93]

Is your feature request related to a problem? Please describe.

We are trying to write gRPC based microservices with frontends and backends in the same project the same way an ASP.NET Core MVC + Razor app would be structured. For this we need support for serving static files (provided by IApplicationBuilder.UseDefaultFiles()), serving gRPC (provided by the NuGet package Grpc.AspNetCore), and serving gRPC Web (provided by our custom middleware https://github.com/KnowitSolutions/Grpc.Helpers/blob/master/Grpc.Web/GrpcWebMiddleware.cs). Browsers consuming the static files and gRPC Web expects HTTP/1.1, while gRPC requires HTTP/2.0, which means Kestrel will have to switch back and forth between serving HTTP/1.1 and HTTP/2.0. Currently this is implemented by setting the Protocol field in Kestrel's listen option to HttpProtocols.Http1AndHttp2, but this implementation works by using ALPN to negotiate the HTTP protocol and therefore does not support plain HTTP as there is no TLS handshake for the ALPN negotiation to take place during. This makes it cumbersome to run this kind of setup in orchestrated container environments like Kubernetes and Docker Swarm because TLS termination in these environments usually happens at the cluster's ingress, and thus Kestrel would only ever see normal HTTP traffic. Introducing extra TLS inside such a cluster is also a hassle because then TLS certificates must be managed and distributed automatically. Solutions for this does exist, but for the problem at hand seems needlessly complicated.

Describe the solution you'd like

Instead of, or in addition to, using ALPN to negotiate the HTTP protocol it would be nice if Kestrel could sniff out the protocol used by the client on the fly. HTTP/2.0 can easily be detected as the protocol defines a 24 octets long mandatory preface before any other communication can take place (https://http2.github.io/http2-spec/#ConnectionHeader). In Kestrel this could be implemented by peeking the incoming stream at this point, looking for the HTTP/2.0 preface, and setting HttpConnectionContext.Protocols accordingly. As a proof of concept we have implemented a middleware that does this and changes HttpConnectionMiddleware._protocols by introspection (https://github.com/KnowitSolutions/Grpc.Helpers/blob/master/Kestrel.ProtocolMultiplexing/ProtocolMultiplexingMiddleware.cs), which works well, but having this as an official feature of Kestrel would be a lot more clean.

Additional context

As we already have a working implementation of this we would be happy to contribute a fix to the project that introduces this feature.

As an extension to this one could also implement the HTTP Upgrade mechanism (https://en.wikipedia.org/wiki/HTTP/1.1_Upgrade_header#Use_with_HTTP/2) to allow for clients to switch to HTTP/2.0 dynamically. This could for example be implemented as an ASP.NET middleware that returns HTTP 101 Switching Protocols if the header Upgrade: h2c is found. This extra automation might make HTTP/2.0 easier to use since many clients supporting it would automatically switch without extra configuration.

[Tratcher]

Yes we could make these changes, but what clients would actually support them? Neither mechanism you've proposed here is supported by any major browser. A few clients support h2c with prior knowledge. I haven't seen any that support the upgrade mechanic.

[hagen93]

Hi. Thanks for the reply. As described above, our specific use case is to implement services that we would like to be callable with both gRPC and gRPC Web (our template for such services can be seen here).

While building our software we have seen that writing more general CRUD services for our backends are usually smart, as they can often be used for multiple purposes. That might be in several different backend to backend contexts and in frontend to backend contexts. What we want to achieve with this change request is to have Kestrel support a setup where such a CRUD service can be available to both frontend clients via gRPC Web (with the help of our middleware, as described above) and backend clients via gRPC.

A setup such as the proposed one is for our case only to avoid having to run two separate web servers, or two separate bindings for Kestrel, etc, to serve these two different kinds of clients. The added suggestion for the HTTP upgrade mechanism was simply a comment to show that this also could be useful for other cases as it broadens the support for the HTTP specification, but is not a feature we care about ourself.

You are correct that most HTTP clients right now don't support HTTP/2.0 with prior knowledge, but in this case it is required by gRPC, while gRPC Web is intended to work for both HTTP/1.1 and HTTP/2.0 clients that can't implement the full gRPC spec for any reason (primarily browsers). Thus to support both at the same time Kestrel would have to be able to switch based on what the client is using.

It is also worth noting that according to https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-WEB.md it is expected that gRPC Web will be unnecessary in a few years as browsers adapt support for the whatwg streams API. This means that assuming browsers don't adopt HTTP/2.0 with prior knowledge anyone wanting to implement gRPC based frontend with ASP.NET would greatly benefit from this.

[Tratcher]

@JamesNK @JunTaoLuo thoughts on the GRPC angle here?

[Tratcher]

This means that assuming browsers don't adopt HTTP/2.0 with prior knowledge

Browsers are moving the other direction towards QUIC and HTTP/3 which mandates TLS, ALPN, etc..

[JamesNK]

@JamesNK @JunTaoLuo thoughts on the GRPC angle here?

gRPC is always going to send HTTP/2. Whether you send the request over HTTP or HTTPS is also decided on the client in advance. gRPC isn't doing any negotiation. Maybe that might change if/when QUIC/HTTP3 become a thing. I don't know.

This makes it cumbersome to run this kind of setup in orchestrated container environments like Kubernetes and Docker Swarm because TLS termination in these environments usually happens at the cluster's ingress, and thus Kestrel would only ever see normal HTTP traffic.

This is the thing to think about. HTTPS/TLS is coming up over and over and termination is one of the problems. Kestrel's Http1AndHttp2 setting isn't useful because it will always be Http1.

I'm not fimilar with HTTP/1.1 and HTTP/2 negotiation. Was the TLS requirement chosen for technical, security or time reasons?

[JamesNK]

@hagen93 FYI, Grpc.AspNetCore.Server will now validate HttpRequest.Protocol is HTTP/2. That might effect the success of what you're doing here: https://github.com/KnowitSolutions/Grpc.Helpers/blob/master/Grpc.Web/GrpcWebMiddleware.cs

grpc/grpc-dotnet#475

[hagen93]

@JamesNK Is that validation part of .NET Core 3.0 preview8? We have this running successfully on preview8 at the moment.

[JamesNK]

It is not public yet. You could try a nightly build if you want to test it: https://github.com/grpc/grpc-dotnet#grpc-nuget-feed

It might be a case of also needing to set HttpRequest.Protocol to HTTP/2 in your middleware.

[hagen93]

Yeah, as HttpContext can be modified I agree that is probably going to solve that problem. However, on the topic of TLS termination, you alluded to that the problem wasn't new? Is there other issues that have been reported you think this change will help with?

[JamesNK]

I don't know all the issues it creates, but I've heard multiple people bring it up as a problem when it comes to using HTTP/2.

[hagen93]

A scenario I could think of is that since our ingress (Traefik) supports HTTP/2.0 with prior knowledge we could potentially speed up incoming connections while still supporting HTTP/1.1 for legacy clients inside our cluster. That is unrelated to the gRPC problem originally posed, but would also be solved by this change.