Enable custom header encoding

[Tratcher]

Officially request and response headers are restricted to ASCII. Historically however ANSI and local code pages were supported. There are also some cases where UTF-8 was used in for Unicode host names prior to the punycode standard (IE and HttpWebReqeuest on intranets), see Host and Location.

Default header encoding/decoding to utf-8. Allow this encoding to be set at startup. For performance reasons the encoding/decoding process can assume simple ASCII until it encounters 8bit values.

Related: #491 pre-encoded response headers.

RE: aspnet/HttpSysServer#123 (comment) [private]

[benaadams]

As there are two servers, could take the IE11+Edge route. If you want to support historical non-standard stuff use WebListener (e.g. intranets/because your site uses them which you should know)?

[Tratcher]

It's primarily an interop issue. You often do not control what the client sends you (or expects back), you just have to find a way to make it work. Standards are definitely the default but we need a well defined behavior for addressing non-standard input/output.

How does Kestrel behave at the moment? Does it throw or mangle the data?

[benaadams]

How does Kestrel behave at the moment? Does it throw or mangle the data?

It stays uninterpreted (tests)|(code) so 255 or 0 stays as 0 or 255 in the string.
As per last section of rfc7230#section-3.2.4

Newly defined header fields SHOULD limit their field values to
US-ASCII octets. A recipient SHOULD treat other octets in field
content (obs-text) as opaque data.

So you could use binary headers if you wanted and that would work (assuming you stayed away from \r in the data as that causes additional delimiter interpretation); which you couldn't do if it was interpreting as utf8 and transforming the data as that would be one-way.

Could add an extension like Header.InterpreteAs(Encoder)?

[benaadams]

Also since it only encodes 0-255 of char (both on input and output) and doesn't do any interpretation of it; you can embed pre-encoded stuff in it currently by putting it in the string that way - although it will use slightly more room - e.g. 4 bytes for 2 bytes

[Tratcher]

String is a very bad representation of opaque data, incoming or outgoing. I've tried the embed-utf-8-in-a-string hack before with HttpClient and it almost works, but it confuses anyone else that tries to look at the data.

I don't think it's appropriate for WebListener to have different behavior on something as basic as header encoding.

[benaadams]

Isn't embedding utf-8 in a string its just maintaining it in its input and output binary representation without changes (other than widening the data type)?

[benaadams]

Am suggesting something like typed headers so you could go

string value;
value = header[value]; // Raw value also 1:1 with ASCII
value = header[value].AsUft8();
value = header[value].AsUft16();
value = header[value].AsUft32();
value = header[value].AsEncoding(encoder);

byte[] buffer;
buffer = header[value].AsBinary(); // thins the datatype back to byte from char

Rather than trying to guess the encoding

[benaadams]

Hmm, ok so that would work for inbound headers but would clobber outbound

[benaadams]

So historically non-ascii should be MIME (RFC 2047) encoded?

e.g. UTF-8 should start: =?UTF-8?...

[Tratcher]

No, that was for smtp and sometimes multipart headers, not http.

[benaadams]

Ok, to be clear, the scenario being supported is an old client communicating with new server-side code (as aspnet5 isn't lift and shift, the servers aside); and depending on the values in the headers to be not ascii?

With IE browsers before 11 passing into support end of life 4 days ago (depending on OS)
https://support.microsoft.com/en-us/lifecycle#gp/Microsoft-Internet-Explorer

[benaadams]

Ah so is a Windows only AD thing http://stackoverflow.com/questions/16837513/how-do-i-know-when-to-do-a-utf8-or-punycode-dns-query

On Windows, there any two possibilities: either the domain name is a normal internet one, in which case it should be mapped to ASCII with punycode and looked up with normal DNS. Or, the domain name is a weird Windows one (eg. Active Directory DNS server) and the lookup should in fact be UTF-8.

Though do the servers check the host header? The request has already be pre-routed to them based on it.

[davidfowl]

Ok, to be clear, the scenario being supported is an old client communicating with new server-side code (as aspnet5 isn't lift and shift, the servers aside); and depending on the values in the headers to be not ascii?

We totally don't care about this because we're not even testing it. We're just looking for a consistent way to decode headers when we find an unexpected characters.