Kestrel rejects requests with non standard ASCII characters

[dkonyayev]

This is the same issue as #9874, except that our application is on .NET Core 3.1

Our application is on .NET Core 3.1, and Kestrel is still rejecting headers with non-standard ASCII characters. Our corporate Single-Sign On proxy inserts the name of the person into the header, so it's very common that we have names with accents or diacritics in EMEA countries.

Here's a sample request:

GET /api/provider/helloworld/tester HTTP/1.1
invalidChar: Sørina
User-Agent: PostmanRuntime/7.28.4
Accept: /
Postman-Token: 447bab9c-5f64-4600-b8dd-e3d9b74771ae
Host: localhost:5003
Connection: keep-alive

Here's the log output from Kestrel

dbug: Microsoft.AspNetCore.Server.Kestrel[17]
Connection id "0HMBNMEJMCCEM" bad request data: "Malformed request: invalid headers."
Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Malformed request: invalid headers.
at Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException.Throw(RequestRejectionReason reason)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1Connection.TryParseRequest(ReadResult result, Boolean& endConnection)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication1 application) at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequestsAsync[TContext](IHttpApplication1 application)
dbug: Microsoft.AspNetCore.Server.Kestrel[10]
Connection id "0HMBNMEJMCCEM" disconnecting.
dbug: Microsoft.AspNetCore.Server.Kestrel[2]
Connection id "0HMBNMEJMCCEM" stopped.
dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets[7]
Connection id "0HMBNMEJMCCEM" sending FIN because: "The Socket transport's send loop completed gracefully."

[Tratcher]

Kestrel accepts UTF-8 headers by default. This example is probably ANSII, not ASCII. A network trace would help confirm the actual bytes used.

3.1 was patched with a new config option Latin1RequestHeaders to allow ANSII header values. #17399

In 5.0 this was replaces with an API for selecting the header encoding: #23233.

[blowdart]

Be aware that in turning on such options, depending on your code, you may end up with unexpected problems because you're using unexpected encodings.

[dkonyayev]

@Tratcher Thanks, I have tested the config option and it's working as expected.