Inconsistent URL decoding of HttpRequest.Path with percent-encoded characters

[emz00]

Describe the bug

HttpContext.Request.Path seems to decode percent-encoded URLs sometimes, but not all the time. The documentation says nothing about encoding. I expect it to never decode the path, but testing on ASP.NET Core 3.1 I get results like this:

Request URL	context.Request.Path	Percent decoded?
http://localhost:5000/test%24	/test$	Yes
http://localhost:5000/test%24aa	/test$aa	Yes
http://localhost:5000/test%25a	/test%25a	No
http://localhost:5000/test%25aa	/test%aa	Yes
http://localhost:5000/test%20aa	/test%20aa	No

To Reproduce

Create an empty ASP.NET Core project and change the Startup class to

    public class Startup
    {
        public void ConfigureServices(IServiceCollection services)
        {
        }

        public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
        {
            app.Use(async (context, next) =>
            {
                Console.WriteLine("Request path = " + context.Request.Path);
                await context.Response.WriteAsync("Request path = " + context.Request.Path);
            });
        }
    }

Program class unchanged from the template:

    public class Program
    {
        public static void Main(string[] args)
        {
            CreateHostBuilder(args).Build().Run();
        }

        public static IHostBuilder CreateHostBuilder(string[] args) => Host.CreateDefaultBuilder(args).ConfigureWebHostDefaults(webBuilder => { webBuilder.UseStartup<Startup>(); });
    }

Then send the above GET requests to it from a web browser or curl.

Further technical details

• ASP.NET Core version: 3.1
• dotnet --info:

dotnet --info
.NET SDK (reflecting any global.json):
 Version:   5.0.200-preview.21079.7
 Commit:    580b1c3770

Runtime Environment:
 OS Name:     Windows
 OS Version:  6.1.7601
 OS Platform: Windows
 RID:         win7-x64
 Base Path:   C:\Program Files\dotnet\sdk\5.0.200-preview.21079.7\

Host (useful for support):
  Version: 5.0.3
  Commit:  c636bbdc8a

.NET SDKs installed:
  3.0.100 [C:\Program Files\dotnet\sdk]
  5.0.102 [C:\Program Files\dotnet\sdk]
  5.0.103 [C:\Program Files\dotnet\sdk]
  5.0.200-preview.21079.7 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.All 2.1.25 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.25 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.25 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.1.12 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.3 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

• The IDE (VS / VS Code/ VS4Mac) you're running on, and its version: VS 16.8.6
• OS: Windows 7 x64

[Tratcher]

Which server are you using (Kestrel, HttpSys, IIS in-proc, IIS out-of-proc)? The servers handle un-escaping the incoming path.

This part of your example is misleading, PathString.ToString will re-encode anything not valid in an encoded url.

Console.WriteLine("Request path = " + context.Request.Path);

To see what the server generated it should be:

Console.WriteLine("Request path = " + context.Request.Path.Value);

Run your tests again and see if you get the expected values now.

[Tratcher]

FYI the docs for the various PathString APIs give more details about the expected encodings.

[emz00]

Thanks, you're right that context.Request.Path.Value consistently returns the URL-decoded text. I'll use that!

So it looks to me like PathString.ToString() is doing the wrong thing here. new PathString("/%aa").ToString() returns "/%aa", but I would expect it to return "/%25aa", since the value passed to the constructor contains an unescaped '%' and ToString() is supposed to escape it.

[emz00]

Further, if I construct a PathString with encoded slashes, there seems to be no way to get those back out:

            var pathStr = PathString.FromUriComponent("/one/two%2Fwith%2Fslashes/three");
            Console.WriteLine(pathStr.Value); // Prints "one/two/with/slashes/three", as expeced
            Console.WriteLine(pathStr.ToString()); // Also prints "one/two/with/slashes/three"!

[Tratcher]

Those are certainly more interesting cases to investigate. I know the servers don't use FromUriComponent, they have their own decoder for various reasons, and they avoid un-escaping %2F because of the ambiguity it would create.

The %aa case I'd have to look at, but it's likely due the the fact that %aa resembles a valid escape sequence (but isn't).

We've moved this issue to the Backlog milestone. This means that it is not going to be worked on for the coming release. We will reassess the backlog following the current release and consider this item at that time. To learn more about our issue management process and to have better expectation regarding different types of issues you can read our Triage Process.

[emz00]

@Tratcher OK, but how am I supposed to get the correct request path from HttpContext.Request ? If the client requested "/one/two%2Fwith%2Fslashes/three" how do I get it, either encoded or decoded? I cannot use Path.Value here, because that doesn't distinguish between "/one/two%2Fwith%2Fslashes/three" and "/one/two/with/slashes/three" and I cannot use Path.ToString() either!

[Tratcher]

HttpContext.Request.Path.Value should be correct when working with an actual server (except maybe IIS?). Which servers have you tried?