WebClient http proxy broken in .Net Core 3 Preview 7

[roglainc]

I am using WebClient and HttpWebRequest to call external Http APIs and is dependent on a proxy to make external calls. After upgrading to .Net Core Preview 7 I get Proxy authentication error (407). The code runs ok in .Net Core 2.1, and .Net Core 3 preview 3.

This is a snippet that
succeeds on .Net Core 2.1 & 3.0 preview 3
but fails on preview 7,
running in Visual Studio 2019 16.2.0:

            string proxy = <_name of Proxy_>;
            string url = "https://www.kemi.se/global/faktablad/faktablad-om-sakerhetsdatablad.pdf";
            var client = new WebClient() {
                Proxy = new WebProxy(proxy, 8080) {
                    UseDefaultCredentials = true
                }
            };
            var data = client.DownloadData(url);

[stephentoub]

cc: @davidsh

[davidsh]

@roglainc Can you give a few more details? Is this on Windows or Linux? I'm thinking Windows because you mention Visual Studio 2019.

Have you also tried HttpClient as well?

[davidsh]

I tried to reproduce this with .NET Core 3.0 Preview 7 but I wasn't able to. I used a proxy (running on "localhost") and it was able to respond to the 407 and send back the appropriate 'Proxy-Authorization: Negotiate base64blob' headers to the proxy.

Have you tried running your program outside of Visual Studio?

[roglainc]

I am running Windows 10. As a workaround I reverted to .Net Core Preview 6, no problem in that version it seems. Since WebClient/HttpWebRequest fits my code pattern I have not tried HttpClient.

Summary:

OS: Windows 10
Visual studio: Visual Studio 2019 16.2.0

.Net Core 2.1: OK
.Net Core 3.0 Preview 3 OK
.Net Core 3.0 Preview 6 OK
.Net Core 3.0 Preview 7 Failure ( Proxy authentication error)

[roglainc]

I reverted to Preview 6 before I saw your question about running outside Visual Studio. I have only tried running from Visual Studio since I'm in development right now.. Our proxy is on a central server.

[davidsh]

I have only tried running from Visual Studio since I'm in development right now..

I just tried the repro using Visual Studio 2019 16.2.0. I'm still not getting a 407 error. It is able to send credentials to the proxy and do the request to the server.

Can you collect a Wireshark trace of this scenario? It will show what is going on and we can diagnose it further.

[roglainc]

Now I switched back to .Net Core Preview 7 and the problem reappear. Success in preview 6 and failure in preview 7. The failure occurs when running outside Visual Studio as well. I am using x64:

Summary:
OS: Windows 10
Visual studio: Visual Studio 2019 16.2.0
Platform: .Net Core x64
Occurs outside Visual Studio as well.

.Net Core 2.1: OK
.Net Core 3.0 Preview 3 OK
.Net Core 3.0 Preview 6 OK
.Net Core 3.0 Preview 7 Failure ( Proxy authentication error)

[davidsh]

Success in preview 6 and failure in preview 7.

If you could collect Wireshark traces for both the preview 6 and preview 7 cases that would be useful. We can then compare them.

[roglainc]

I need permission to use Wireshark in my organisation. We are in timezone UTC+2 so I have to wait until tomorrow before I can have traces.

[davidsh]

I need permission to use Wireshark in my organisation. We are in timezone UTC+2 so I have to wait until tomorrow before I can have traces.

Ok. Please do not post any traces that contain confidential or proprietary information. You can remove any of that info prior to posting. Or perhaps post screenshots of traces masking out confidential information.

It does sound like something is going on since your proxy scenario is working in Preview 6 but not Preview 7. But I'm not able to reproduce the problem. It could be related to something more specific in your corporate environment. The traces will help.

[roglainc]

Now I have traces from preview 6 (ok)

and preview7 (failure)

[roglainc]

The Proxy authentication token seems to differ between preview 6 and 7:

Preview 7:
Proxy-Authorization: Negotiate .... (56 bytes, padded with == a end and followed by CR/LF) [Failure]

Preview 6:
Proxy-Authorization: Negotiate ... (does not see padding or any CR/LF) [OK]

I did a quick hack to create a middle man process and injected it between the application and the proxy server to log the dialogue. The call did not get through, might due to my error or some protection in the protocol to prevent middle mans, but the result was:

Preview 7:
Proxy-Authorization: Negotiate .... (56 bytes, padded with == a end and followed by CR/LF) [Failure]

Preview 6:
Proxy-Authorization: Negotiate ... (176 bytes, no padding and followed by CR/LF) [Failure]

[davidsh]

Preview 6:
Proxy-Authorization: Negotiate ... (does not see padding or any CR/LF) [OK]

I don't quite understand this. The 'Proxy-Authorization' header should always have CR/LF after it. Just like any other header in the request. Are you saying that it's missing entirely?

[davidsh]

I think I am beginning to see a problem now.

I did a quick test between .NET Core 2.1 and .NET Core 3.0 Preview 7. I wrote a small test app that connects to an HTTPS server thru an authenticating proxy server that uses Negotiate scheme.

I see a difference in the base64 blob that is sent. For example:

.NET Core 2.1

HTTP/1.0 407 Proxy Authentication Required
Proxy-agent: Netscape-Proxy/1.1
Proxy-Authenticate: Negotiate
Content-Length: 0
Date: Wed, 31 Jul 2019 19:04:11 GMT

CONNECT corefx-net.cloudapp.net:443 HTTP/1.1
Host: corefx-net.cloudapp.net:443
Proxy-Authorization: Negotiate YH4GBi<omitted>VY=

.NET Core 3.0 Preview 7

HTTP/1.0 407 Proxy Authentication Required
Proxy-agent: Netscape-Proxy/1.1
Proxy-Authenticate: Negotiate
Content-Length: 0
Date: Wed, 31 Jul 2019 19:00:48 GMT

CONNECT corefx-net.cloudapp.net:443 HTTP/1.1
Host: corefx-net.cloudapp.net:443
Proxy-Authorization: Negotiate TlRMTV<omitted>==

You'll notice in the .NET Core 2.1 case, the base64 token begins with "Y". That means that SPNEGO (Kerberos) protocol is being used against the 'Negotiate' scheme offered by the proxy.

But in the .NET Core 3.0 Preview 7 case, the base64 token begins with "T". That means that NTLM is being used against the 'Negotiate' scheme offered by the proxy.

My guess is that your corporate proxy is very strict and requires SPNEGO protocol only when 'Negotiate' is challenged.

Can you confirm that you see a similar difference in the base64 tokens for the 'Proxy-Authorization' request headers that are being sent? Thanks.

By the way, I didn't see any problem when connecting to a non-HTTPS server thru an authenticating proxy. It always seemed to use SPNEGO protocol for both .NET Core 2.1 and .NET Core 3.0 Preview 7.