NTLM authentication sometimes broken by multiple WWW-Authenticate headers

[rmkerr]

This issue has been split off from #17545, which turned out to be a problem in the tool being used to observe network traffic. Other users saw similar results, but under different conditions and not caused by the testing tool. That issue will be tracked here to clearly separate the two issues.

The issue tracked here occurs with the following code, targeting .NET Core 2.0:

var creds = new CredentialCache();
creds.Add(new Uri(addy),"NTLM",new NetworkCredential(Username,Password));
var handler = new HttpClientHandler
{
	Credentials = creds,
};

HttpClient client = new HttpClient(handler);
client.BaseAddress = new Uri(addy);
            
var response = await client.GetAsync("api/myresource");

In Windows, the server receives the following headers, but does not initiate the NTLM handshake:

HTTP/1.1 401 Unauthorized
Content-Type: text/html
Server: Microsoft-IIS/8.5
WWW-Authenticate: NTLM
WWW-Authenticate: Negotiate
X-Powered-By: ASP.NET
Date: Thu, 01 Mar 2018 22:17:34 GMT
Content-Length: 1293

@dbrownxc, can you provide more information on the situation in which you were able to reproduce this issue? It would be good to have full logs for the unsuccessful authentication attempt.

cc: @seriouz @karelz @davidsh

[dbrownxc]

I am developing on Win7 using Visual Studio. My ultimate target is a ASP.NET Core application running in Docker hosted on CentOS. I am using .NET Core 2.0.

I am reproducing the issue with a unit test running as .NET Core 2.0 application. Here is my csproj

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>netcoreapp2.0</TargetFramework>
  </PropertyGroup>

  <PropertyGroup>
    <!-- Assembly Info (for build replacement-->
    <Version>1.0.0.0</Version>
  </PropertyGroup>


  <ItemGroup>
    <!--Required for NUnit and TeamCity integration -->
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.5.0" />
    <PackageReference Include="NSubstitute" Version="3.1.0" />
    <PackageReference Include="NUnit" Version="3.9.0" />
    <PackageReference Include="NUnit3TestAdapter" Version="3.9.0" />
    <PackageReference Include="FluentAssertions" version="5.0.0-beta0004" />
    <PackageReference Include="TeamCity.VSTest.TestAdapter" Version="1.0.6" />
  </ItemGroup>

  <ItemGroup>
  </ItemGroup>
 
</Project>

I'm running with Resharper in Visual Studio and can consistently get the issue when using NTLM as the type.

On the same machine, I have a CentOS VM running Docker and I'm using the following dockerfile and just issuing a docker image build. I've mapped the directory directly to the VM so I'm sure it is the same code.

FROM microsoft/aspnetcore-build:2.0 as build
ARG TEAMCITY_VERSION=2017.2
WORKDIR /solution
COPY . ./
RUN dotnet build
ENV TEAMCITY_VERSION $TEAMCITY_VERSION
RUN ls *Test*/*.csproj | xargs -L1 dotnet test 

The from the teamcity output, I'm seeing a success using the same code that fails in Windows. If I switch the type to Negotiate, I get the opposite results. Linux fails and Windows succeeds.

These calls are to the exact same IIS server and service.

[dbrownxc]

I am working on getting cleared to send the network trace, but I do notice a difference in the failures.

After the test receives the Unauthorized HTTP packet:
When Windows fails, the test sends an ACK and receives a RST, ACK almost immediately and closes.
When Linux fails, the test sends an ACK and then sends a FIN, ACK and receives a FIN,ACK, then ACKs to close.

These results are consistently reproduceable.

[dbrownxc]

Is it ok to send trace traces directly to you?

[rmkerr]

I'll have to check if that's okay and get back to you. In the mean time though, I have some questions you might be able to answer without having to worry about disclosure issues. The key question for me is about the client response to the 401.

1. Does the client make any attempt to authenticate? If so, which auth protocol is it using?
2. Is the behavior the same when you disable either NTLM or Negotiate on the server side?
3. When Negotiate works on Windows, is it using NTLM or Kerberos as the underlying auth protocol?

Unfortunately we're probably seeing two separate issues here, since authentication handling is completely different between Windows and Linux

[dbrownxc]

1. No. After the 401, I see an ACK, and then the server sends a RST, ACK.
2. I'll try that.
3. I see Negotiate, but it says NTLMSSP
   Server > 401 Unauthorized
   Client > NTLMSSP_NEGOTIATE.
   Server > NTLMSSP_CHALLENGE
   Client > NTLMSSP_AUTH

[karelz]

@dbrownxc is it possible to create a smaller repro on your side from which you can publish the traces (and source)?

[dbrownxc]

Yes. I'll get one out later today.

[karelz]

@dbrownxc any update on the repro? Our chances to fix it in 2.1 are slowly going down. Thanks!

[dbrownxc]

HttpClientErrorRepro.zip

Sorry! It got busy around here.
To test on Windows I just run the test against an IIS hosted web site in visual studio
To test on Docker, I run

docker build .

I make it always fail (hack) so I see the returned output even if the call succeeds.

[karelz]

@rmkerr can you please take a look?

[rmkerr]

Yep! Thanks for getting around to sending out the repro @dbrownxc -- it makes things way more manageable on our end.

[rmkerr]

Something definitely looks wrong here. With NTLM credentials set on the client side, and both NTLM and Negotiate enabled on the server side, we don't even attempt to authenticate.

GET / HTTP/1.1
Connection: Keep-Alive
Host: localhost

HTTP/1.1 401 Unauthorized
Cache-Control: private
Content-Type: text/html; charset=utf-8
Server: Microsoft-IIS/10.0
WWW-Authenticate: Basic realm="localhost"
WWW-Authenticate: Negotiate
WWW-Authenticate: NTLM

After the exchange above, the client closes the connection to the server without even attempting to authenticate. If I leave NTLM enabled but disable Negotiate, it works. If I change the credential type to Negotiate (as provided in the repro), it works.

I don't have time to investigate more today, but this looks pretty serious so I'll be back on it tomorrow.

[rmkerr]

I was concerned that this issue might lie lower down the stack, or that it was an issue of machine policy blocking NTLM. I've gone ahead and tested it directly with the WinHttp authentication API and I don't see the same result, so that isn't the case. Since the issue does not occur on Linux I expect that the problem is somewhere in our WinHttpHandler authentication code.

Besides that I haven't made a lot of progress here -- I'm still mostly collecting information to help me track down the issue.

[seriouz]

@rmkerr Do i understand you correctly: There is no issue with dotnet core (wether on linux nor on windows). The problem lies within iis based services like OWA which use WinHttpHandler?