SocketsHttpHandler: NTLM auth does not work by default on Unix

[wfurt]

HttpClient with credentials explicitly set. SocketHttpHandler throws on NTLM authentication requested by server:

System.ComponentModel.Win32Exception (0x80090020): GSSAPI operation failed with error - An invalid status code was supplied (Unknown error).
  at System.Net.Security.NegotiateStreamPal.AcquireCredentialsHandle(String package, Boolean isServer, NetworkCredential credential) in /home/build/github/corefx/src/Common/src/System/Net/Security/NegotiateStreamPal.Unix.cs:line 316
  at System.Net.NTAuthentication.Initialize(Boolean isServer, String package, NetworkCredential credential, String spn, ContextFlagsPal requestedContextFlags, ChannelBinding channelBinding) in /home/build/github/corefx/src/Common/src/System/Net/NTAuthentication.Common.cs:line 128
  at System.Net.NTAuthentication..ctor(Boolean isServer, String package, NetworkCredential credential, String spn, ContextFlagsPal requestedContextFlags, ChannelBinding channelBinding) in /home/build/github/corefx/src/Common/src/System/Net/NTAuthentication.Common.cs:line 98
  at System.Net.Http.AuthenticationHelper.SendWithNtAuthAsync(HttpRequestMessage request, Uri authUri, ICredentials credentials, Boolean isProxyAuth, HttpConnection connection, CancellationToken cancellationToken) in /home/build/github/corefx/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/AuthenticationHelper.NtAuth.cs:line 61
  at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) in /home/build/github/corefx/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 283
  at System.Net.Http.AuthenticationHelper.SendWithAuthAsync(HttpRequestMessage request, Uri authUri, ICredentials credentials, Boolean preAuthenticate, Boolean isProxyAuth, Boolean doRequestAuth, HttpConnectionPool pool, CancellationToken cancellationToken) in /home/build/github/corefx/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/AuthenticationHelper.cs:line 204
  at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) in /home/build/github/corefx/src/Sys

Workarounds:

1. Install package called gss-ntlmssp (plugin into MIT Kerberos GSSAPI layer)
2. Install Heimdal (alternative Kerberos implementation with NTLM support): https://www.h5l.org/manual/heimdal-1-5-branch/ntlm/

One of the options installed is sufficient to make SocketHttpHandler work for NTLM without any additional changes to CoreFX or the app.

Details:
.NET Core 2.0 / 2.1 with libcurl works fine. This is because curl has it's own implementation but SocketHttpHandler depends on authentication code shared with NegotiateStream class.

The shared authentication depends on GSSAPI and that is typically provided by MIT kerberos. (this is also true for 2.1.300 preview docker images)
It does not have ability to do NTLM.

This is follow up on #25370 and #25368. This may be also related to dotnet/wcf#943

[stephentoub]

This is because curl has it's own implementation

libcurl doesn't just depend on one of those packages as well? When I do:

ldd /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4.4.0

I see this:

        linux-vdso.so.1 =>  (0x00007fffdb127000)
        libidn.so.11 => /usr/lib/x86_64-linux-gnu/libidn.so.11 (0x00007f6de7fa0000)
        librtmp.so.1 => /usr/lib/x86_64-linux-gnu/librtmp.so.1 (0x00007f6de7d80000)
        libnettle.so.6 => /usr/lib/x86_64-linux-gnu/libnettle.so.6 (0x00007f6de7b40000)
        libgnutls.so.30 => /usr/lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007f6de7800000)
        libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f6de75b0000)
        liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007f6de73a0000)
        libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007f6de7130000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f6de6f10000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f6de6cf0000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f6de6910000)
        libhogweed.so.4 => /usr/lib/x86_64-linux-gnu/libhogweed.so.4 (0x00007f6de66d0000)
        libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f6de6450000)
        libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007f6de61d0000)
        libtasn1.so.6 => /usr/lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007f6de5fb0000)
        libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f6de5cd0000)
        libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f6de5a90000)
        libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f6de5880000)
        libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f6de5670000)
        libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f6de5440000)
        libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f6de5220000)
        libgssapi.so.3 => /usr/lib/x86_64-linux-gnu/libgssapi.so.3 (0x00007f6de4fd0000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f6de8600000)
        libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007f6de4db0000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f6de4ba0000)
        libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f6de4990000)
        libheimntlm.so.0 => /usr/lib/x86_64-linux-gnu/libheimntlm.so.0 (0x00007f6de4770000)
        libkrb5.so.26 => /usr/lib/x86_64-linux-gnu/libkrb5.so.26 (0x00007f6de44e0000)
        libasn1.so.8 => /usr/lib/x86_64-linux-gnu/libasn1.so.8 (0x00007f6de4220000)
        libhcrypto.so.4 => /usr/lib/x86_64-linux-gnu/libhcrypto.so.4 (0x00007f6de3fe0000)
        libroken.so.18 => /usr/lib/x86_64-linux-gnu/libroken.so.18 (0x00007f6de3dc0000)
        libwind.so.0 => /usr/lib/x86_64-linux-gnu/libwind.so.0 (0x00007f6de3b80000)
        libheimbase.so.1 => /usr/lib/x86_64-linux-gnu/libheimbase.so.1 (0x00007f6de3970000)
        libhx509.so.5 => /usr/lib/x86_64-linux-gnu/libhx509.so.5 (0x00007f6de3720000)
        libsqlite3.so.0 => /usr/lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007f6de3430000)
        libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f6de31f0000)

which includes:

        libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f6de75b0000)
        libgssapi.so.3 => /usr/lib/x86_64-linux-gnu/libgssapi.so.3 (0x00007f6de4fd0000)
        libkrb5.so.26 => /usr/lib/x86_64-linux-gnu/libkrb5.so.26 (0x00007f6de44e0000)
        libheimntlm.so.0 => /usr/lib/x86_64-linux-gnu/libheimntlm.so.0 (0x00007f6de4770000)

[Petermarcu]

@danmosemsft @karelz . This is a place where we should make sure the error about missing dependencies is better so people know what to do when this happens.

[karelz]

Yes, it would be nice to have specific exception with reason "NTLM is not supported by the underlying OS" with permalink to article with potential workarounds.
Assuming we can detect from the GSSAPI error code that the root-cause is missing support for NTLM and nothing else.

[richlander]

I agree with @Petermarcu ... Your proposal sounds good @karelz.

[greggbjensen]

I am using microsoft/dotnet:2.1-aspnetcore-runtime image, and I did install gss-ntlmssp. It makes the GSSAPI operation failed error go away.

However, then I get System.PlatformNotSupportedException: No support for channel binding on operating systems other than Windows.

Is there another package I'm missing?

[ncsurfus]

@greggbjensen I've ran into the same issue, I traced that down to this.

https://github.com/dotnet/corefx/blob/master/src/Common/src/System/Net/Security/NegotiateStreamPal.Unix.cs
https://github.com/dotnet/corefx/issues/3718
// TODO (Issue dotnet/runtime#15393): The second buffer can contain a channel binding which is not supported****

[AlexanderLeonov]

@ncsurfus @greggbjensen I'm having the same issue. In my case though it is strictly tied to connection over https. It works fine if I just change url to http.

Can someone elaborate a bit on this message? What are those "channel bindings" that are not supported on linux? Is there any fix/workaround for this issue? Using http scheme is not an option in our case cause it is a connection over the internet transferring potentially sensitive information.

[tigertop]

@AlexanderLeonov we had issues with both NTLM and then with HTTPS connections in .NET Core 2.1
For the NTLM issue we just felled back to using LibCurl implementation of HttpClient using the AppContext.SetSwitch("System.Net.Http.UseSocketsHttpHandler", false);. Then the NTLM worked fine but we had issues with HTTPS requests to reCaptcha service. THe app was crashing wt Segmentation Fault. We used self-contained mode for the app. The problem was that in our environment (Ubuntu 14.04) the System.Net.Http.Native.dll was linked against libcurl.1.0.0 and our application during run-time was loading the libcurl.1.0.2.
Our workaround was to copy libcurl.1.0.0 in our app folder and setting an environmental variable that was adding that folder to the library lookup path.

[ncsurfus]

I'd also like to note that switching back to the libcurl implementation AppContext.SetSwitch("System.Net.Http.UseSocketsHttpHandler", false); is what I did as a work-around to get NTLM working properly on Linux.... hopefully this can be fixed/implemented.

[AlexanderLeonov]

In our case we were hoping to solve memory leak problem by switching to the new HttpClient implementation in 2.1. And it actually worked... except for this ntlm/https problem - which unfortunately is a deal breaker for us. Thanks for your suggestions guys, I know I could do those things, but the whole point was to make use of new HttpClient implementation. We'll probably have to go with something else for now while waiting for a fix to this problem.

EDIT: So, long story short - as of now new HttpClient implementation does not work with ntlm/negotiate authentication (and I assume any other authentication method as well) over https. Can anyone of corefx developers confirm it? Is there a plan to do something about it?

[ncsurfus]

I think the problem is authenticating with servers that support NTLM Extended Protection (Channel Binding). It binds the NTLM auth to a specific TLS session to prevent authenticating through a Proxy/MITM. I think there is a flag that gets set from the server’s NTLM Challenge Message that specifies it supports channel binding... however in the case of IIS, you can support extended protection, but not require it. SocketsHttpHandler doesn’t try to continue the authentication (as if it were unaware of channel binding).

Having a consistent experience between Linux and Windows would be a huge bonus!

[greggbjensen]

@ncsurfus the AppContext.SetSwitch workaround you suggested worked perfectly and saved our project a lot of time. Thanks.

[tmds]

I reported https://github.com/dotnet/corefx/issues/31005. This is also due to missing gss-ntlmssp.

In this case, a proxy server provided these auth mechanisms:

< Proxy-Authenticate: Negotiate
< Proxy-Authenticate: NTLM

Behavior was different from what is described in this issue:

SocketHttpHandler throws on NTLM authentication requested by server:

--> no exception was thrown, HttpClient returned 407 Proxy Authentication Required

.NET Core 2.0 / 2.1 with libcurl works fine. This is because curl has it's own implementation but SocketHttpHandler depends on authentication code shared with NegotiateStream class.

--> AppContext.SetSwitch (i.e. curlhandler) does not work. Running curl on the command line prints gss_init_sec_context() failed: SPNEGO cannot find mechanisms to negotiate..

It would be nice if there was an exception thrown with a clear message.
There is no documentation gss-ntlmssp is required to do Windows auth on Linux.

[davidsh]

I think the problem is authenticating with servers that support NTLM Extended Protection (Channel Binding). It binds the NTLM auth to a specific TLS session to prevent authenticating through a Proxy/MITM. I think there is a flag that gets set from the server’s NTLM Challenge Message that specifies it supports channel binding... however in the case of IIS, you can support extended protection, but not require it. SocketsHttpHandler doesn’t try to continue the authentication (as if it were unaware of channel binding).
Having a consistent experience between Linux and Windows would be a huge bonus!

We seem to be getting this error all the time for any https site using NTLM even if Extended Protection (i.e. "channel binding") is OFF.

Unhandled Exception: System.PlatformNotSupportedException: No support for channel binding on operating systems other than Windows.
   at System.Net.Security.NegotiateStreamPal.InitializeSecurityContext(SafeFreeCredentials credentialsHandle, SafeDeleteContext& securityContext, String spn, ContextFlagsPal requestedContextFlags, SecurityBuffer[] inSecurityBufferArray, SecurityBuffer outSecurityBuffer, ContextFlagsPal& contextFlags)
   at System.Net.NTAuthentication.GetOutgoingBlob(Byte[] incomingBlob, Boolean throwOnError, SecurityStatusPal& statusCode)
   at System.Net.NTAuthentication.GetOutgoingBlob(String incomingBlob)
   at System.Net.Http.AuthenticationHelper.SendWithNtAuthAsync(HttpRequestMessage request, Uri authUri, ICredentials credentials, Boolean isProxyAuth, HttpConnection connection, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.AuthenticationHelper.SendWithAuthAsync(HttpRequestMessage request, Uri authUri, ICredentials credentials, Boolean preAuthenticate, Boolean isProxyAuth, Boolean doRequestAuth, HttpConnectionPool pool, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at httptest.Program.Main(String[] args) in /home/root/httptest/Program.cs:line 18

So, at a minimum, this should get fixed to at least support NTLM connections where channel binding is Off or Accept'd.

[juliobbv]

+1 on @davidsh's comment.

We're working on upgrading our Azure Pipelines agent project from .NET Core 2.0 to 2.1, and unfortunately we had to disable the new SocketHttpHandler to work around the "No support for channel binding..." issue in Linux, as we support connecting our agent to on-prem TFS instances.