NTLM interop problems with MacOS

[davidsh]

.NET Core NTLM support is broken on MacOS. It has probably been broken for a while. But due to lack of test coverage we didn't notice. I did my testing on MacOS X 10.15.2 (Catalina).

The root cause is complex and due to possible interop issues with the NTLM support in the Heimdal library that ships with the Mac. Prior to MacOS 10.7, the Mac used MIT Kerberos open-source libraries. In 10.7, it switched to Heimdal open-source libraries. You can see the source code of the various open-source libraries used in the Mac here.

We have had some reports by customers of problems with Mac .NET Core clients using NTLM. At first, we thought that some NTLM support package was missing on the Mac. On Linux, we require the installation of the 'gss-ntlmssp' package which by default is not present on Linux distros nor on our .NET Core docker images. But the Mac already has NTLM support installed in the GSS-API mechanisms.

We have some simple NTLM tests in our repo:

runtime/src/libraries/System.Net.Http/tests/FunctionalTests/HttpClientHandlerTest.Authentication.cs

Lines 588 to 593 in 57df202

	[ConditionalTheory(nameof(IsNtlmInstalled))]
	[InlineData("NTLM")]
	[InlineData("Negotiate")]
	public async Task Credentials_ServerChallengesWithWindowsAuth_ClientSendsWindowsAuthHeader(string authScheme)
	{
	await LoopbackServerFactory.CreateClientAndServerAsync(

And these were running and passing on the Mac. However, the tests only validate that an initial NTLM base64 token is generated and sent to the server. Those tests don't do end-to-end validation of a username/password.

We have other tests for HttpClient that do end-to-end validation:

runtime/src/libraries/System.Net.Http/tests/FunctionalTests/HttpClientHandlerTest.Authentication.cs

Lines 568 to 570 in 57df202

	[ConditionalTheory(nameof(IsNtlmInstalled), nameof(IsWindowsServerAvailable))]
	[MemberData(nameof(ServerUsesWindowsAuthentication_MemberData))]
	public async Task Credentials_ServerUsesWindowsAuthentication_Success(string server)

But those are not turned on right now. They require environment variables to be enabled to point to a remote NTLM server. After turning on those tests on my local Mac, I discovered that NTLM is broken for Mac .NET Core.

More NTLM (and Kerberos) tests will be enabled for CI as part of follow-up PRs to #463

If I use Safari on the Mac, it works fine against the NTLM test server. But the Safari app on the Mac uses its own NTLM library and not the GSS-API Heimdal library that .NET Core uses.

Looking at several WireShark traces of working and non-working scenarios shows differences in various NTLM protocol fields. Although the Mac is sending NTLM packets to the server, the end result is that the server (Windows in this case) denies the authentication. We'll need to work with the Windows protocol team to help enable advanced diagnostics to see why the Windows server denied the authentication.

I will continue looking into the problem by trying out different NTLM servers to test against (such as Linux Apache) to get more information on working versus non-working scenarios. My current hypothesis is that there is a problem with Heimdal's NTLM v2 support. The NTLM protocol has many differences between NTLMv1 versus NTLMv2.

[scalablecory]

triage: please upvote this if you're seeing this issue.

[scholtz]

Hi @davidsh , I am trying to do windows authentication in linux according to this: https://docs.microsoft.com/cs-cz/aspnet/core/security/authentication/windowsauth?view=aspnetcore-3.1&tabs=visual-studio#kestrel

What I have observed, .net core 3.1 connects to System.Net.NTAuthentication which calls System.Net.Security.NegotiateStreamPal which calls unix Interop.NetSecurityNative.InitSecContext which invokes somehow the gss method in linux

without gss-ntlmssp library it throws error "An unsupported mechanism was requested." because there is no mechanism defined in /etc/gss/mech.d/ folder.

when i install gss-ntlmssp library, it throws error: "No credentials were supplied, or the credentials were unavailable or inaccessible"

i have tried to compile my own gss-ntlmssp, and found out that it does not even trigger any gss_ methods.
when i comment the line in mech.conf (gssntlmssp_v1 1.3.6.1.4.1.311.2.2.10 @libdir@/gssntlmssp/gssntlmssp.so) it shows unsupported mech, when i uncomment it show no credentials supplied without triggering the library.

i tested this on centos7 vm, ubuntu windows subsytem (18.04) and debian docker image (customized mcr.microsoft.com/dotnet/core/sdk:3.1-buster) - all behave the same

please see my full config in here: https://stackoverflow.com/questions/60296237/how-can-i-enable-debug-log-with-microsoft-aspnetcore-authentication-negotiate-pa

also note, that klist shows my spn, and i am able to connect to mssql using integrated security

what am i missing?? does this work for anybody?

[davidsh]

@scholtz The problem you are describing is not related to this particular issue. This issue is only about NTLM problems (client-side) on MacOS. So, it would be best for you to open a new issue to discuss your authentication issues on Linux.

That said, the problem you are describing is a server-side problem with asp.net core. Kestrel server does NOT support NTLM protocol. There is no way to tell it to send back 'Www-Authenticate: NTLM'. It only supports 'Www-Authenticate: Negotiate' protocol. 'Negotiate' protocol is usually Kerberos but can fall back to using NTLM packets (over SPNEGO protocol) in certain cases. NTLM fallback with Negotiate is something that isn't really supported either in Kestrel although I was able to make it work better in .NET 5. See: dotnet/corefx#42522

So, are you really trying to get NTLM working? You mention 'klist shows my spn and mssql works with integrated security'. So, that probably means you want to use Kerberos anyways. The 'gss-ntlmssp' component is not going to help you here since it is only for NTLM and not Kerberos.

You might want to open an issue in aspnetcore repo to get help on getting Kestrel working with Kerberos.

I also recommend you set the following environment variable on Linux to get GSS-API traces, i.e

export KRB5_TRACE=/dev/stdout

@Tratcher