Fix Negotiate/SPNEGO Kerberos to NTLM fallback on Linux by davidsh · Pull Request #35383 · dotnet/corefx

davidsh · 2019-02-17T18:30:52Z

This PR adds support to our Linux gssapi layer so that we properly handle the Negotiate
protocol (SPNEGO) being able to to fall back from Kerberos to NTLM.

Windows Negotiate SSPI implements the SPNEGO protocol. However, it has built-in error
handling for dealing with quick fallback from Kerberos to NTLM in error cases such as where
the client is not on a domain/Kerberos realm, the client is using a local credential, the
client is talking to a non-domain server, etc.

However, the Linux implementation of SPNEGO doesn't handle those error cases very well.
So, I added the proper retry logic when generating the initial context token. Now, we will
be able to fall back from SPNEGO-wrapped Kerberos to SPNEGO-wrapped NTLM protocol blobs.

A similar bug exists in libcurl which is why Curl and CurlHandler are unable to connect to
Negotiate servers when not part of the Kerberos realm. I am considering proposing my fixes
to the libcurl repo as well.

I added more tests to support this PR and did manual testing. I added some additional
EventSource tracing as well.

Contributes to #34878

This PR adds support to our Linux gssapi layer so that we properly handle the Negotiate protocol (SPNEGO) being able to to fall back from Kerberos to NTLM. Windows Negotiate SSPI implements the SPNEGO protocol. However, it has built-in error handling for dealing with quick fallback from Kerberos to NTLM in error cases such as where the client is not on a domain/Kerberos realm, the client is using a local credential, the client is talking to a non-domain server, etc. However, the Linux implementation of SPNEGO doesn't handle those error cases very well. So, I added the proper retry logic when generating the initial context token. Now, we will be able to fall back from SPNEGO-wrapped Kerberos to SPNEGO-wrapped NTLM protocol blobs. A similar bug exists in libcurl which is why Curl and CurlHandler are unable to connect to Negotiate servers when not part of the Kerberos realm. I am considering proposing my fixes to the libcurl repo as well. I added more tests to support this PR and did manual testing. I added some additional EventSource tracing as well. Contributes to #34878

davidsh · 2019-02-17T19:48:06Z

/azp run corefx-outerloop-linux
/azp run corefx-outerloop-osx
/azp run corefx-outerloop-windows

azure-pipelines · 2019-02-17T19:48:11Z

No pipelines are associated with this pull request.

src/Native/Unix/System.Net.Security.Native/pal_gssapi.c

src/System.Net.Http/tests/FunctionalTests/HttpClientHandlerTest.Authentication.cs

src/Common/src/System/Net/Security/Unix/SafeDeleteNegoContext.cs

stephentoub

LGTM

davidsh · 2019-02-19T04:12:25Z

/azp run corefx-outerloop-linux
/azp run corefx-outerloop-osx
/azp run corefx-outerloop-windows

azure-pipelines · 2019-02-19T04:12:31Z

No pipelines are associated with this pull request.

vcsjones · 2019-02-19T04:35:59Z

@davidsh this is going to sound like a weird suggestion but try doing one pipeline trigger per comment.

davidsh · 2019-02-19T04:37:33Z

/azp run corefx-outerloop-linux

azure-pipelines · 2019-02-19T04:37:44Z

Azure Pipelines successfully started running 1 pipeline(s).

davidsh · 2019-02-19T04:38:22Z

/azp run corefx-outerloop-windows

davidsh · 2019-02-19T04:38:34Z

/azp run corefx-outerloop-osx

azure-pipelines · 2019-02-19T04:38:34Z

Azure Pipelines successfully started running 1 pipeline(s).

azure-pipelines · 2019-02-19T04:38:46Z

Azure Pipelines successfully started running 1 pipeline(s).

vcsjones · 2019-02-19T04:40:38Z

Ugh. That's frustrating. I learned from this PR that the pipelines trigger considers everything after azp to be part of the pipeline you are trying to trigger including new lines.

davidsh · 2019-02-19T04:42:02Z

@vcsjones Thx for the suggestion! I'm going to be talking to the Infrastructure team about this to make it better.

safern · 2019-02-19T18:19:53Z

@vcsjones please don't forget to ping me if you have any feedback to bring it over to azure devops folks to make this experience better

While debugging some Linux enterprise auth (Negotiate/NTLM) scenarios, I discoverd that the logging I added in PR dotnet#35383 wasn't working. This was due to not passing in a 'null' value for the context object for NetEventSource.Info. It was using the string object as the context object and not logging the string data. As part of fixing that I optimized some code for remembering what auth package (NTLM or SPNEGO) is used.

While debugging some Linux enterprise auth (Negotiate/NTLM) scenarios, I discovered that the logging I added in PR #35383 wasn't working. This was due to not passing in a 'null' value for the context object for NetEventSource.Info. It was using the string object as the context object and not logging the string data. As part of fixing that I optimized some code for remembering what auth package (NTLM or SPNEGO) is used.

While debugging some Linux enterprise auth (Negotiate/NTLM) scenarios, I discovered that the logging I added in PR dotnet#35383 wasn't working. This was due to not passing in a 'null' value for the context object for NetEventSource.Info. It was using the string object as the context object and not logging the string data. As part of fixing that I optimized some code for remembering what auth package (NTLM or SPNEGO) is used.

hacst · 2019-03-21T16:35:42Z

Are there plans to backport this into the 2.2 (or 2.1) series?

davidsh · 2019-03-21T16:43:15Z

Are there plans to backport this into the 2.2 (or 2.1) series?

This is a fix among several that I've already made addressing Enterprise Authentication scenarios. We are considering backporting them as a group to 2.1/2.2. But the decision hasn't been made yet. We are waiting to finish a few more fixes in master branch (.NET Core 3.0) before considering this.

@karelz

karelz · 2019-03-21T16:57:36Z

For porting to 2.1/2.2 we would need a very good and strong business case - number of customers who are blocked without the ports and who cannot upgrade to 3.0 from good reasons.

This PR addresses some issues reported by a customer (not thru GitHub). They noticed that multiple domain scenarios were working in .NET Core 2.0 but broke in .NET Core 2.1 on Linux While the previous PR dotnet#35383 solved the Negotiate Kerberos to NTLM fallback issue, it added more complexity than necessary. The retry logic I added wasn't really necessary because the original code that used GSS_KRB5_NT_PRINCIPAL_NAME format for the target name was wrong. That logic only works for pure Kerberos environments and doesn't handle domain or realm referrals. So, it only handles the default Kerberos realm on the the single Linux client. In addition, using GSS_KRB5_NT_PRINCIPAL_NAME defeats the logic of the SPNEGO mechanism. That is why I originally needed to add the retry logic using GSS_C_NT_HOSTBASED_SERVICE format for the target name. The multiple domain/realm scenario worked in .NET Core 2.0 because it used CurlHandler. And libcurl always uses GSS_C_NT_HOSTBASED_SERVICE format for target name. This PR reworks the logic to use the GSS_C_NT_HOSTBASED_SERVICE format. It also removes the now unneeded retry logic. I tested this against Windows and Linux as well as pure Kerberos and Kerberos to NTLM fallback (using SPNEGO). I added more tests. These tests are currently part of the enterprise scenario tests we are building. They are activated via environment variables. By definining all of the environment variables below, I am able to run the System.Net.Security and System.Net.Http enterprise-scenario tests. Both SocketsHttpHandler in System.Net.Http and NegotiateStream in System.Net.Security use the same common GSS-API logic in CoreFx. Define domain-joined server remote endpoint: * COREFX_NET_SECURITY_NEGOSERVERURI * COREFX_DOMAINJOINED_HTTPHOST * COREFX_NET_AD_DOMAINNAME * COREFX_NET_AD_PASSWORD * COREFX_NET_AD_USERNAME Define standalone server remote endpoint: * COREFX_NET_SERVER_PASSWORD * COREFX_NET_SERVER_USERNAME * COREFX_WINDOWSSERVER_HTTPHOST

…6329) This PR addresses some issues reported by a customer (not thru GitHub). They noticed that multiple domain scenarios were working in .NET Core 2.0 but broke in .NET Core 2.1 on Linux. While the previous PR #35383 solved the Negotiate Kerberos to NTLM fallback issue, it added more complexity than necessary. The retry logic I added wasn't really necessary because the original code that used GSS_KRB5_NT_PRINCIPAL_NAME format for the target name was wrong. That logic only works for pure Kerberos environments and doesn't handle domain or realm referrals. So, it only handles the default Kerberos realm on the the single Linux client. In addition, using GSS_KRB5_NT_PRINCIPAL_NAME defeats the logic of the SPNEGO mechanism. That is why I originally needed to add the retry logic using GSS_C_NT_HOSTBASED_SERVICE format for the target name. The multiple domain/realm scenario worked in .NET Core 2.0 because it used CurlHandler. And libcurl always uses GSS_C_NT_HOSTBASED_SERVICE format for target name. This PR reworks the logic to use the GSS_C_NT_HOSTBASED_SERVICE format. It also removes the now unneeded retry logic. I tested this against Windows and Linux as well as pure Kerberos and Kerberos to NTLM fallback (using SPNEGO). I added more tests. These tests are currently part of the enterprise scenario tests we are building. They are activated via environment variables. By defining all of the environment variables below, I am able to run the System.Net.Security and System.Net.Http enterprise-scenario tests. Both SocketsHttpHandler in System.Net.Http and NegotiateStream in System.Net.Security use the same common GSS-API logic in CoreFx. Define domain-joined server remote endpoint: * COREFX_NET_SECURITY_NEGOSERVERURI * COREFX_DOMAINJOINED_HTTPHOST * COREFX_NET_AD_DOMAINNAME * COREFX_NET_AD_PASSWORD * COREFX_NET_AD_USERNAME Define standalone server remote endpoint: * COREFX_WINDOWSSERVER_HTTPHOST * COREFX_NET_SERVER_PASSWORD * COREFX_NET_SERVER_USERNAME

…x#35383) This PR adds support to our Linux gssapi layer so that we properly handle the Negotiate protocol (SPNEGO) being able to to fall back from Kerberos to NTLM. Windows Negotiate SSPI implements the SPNEGO protocol. However, it has built-in error handling for dealing with quick fallback from Kerberos to NTLM in error cases such as where the client is not on a domain/Kerberos realm, the client is using a local credential, the client is talking to a non-domain server, etc. However, the Linux implementation of SPNEGO doesn't handle those error cases very well. So, I added the proper retry logic when generating the initial context token. Now, we will be able to fall back from SPNEGO-wrapped Kerberos to SPNEGO-wrapped NTLM protocol blobs. A similar bug exists in libcurl which is why Curl and CurlHandler are unable to connect to Negotiate servers when not part of the Kerberos realm. I am considering proposing my fixes to the libcurl repo as well. I added more tests to support this PR and did manual testing. I added some additional EventSource tracing as well. Contributes to dotnet/corefx#34878 Commit migrated from dotnet/corefx@1fd3ba1

…5918) While debugging some Linux enterprise auth (Negotiate/NTLM) scenarios, I discovered that the logging I added in PR dotnet/corefx#35383 wasn't working. This was due to not passing in a 'null' value for the context object for NetEventSource.Info. It was using the string object as the context object and not logging the string data. As part of fixing that I optimized some code for remembering what auth package (NTLM or SPNEGO) is used. Commit migrated from dotnet/corefx@21db29a

…tnet/corefx#36329) This PR addresses some issues reported by a customer (not thru GitHub). They noticed that multiple domain scenarios were working in .NET Core 2.0 but broke in .NET Core 2.1 on Linux. While the previous PR dotnet/corefx#35383 solved the Negotiate Kerberos to NTLM fallback issue, it added more complexity than necessary. The retry logic I added wasn't really necessary because the original code that used GSS_KRB5_NT_PRINCIPAL_NAME format for the target name was wrong. That logic only works for pure Kerberos environments and doesn't handle domain or realm referrals. So, it only handles the default Kerberos realm on the the single Linux client. In addition, using GSS_KRB5_NT_PRINCIPAL_NAME defeats the logic of the SPNEGO mechanism. That is why I originally needed to add the retry logic using GSS_C_NT_HOSTBASED_SERVICE format for the target name. The multiple domain/realm scenario worked in .NET Core 2.0 because it used CurlHandler. And libcurl always uses GSS_C_NT_HOSTBASED_SERVICE format for target name. This PR reworks the logic to use the GSS_C_NT_HOSTBASED_SERVICE format. It also removes the now unneeded retry logic. I tested this against Windows and Linux as well as pure Kerberos and Kerberos to NTLM fallback (using SPNEGO). I added more tests. These tests are currently part of the enterprise scenario tests we are building. They are activated via environment variables. By defining all of the environment variables below, I am able to run the System.Net.Security and System.Net.Http enterprise-scenario tests. Both SocketsHttpHandler in System.Net.Http and NegotiateStream in System.Net.Security use the same common GSS-API logic in CoreFx. Define domain-joined server remote endpoint: * COREFX_NET_SECURITY_NEGOSERVERURI * COREFX_DOMAINJOINED_HTTPHOST * COREFX_NET_AD_DOMAINNAME * COREFX_NET_AD_PASSWORD * COREFX_NET_AD_USERNAME Define standalone server remote endpoint: * COREFX_WINDOWSSERVER_HTTPHOST * COREFX_NET_SERVER_PASSWORD * COREFX_NET_SERVER_USERNAME Commit migrated from dotnet/corefx@658ec73

davidsh added os-linux Linux OS (any supported distro) area-System.Net.Http.SocketsHttpHandler labels Feb 17, 2019

davidsh added this to the 3.0 milestone Feb 17, 2019

davidsh self-assigned this Feb 17, 2019

davidsh requested review from a team and stephentoub February 19, 2019 02:41

stephentoub reviewed Feb 19, 2019

View reviewed changes

src/Native/Unix/System.Net.Security.Native/pal_gssapi.c Outdated Show resolved Hide resolved

stephentoub reviewed Feb 19, 2019

View reviewed changes

src/System.Net.Http/tests/FunctionalTests/HttpClientHandlerTest.Authentication.cs Outdated Show resolved Hide resolved

stephentoub reviewed Feb 19, 2019

View reviewed changes

src/Common/src/System/Net/Security/Unix/SafeDeleteNegoContext.cs Outdated Show resolved Hide resolved

stephentoub approved these changes Feb 19, 2019

View reviewed changes

Address PR feedback

2dcf905

Fix assert

a0dbe4f

davidsh merged commit 1fd3ba1 into dotnet:master Feb 19, 2019

davidsh deleted the kerb_ntlm branch February 19, 2019 17:14

davidsh mentioned this pull request Mar 10, 2019

Fix NetEventSource logging for Windows auth on Linux #35918

Merged

davidsh mentioned this pull request Mar 24, 2019

Fix Kerberos/NTLM for multiple domain/realm environments in Linux #36329

Merged

MichalStrehovsky mentioned this pull request Apr 25, 2019

System.Data.SqlClient build fails on macOS dotnet/corert#7353

Closed

dnewsholme mentioned this pull request May 24, 2019

Linux Compatibility with Powershell Core dnewsholme/PasswordState-Management#20

Closed

jkotas mentioned this pull request Feb 1, 2020

Out-of-band System.Data.SqlClient depends on inbox native shims dotnet/runtime#29369

Closed

MihuBot mentioned this pull request Sep 3, 2025

[✨ Triage] dotnet/runtime#119298 by FixRM - Negotiate to NTLM authentication fallback not working MihuBot/runtime-utils#1431

Open

Conversation

davidsh commented Feb 17, 2019

Uh oh!

davidsh commented Feb 17, 2019

Uh oh!

azure-pipelines bot commented Feb 17, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

stephentoub left a comment

Choose a reason for hiding this comment

Uh oh!

davidsh commented Feb 19, 2019

Uh oh!

azure-pipelines bot commented Feb 19, 2019

Uh oh!

vcsjones commented Feb 19, 2019

Uh oh!

davidsh commented Feb 19, 2019

Uh oh!

azure-pipelines bot commented Feb 19, 2019

Uh oh!

davidsh commented Feb 19, 2019

Uh oh!

davidsh commented Feb 19, 2019

Uh oh!

azure-pipelines bot commented Feb 19, 2019

Uh oh!

azure-pipelines bot commented Feb 19, 2019

Uh oh!

vcsjones commented Feb 19, 2019

Uh oh!

davidsh commented Feb 19, 2019

Uh oh!

safern commented Feb 19, 2019

Uh oh!

hacst commented Mar 21, 2019

Uh oh!

davidsh commented Mar 21, 2019

Uh oh!

karelz commented Mar 21, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants