Remove Debug.Assert in SetSslCertificate that aborts process during dispose-during-handshake race

[Copilot]

During the race between AuthenticateAsServerAsync and Dispose(), certificate/key SafeHandles can become invalid before reaching SetSslCertificate. The Debug.Assert on handle validity fires and terminates the process with SIGABRT (exit code 134) instead of letting the p/invoke marshaller throw ObjectDisposedException as expected.

Changes

• Interop.OpenSsl.cs: Remove the two Debug.Assert validity checks in SetSslCertificate. The p/invoke marshaller already throws ObjectDisposedException when closed SafeHandles are passed across the native boundary, making these asserts both redundant and harmful.

// Removed:
Debug.Assert(certPtr != null && !certPtr.IsInvalid);
Debug.Assert(keyPtr != null && !keyPtr.IsInvalid);

Original prompt

---

This section details on the original issue you should resolve

<issue_title>System.Net.Security.Tests.SslStreamDisposeTest.Dispose_ParallelWithHandshake_ThrowsODE crashes with Debug.Assert in SetSslCertificate</issue_title>
<issue_description>## Description

System.Net.Security.Tests.SslStreamDisposeTest.Dispose_ParallelWithHandshake_ThrowsODE (OuterLoop) crashes the test process with a Debug.Assert failure (SIGABRT, exit code 134) on Azure Linux 3.0 x64.

Stack Trace

Process terminated.
Assertion failed.
certPtr != null && !certPtr.IsInvalid
   at Interop.OpenSsl.SetSslCertificate(SafeSslContextHandle contextPtr, SafeX509Handle certPtr, SafeEvpPKeyHandle keyPtr)
     in src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.OpenSsl.cs:line 1063
   at Interop.OpenSsl.AllocateSslContext(SslAuthenticationOptions sslAuthenticationOptions, SslProtocols protocols, Boolean enableResume)
     in src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.OpenSsl.cs:line 285
   at Interop.OpenSsl.<>c.<GetOrCreateSslContextHandle>b__14_0(ValueTuple`3 args)
     in src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.OpenSsl.cs:line 202
   at System.Net.SafeHandleCache`2.GetOrCreate[TContext](TKey key, Func`2 factory, TContext factoryContext)
   at Interop.OpenSsl.GetOrCreateSslContextHandle(SslAuthenticationOptions sslAuthenticationOptions, Boolean allowCached)
   at Interop.OpenSsl.AllocateSslHandle(SslAuthenticationOptions sslAuthenticationOptions)
   at System.Net.Security.SslStreamPal.HandshakeInternal(SafeDeleteSslContext& context, ...)
   at System.Net.Security.SslStreamPal.AcceptSecurityContext(...)
   at System.Net.Security.SslStream.GenerateToken(...)
   at System.Net.Security.SslStream.NextMessage(...)
   at System.Net.Security.SslStream.ProcessTlsFrame(Int32 frameSize)
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](...)
   ...
   at System.Net.Security.Tests.SslStreamDisposeTest.<>c__DisplayClass3_1.<Dispose_ParallelWithHandshake_ThrowsODE>b__3()
     in src/libraries/System.Net.Security/tests/FunctionalTests/SslStreamDisposeTest.cs:line 136

Analysis

The test runs 10,000 parallel iterations that start a TLS handshake and immediately dispose the SslStream. During the race between the server-side AuthenticateAsServerAsync handshake and Dispose(), the CertificateContext.CertificateHandle becomes invalid by the time AllocateSslContext passes it to SetSslCertificate. The Debug.Assert on the certificate handle validity fires and aborts the process instead of throwing an ObjectDisposedException.

The Debug.Assert at Interop.OpenSsl.cs:1063 should probably be converted to a proper check that throws an appropriate exception (e.g., ObjectDisposedException) to handle the dispose-during-handshake race gracefully.

Environment

• OS: Azure Linux 3.0 (amd64), Docker image mcr.microsoft.com/dotnet-buildtools/prereqs:azurelinux-3.0-helix-amd64
• Test category: OuterLoop
• Helix job: 56ac9818-ceaa-4127-8348-605724497d33

Log

Full Helix console log

Observed in PR #124591 (unrelated change — HTTP3 nginx interop test re-enable).
</issue_description>

<agent_instructions>Remove the offending assert</agent_instructions>

Comments on the Issue (you are @copilot in this section)

@vcsjones > The Debug.Assert at Interop.OpenSsl.cs:1063 should probably be converted to a proper check that throws an appropriate exception (e.g., ObjectDisposedException) to handle the dispose-during-handshake race gracefully.

You can probably just get rid of the assert. The marshaller at the p/invoke boundary will throw ODE if the SafeHandles are closed.</comment_new>

• Fixes System.Net.Security.Tests.SslStreamDisposeTest.Dispose_ParallelWithHandshake_ThrowsODE crashes with Debug.Assert in SetSslCertificate #124602

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/ncl, @bartonjs, @vcsjones
See info in area-owners.md if you want to be subscribed.

[Copilot]

Pull request overview

This PR fixes a test crash caused by Debug.Assert statements that abort the process during a race condition between Dispose() and AuthenticateAsServerAsync. The assertions were checking SafeHandle validity before passing certificate and key handles to OpenSSL p/invoke methods. When the handles become invalid due to disposal during the handshake, the asserts fire and terminate the process with SIGABRT instead of allowing the p/invoke marshaller to throw ObjectDisposedException as expected.

Changes:

• Remove two Debug.Assert statements in SetSslCertificate that check certificate and key handle validity
• Allow the .NET p/invoke marshaller to naturally throw ObjectDisposedException when closed SafeHandles are passed across the native boundary

[wfurt]

What will happen if we pass NULL to the native call? Should we check and simply return on such condition?