Mac Catalina compatibility: System.Security.Cryptography.X509Certificates.Tests crashing

[buyaa-n]

System.Security.Cryptography.X509Certificates.Tests are crashing without producing result

Starting:    System.Security.Cryptography.X509Certificates.Tests (parallel test collections = on, max threads = 4)
  Warning: unable to build chain to self-signed root for signer "(null)"Process terminated. Assertion failed.
  X509ChainGetStatusAtIndex returned unexpected error 2
     at Internal.Cryptography.Pal.SecTrustChainPal.ParseResults(SafeX509ChainHandle chainHandle, X509RevocationMode revocationMode) in /Users/dotnet/new_source/corefx/src/System.Security.Cryptography.X509Certificates/src/Internal/Cryptography/Pal.OSX/ChainPal.cs:line 250
     at Internal.Cryptography.Pal.SecTrustChainPal.Execute(DateTime verificationTime, Boolean allowNetwork, OidCollection applicationPolicy, OidCollection certificatePolicy, X509RevocationFlag revocationFlag) in /Users/dotnet/new_source/corefx/src/System.Security.Cryptography.X509Certificates/src/Internal/Cryptography/Pal.OSX/ChainPal.cs:line 210
     at Internal.Cryptography.Pal.ChainPal.BuildChain(Boolean useMachineContext, ICertificatePal cert, X509Certificate2Collection extraStore, OidCollection applicationPolicy, OidCollection certificatePolicy, X509RevocationMode revocationMode, X509RevocationFlag revocationFlag, DateTime verificationTime, TimeSpan timeout) in /Users/dotnet/new_source/corefx/src/System.Security.Cryptography.X509Certificates/src/Internal/Cryptography/Pal.OSX/ChainPal.cs:line 574
     at System.Security.Cryptography.X509Certificates.X509Chain.Build(X509Certificate2 certificate, Boolean throwOnException) in /Users/dotnet/new_source/corefx/src/System.Security.Cryptography.X509Certificates/src/System/Security/Cryptography/X509Certificates/X509Chain.cs:line 118
     at System.Security.Cryptography.X509Certificates.X509Chain.Build(X509Certificate2 certificate) in /Users/dotnet/new_source/corefx/src/System.Security.Cryptography.X509Certificates/src/System/Security/Cryptography/X509Certificates/X509Chain.cs:line 105
     at System.Security.Cryptography.X509Certificates.Tests.DynamicChainTests.TestInvalidAia() in /Users/dotnet/new_source/corefx/src/System.Security.Cryptography.X509Certificates/tests/DynamicChainTests.cs:line 240
     at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
     at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture) in /_/src/System.Private.CoreLib/src/System/Reflection/RuntimeMethodInfo.cs:line 475
     at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters) in /_/src/System.Private.CoreLib/shared/System/Reflection/MethodBase.cs:line 54
     at Xunit.Sdk.TestInvoker`1.CallTestMethod(Object testClassInstance) in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestInvoker.cs:line 150
     at Xunit.Sdk.TestInvoker`1.<>c__DisplayClass48_1.<<InvokeTestMethodAsync>b__1>d.MoveNext() in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestInvoker.cs:line 257
     at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs:line 1015
     at Xunit.Sdk.TestInvoker`1.<>c__DisplayClass48_1.<InvokeTestMethodAsync>b__1()
     at Xunit.Sdk.ExecutionTimer.AggregateAsync(Func`1 asyncAction) in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\ExecutionTimer.cs:line 48
     at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs:line 1015
     at Xunit.Sdk.ExecutionTimer.AggregateAsync(Func`1 asyncAction)
     at Xunit.Sdk.TestInvoker`1.<>c__DisplayClass48_1.<InvokeTestMethodAsync>b__0() in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestInvoker.cs:line 242
     at Xunit.Sdk.ExceptionAggregator.RunAsync(Func`1 code) in C:\Dev\xunit\xunit\src\xunit.core\Sdk\ExceptionAggregator.cs:line 90
     at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs:line 1015
     at Xunit.Sdk.ExceptionAggregator.RunAsync(Func`1 code)
     at Xunit.Sdk.TestInvoker`1.InvokeTestMethodAsync(Object testClassInstance) in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestInvoker.cs:line 241
     at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs:line 1015
     at Xunit.Sdk.TestInvoker`1.InvokeTestMethodAsync(Object testClassInstance)
     at Xunit.Sdk.XunitTestInvoker.InvokeTestMethodAsync(Object testClassInstance) in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\XunitTestInvoker.cs:line 112
     at Xunit.Sdk.TestInvoker`1.<RunAsync>b__47_0() in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestInvoker.cs:line 206
     at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs:line 1015
     at Xunit.Sdk.TestInvoker`1.<RunAsync>b__47_0()
     at Xunit.Sdk.ExceptionAggregator.RunAsync[T](Func`1 code) in C:\Dev\xunit\xunit\src\xunit.core\Sdk\ExceptionAggregator.cs:line 107
     at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs:line 1015
     at Xunit.Sdk.ExceptionAggregator.RunAsync[T](Func`1 code)
     at Xunit.Sdk.TestInvoker`1.RunAsync() in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestInvoker.cs:line 189
     at Xunit.Sdk.XunitTestRunner.InvokeTestMethodAsync(ExceptionAggregator aggregator) in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\XunitTestRunner.cs:line 84
     at Xunit.Sdk.XunitTestRunner.InvokeTestAsync(ExceptionAggregator aggregator) in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\XunitTestRunner.cs:line 67
     at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs:line 1015
     at Xunit.Sdk.XunitTestRunner.InvokeTestAsync(ExceptionAggregator aggregator)
     at Xunit.Sdk.TestRunner`1.<>c__DisplayClass43_0.<RunAsync>b__0() in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestRunner.cs:line 149
     at Xunit.Sdk.ExceptionAggregator.RunAsync[T](Func`1 code) in C:\Dev\xunit\xunit\src\xunit.core\Sdk\ExceptionAggregator.cs:line 107
     at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs:line 1015
     at Xunit.Sdk.ExceptionAggregator.RunAsync[T](Func`1 code)
     at Xunit.Sdk.TestRunner`1.RunAsync() in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestRunner.cs:line 149
     at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs:line 1015
     at Xunit.Sdk.TestRunner`1.RunAsync()
     at Xunit.Sdk.XunitTestCaseRunner.RunTestAsync() in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\XunitTestCaseRunner.cs:line 139
     at Xunit.Sdk.TestCaseRunner`1.RunAsync() in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestCaseRunner.cs:line 82
     at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs:line 1015
     at Xunit.Sdk.TestCaseRunner`1.RunAsync()
     at Xunit.Sdk.XunitTestCase.RunAsync(IMessageSink diagnosticMessageSink, IMessageBus messageBus, Object[] constructorArguments, ExceptionAggregator aggregator, CancellationTokenSource cancellationTokenSource) in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\XunitTestCase.cs:line 162
     at Xunit.Sdk.XunitTestMethodRunner.RunTestCaseAsync(IXunitTestCase testCase) in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\XunitTestMethodRunner.cs:line 45
     at Xunit.Sdk.TestMethodRunner`1.RunTestCasesAsync() in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestMethodRunner.cs:line 136
     at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs:line 1015
     at Xunit.Sdk.TestMethodRunner`1.RunTestCasesAsync()
     at Xunit.Sdk.TestMethodRunner`1.RunAsync() in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestMethodRunner.cs:line 106
     at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs:line 1015
     at Xunit.Sdk.TestMethodRunner`1.RunAsync()
     at Xunit.Sdk.XunitTestClassRunner.RunTestMethodAsync(ITestMethod testMethod, IReflectionMethodInfo method, IEnumerable`1 testCases, Object[] constructorArguments) in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\XunitTestClassRunner.cs:line 168
     at Xunit.Sdk.TestClassRunner`1.RunTestMethodsAsync() in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestClassRunner.cs:line 213
     at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs:line 1015
     at Xunit.Sdk.TestClassRunner`1.RunTestMethodsAsync()
     at Xunit.Sdk.TestClassRunner`1.RunAsync() in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestClassRunner.cs:line 171
     at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs:line 1015
     at Xunit.Sdk.TestClassRunner`1.RunAsync()
     at Xunit.Sdk.XunitTestCollectionRunner.RunTestClassAsync(ITestClass testClass, IReflectionTypeInfo class, IEnumerable`1 testCases) in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\XunitTestCollectionRunner.cs:line 158
     at Xunit.Sdk.TestCollectionRunner`1.RunTestClassesAsync() in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestCollectionRunner.cs:line 130
     at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs:line 1015
     at Xunit.Sdk.TestCollectionRunner`1.RunTestClassesAsync()
     at Xunit.Sdk.TestCollectionRunner`1.RunAsync() in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestCollectionRunner.cs:line 101
     at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs:line 1015
     at Xunit.Sdk.TestCollectionRunner`1.RunAsync()
     at Xunit.Sdk.XunitTestAssemblyRunner.RunTestCollectionAsync(IMessageBus messageBus, ITestCollection testCollection, IEnumerable`1 testCases, CancellationTokenSource cancellationTokenSource) in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\XunitTestAssemblyRunner.cs:line 235
     at Xunit.Sdk.XunitTestAssemblyRunner.<>c__DisplayClass14_2.<RunTestCollectionsAsync>b__2() in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\XunitTestAssemblyRunner.cs:line 184
     at System.Threading.Tasks.Task`1.InnerInvoke() in /_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Future.cs:line 518
     at System.Threading.Tasks.Task.<>c.<.cctor>b__274_0(Object obj) in /_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs:line 2428
     at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/System.Private.CoreLib/shared/System/Threading/ExecutionContext.cs:line 172
     at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) in /_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs:line 2385
     at System.Threading.Tasks.Task.ExecuteEntry() in /_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs:line 2296
     at System.Threading.Tasks.SynchronizationContextTaskScheduler.<>c.<.cctor>b__8_0(Object s) in /_/src/System.Private.CoreLib/shared/System/Threading/Tasks/TaskScheduler.cs:line 641
     at Xunit.Sdk.MaxConcurrencySyncContext.RunOnSyncContext(SendOrPostCallback callback, Object state) in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\MaxConcurrencySyncContext.cs:line 107
     at Xunit.Sdk.MaxConcurrencySyncContext.<>c__DisplayClass11_0.<WorkerThreadProc>b__0(Object _) in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\MaxConcurrencySyncContext.cs:line 96
     at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/System.Private.CoreLib/shared/System/Threading/ExecutionContext.cs:line 172
     at Xunit.Sdk.ExecutionContextHelper.Run(Object context, Action`1 action) in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Utility\ExecutionContextHelper.cs:line 111
     at Xunit.Sdk.MaxConcurrencySyncContext.WorkerThreadProc() in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\MaxConcurrencySyncContext.cs:line 89
     at Xunit.Sdk.XunitWorkerThread.<>c.<QueueUserWorkItem>b__5_0(Object _) in C:\Dev\xunit\xunit\src\common\XunitWorkerThread.cs:line 37
     at System.Threading.Tasks.Task.InnerInvoke() in /_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs:line 2449
     at System.Threading.Tasks.Task.<>c.<.cctor>b__274_0(Object obj) in /_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs:line 2428
     at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/System.Private.CoreLib/shared/System/Threading/ExecutionContext.cs:line 172
     at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) in /_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs:line 2385
     at System.Threading.Tasks.Task.ExecuteEntryUnsafe(Thread threadPoolThread) in /_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs:line 2327
     at System.Threading.Tasks.ThreadPoolTaskScheduler.<>c.<.cctor>b__10_0(Object s) in /_/src/System.Private.CoreLib/shared/System/Threading/Tasks/ThreadPoolTaskScheduler.cs:line 37
     at System.Threading.ThreadHelper.ThreadStart_Context(Object state) in /_/src/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs:line 50
     at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/System.Private.CoreLib/shared/System/Threading/ExecutionContext.cs:line 172
     at System.Threading.ThreadHelper.ThreadStart(Object obj) in /_/src/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs:line 83
  Unknown Chain Status: UnparseableExtension
  /Users/dotnet/new_source/corefx/artifacts/bin/System.Security.Cryptography.X509Certificates.Tests/netcoreapp-OSX-Debug/RunTests.sh: line 161: 61473 Abort trap: 6           "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Security.Cryptography.X509Certificates.Tests.runtimeconfig.json xunit.console.dll System.Security.Cryptography.X509Certificates.Tests.dll -xml testResults.xml -nologo -notrait category=nonnetcoreapptests -notrait category=nonosxtests -notrait category=OuterLoop -notrait category=failing $RSP_FILE
  ~/new_source/corefx/src/System.Security.Cryptography.X509Certificates/tests

[bartonjs]

The string UnparseableExtension isn't known to the mapper (https://github.com/dotnet/corefx/blob/249b5122a33ea08d2d00581ed691642c18ce1448/src/Native/Unix/System.Security.Cryptography.Native.Apple/pal_x509chain.c#L158-L195).

Might be worth expanding https://github.com/dotnet/corefx/blob/d7631df16a92da0e21bcb64240d5007f0dab95ed/src/System.Security.Cryptography.X509Certificates/tests/DynamicChainTests.cs#L241 to also show what chain codes are reported by Windows and/or OpenSSL; then see if there's something better than "ignore" that we can map this new code to.

[krwq]

repros for me as well

[buyaa-n]

if add that new "UnparseableExtension" status to ignored case all tests passed

[krwq]

@buyaa-n are you planning to send PR with that fix or is this something I should pick up? (unless it's already in the PR/merged)

[buyaa-n]

as @bartonjs mentioned ignoring the status might not enough, my effort was just to see test result if there is any other failure. If you have capacity to look into it please go ahead, you might find the mapping value for new code faster than me.

[buyaa-n]

@krwq am gonna raise PR for ignoring the new status as temporary fix. But will leave this issue open for you to look at mapping that value to desired response code

[krwq]

Per offline conversation we will map that error to PAL_X509InvalidExtension/X509ChainStatusFlags.InvalidExtension since it seems to be fitting best. Currently Windows does not propagate to X509Chain.ChainStatus so we will leave that part out of this issue and @buyaa-n will create a new one (we haven't checked Linux, on OSX propagation is for free and we do not need to do any extra changes)

[krwq]

@buyaa-n are you planning to open 3.0 PR?

[buyaa-n]

yes, on its way

[karelz]

Reopening to track the 3.0 port in PR dotnet/corefx#40117

[buyaa-n]

Closing as 3.0 port merged