System.Net.Http.Functional.Tests failing with "Process terminated. Assertion failed." in SetParentHandle

[ahsonkhan]

From dotnet/corefx#38469

https://mc.dot.net/#/user/dotnet-bot/pr~2Fdotnet~2Fcorefx~2Frefs~2Fpull~2F38469~2Fmerge/test~2Ffunctional~2Fcli~2Finnerloop~2F/20190624.21/workItem/System.Net.Http.Functional.Tests/wilogs

Windows.81.Amd64.Open-x64:Debug
System.Net.Http.Functional.Tests

C:\dotnetbuild\work\5965e5fe-719d-4ba9-a256-d3d60b6a7a70\Work\ed3631b2-b397-4473-8088-590754b65a5a\Exec>"C:\dotnetbuild\work\5965e5fe-719d-4ba9-a256-d3d60b6a7a70\Payload\dotnet.exe" xunit.console.dll System.Net.Http.Functional.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=nonnetcoreapptests -notrait category=nonwindowstests -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing  
  Discovering: System.Net.Http.Functional.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Net.Http.Functional.Tests (found 761 of 1097 test cases)
  Starting:    System.Net.Http.Functional.Tests (parallel test collections = on, max threads = 2)
    System.Net.Http.Functional.Tests.PlatformHandler_HttpClientEKUTest.HttpClient_ServerEKUClientAuth_Fails [SKIP]
      Condition(s) not met: "CanTestClientCertificates"
    System.Net.Http.Functional.Tests.PlatformHandler_HttpClientEKUTest.HttpClient_NoEKUClientAuth_Ok [SKIP]
      Condition(s) not met: "CanTestClientCertificates"
    System.Net.Http.Functional.Tests.PlatformHandler_HttpClientEKUTest.HttpClient_NoEKUServerAuth_Ok [SKIP]
      Condition(s) not met: "CanTestCertificates"
    System.Net.Http.Functional.Tests.PlatformHandler_HttpClientEKUTest.HttpClient_ClientEKUServerAuth_Fails [SKIP]
      Condition(s) not met: "CanTestCertificates"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClient_SelectedSites_Test.RetrieveSite_Succeeds [SKIP]
      Condition(s) not met: "IsSelectedSitesTestEnabled"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClient_SelectedSites_Test.RetrieveSite_Debug_Helper [SKIP]
      Condition(s) not met: "IsSelectedSitesTestEnabled"
    System.Net.Http.Functional.Tests.PlatformHandler_HttpClient_SelectedSites_Test.RetrieveSite_Succeeds [SKIP]
      Condition(s) not met: "IsSelectedSitesTestEnabled"
    System.Net.Http.Functional.Tests.PlatformHandler_HttpClient_SelectedSites_Test.RetrieveSite_Debug_Helper [SKIP]
      Condition(s) not met: "IsSelectedSitesTestEnabled"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientMiniStress_Http2.UnreadResponseMessage_Collectible [SKIP]
      Condition(s) not met: "IsStressModeEnabled"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientMiniStress_Http2.ManyClients_ManyGets [SKIP]
      Condition(s) not met: "IsStressModeEnabled"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientMiniStress_Http2.SingleClient_ManyGets_Async [SKIP]
      Condition(s) not met: "IsStressModeEnabled"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientMiniStress_Http2.MakeAndFaultManyRequests [SKIP]
      Condition(s) not met: "IsStressModeEnabled"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientMiniStress_Http2.SingleClient_ManyGets_Sync [SKIP]
      Condition(s) not met: "IsStressModeEnabled"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientMiniStress_Http11.UnreadResponseMessage_Collectible [SKIP]
      Condition(s) not met: "IsStressModeEnabled"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientMiniStress_Http11.MakeAndFaultManyRequests [SKIP]
      Condition(s) not met: "IsStressModeEnabled"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientMiniStress_Http11.ManyClients_ManyPosts_Async [SKIP]
      Condition(s) not met: "IsStressModeEnabled"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientMiniStress_Http11.SingleClient_ManyGets_Async [SKIP]
      Condition(s) not met: "IsStressModeEnabled"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientMiniStress_Http11.SingleClient_ManyGets_Sync [SKIP]
      Condition(s) not met: "IsStressModeEnabled"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientMiniStress_Http11.ManyClients_ManyGets [SKIP]
      Condition(s) not met: "IsStressModeEnabled"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientEKUTest.HttpClient_ClientEKUServerAuth_Fails [SKIP]
      Condition(s) not met: "CanTestCertificates"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientEKUTest.HttpClient_NoEKUServerAuth_Ok [SKIP]
      Condition(s) not met: "CanTestCertificates"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientEKUTest.HttpClient_NoEKUClientAuth_Ok [SKIP]
      Condition(s) not met: "CanTestClientCertificates"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientEKUTest.HttpClient_ServerEKUClientAuth_Fails [SKIP]
      Condition(s) not met: "CanTestClientCertificates"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientMiniStress_NoVersion.MakeAndFaultManyRequests [SKIP]
      Condition(s) not met: "IsStressModeEnabled"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientMiniStress_NoVersion.CreateAndDestroyManyClients [SKIP]
      Condition(s) not met: "IsStressModeEnabled"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientMiniStress_NoVersion.ManyClients_ManyGets [SKIP]
      Condition(s) not met: "IsStressModeEnabled"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientMiniStress_NoVersion.SingleClient_ManyGets_Sync [SKIP]
      Condition(s) not met: "IsStressModeEnabled"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientMiniStress_NoVersion.UnreadResponseMessage_Collectible [SKIP]
      Condition(s) not met: "IsStressModeEnabled"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientMiniStress_NoVersion.SingleClient_ManyGets_Async [SKIP]
      Condition(s) not met: "IsStressModeEnabled"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Authentication_Test.Credentials_DomainJoinedServerUsesKerberos_UseIpAddressAndHostHeader_Success [SKIP]
      Condition(s) not met: "IsDomainJoinedServerAvailable"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Authentication_Test.Credentials_ServerUsesWindowsAuthentication_Success [SKIP]
      Condition(s) not met: "IsWindowsServerAvailable"
    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Authentication_Test.Credentials_DomainJoinedServerUsesKerberos_Success [SKIP]
      Condition(s) not met: "IsDomainJoinedServerAvailable"
    System.Net.Http.Functional.Tests.PlatformHandler_HttpClientHandler_Authentication_Test.Credentials_ServerUsesWindowsAuthentication_Success [SKIP]
      Condition(s) not met: "IsWindowsServerAvailable"
    System.Net.Http.Functional.Tests.PlatformHandler_HttpClientHandlerTest.Ctor_ExpectedDefaultPropertyValues_UapPlatform [SKIP]
      Condition(s) not met: "IsUap"
    System.Net.Http.Functional.Tests.PlatformHandler_HttpClientHandler_Authentication_Test.Credentials_DomainJoinedServerUsesKerberos_Success [SKIP]
      Condition(s) not met: "IsDomainJoinedServerAvailable"
    System.Net.Http.Functional.Tests.PlatformHandler_HttpClientHandler_Authentication_Test.Credentials_DomainJoinedServerUsesKerberos_UseIpAddressAndHostHeader_Success [SKIP]
      Condition(s) not met: "IsDomainJoinedServerAvailable"
Process terminated. Assertion failed.
   at Interop.WinHttp.SafeWinHttpHandle.SetParentHandle(SafeWinHttpHandle parentHandle) in /_/src/Common/src/Interop/Windows/WinHttp/Interop.SafeWinHttpHandle.cs:line 35
   at System.Net.Http.WinHttpHandler.StartRequestAsync(WinHttpRequestState state) in /_/src/System.Net.Http.WinHttpHandler/src/System/Net/Http/WinHttpHandler.cs:line 813
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs:line 1019
   at System.Net.Http.WinHttpHandler.StartRequestAsync(WinHttpRequestState state)
   at System.Net.Http.WinHttpHandler.<>c.<SendAsync>b__101_0(Object s) in /_/src/System.Net.Http.WinHttpHandler/src/System/Net/Http/WinHttpHandler.cs:line 560
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/System.Private.CoreLib/shared/System/Threading/ExecutionContext.cs:line 289
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) in /_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs:line 2389
   at System.Threading.ThreadPoolWorkQueue.Dispatch() in /_/src/System.Private.CoreLib/shared/System/Threading/ThreadPool.cs:line 663
----- end Mon 06/24/2019 17:30:26.88 ----- exit code -2146232797 ----------------------------------------------------------
 Mon 06/24/2019-17:30:26.91
 Mon 06/24/2019-17:30:26.97
Got args https://dev.azure.com/dnceng/ public 5865096 eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6Im9PdmN6NU1fN3AtSGpJS2xGWHo5M3VfVjBabyJ9.eyJuYW1laWQiOiJlNDI4NWM4Yy0zZmQ1LTQyNjctOWIxYy00MjE5NWM0N2E1NTMiLCJzY3AiOiJMb2NhdGlvblNlcnZpY2UuQ29ubmVjdCBSZWFkQW5kUHVibGlzaFRlc3Q6OWVlNmQ0NzgtZDI4OC00N2Y3LWFhY2MtZjZlNmQwODJhZTZkIFJlYWRBbmRVcGRhdGVCdWlsZEJ5VXJpOjllZTZkNDc4LWQyODgtNDdmNy1hYWNjLWY2ZTZkMDgyYWU2ZC9kb3RuZXQvY29yZWZ4LzE5NjpCdWlsZC9CdWlsZC8yMzY0MzkiLCJhdWkiOiJiYjI5YmM5ZC1iZTIzLTRiNmEtYjBjMC1iMzczNTk4MzdiZWIiLCJzaWQiOiJkMmIwMTg3Ni0wZjc0LTRhMjMtYjc4Yi1iMmUzNjc5ZTRkMWUiLCJpc3MiOiJhcHAudnN0b2tlbi52aXN1YWxzdHVkaW8uY29tIiwiYXVkIjoiYXBwLnZzdG9rZW4udmlzdWFsc3R1ZGlvLmNvbXx2c286YjU1ZGU0ZWQtNGI1YS00MjE1LWE4ZTQtMGEwYTVmNzFlN2Q4IiwibmJmIjoxNTYxMzk0MTA3LCJleHAiOjE1NjE0MDQzMDd9.Oi-dWTU2rVYMoB119ZAQFBKleHR_SbbPZ2J66UcOqQoqWdmV41_kJunEgG-Ao0mRPVU-quxXurdwgtBN_bzDCCvhxWOpIg07qijI04O1_lIDEUGISFvcjPVFyuzQmfAahxR6AIIR5iBDS9Z-vzOxKEv2V5jMmvpn8NWvVPc2SH-6bHIDrdAKNXfh8hYZ86G_kntCGb67EIDmOMxp39Fn-ux6S896xsHVYQRALQoGEJ9Ef9G9zuPphzkoTKhFsfnEHpMxCk3Wiy4OiEyXmiK9Pk_I4E3uLRter-3GV_seZ2lMSoaGSdQc5dsLfPVUq3Ea0ZMtrtzWVzlErH1gT0QH3Q
Main thread starting workers
Worker 0: starting...
Worker 1: starting...
Worker 2: starting...
Worker 3: starting...
Worker 4: starting...
Worker 5: starting...
Worker 6: starting...
Worker 7: starting...
Worker 8: starting...
Worker 9: starting...
Beginning reading of test results.
Uploading results in batches of size 1000
Searching 'C:\dotnetbuild\work\5965e5fe-719d-4ba9-a256-d3d60b6a7a70\Work\ed3631b2-b397-4473-8088-590754b65a5a\Exec\..' for log files
Found log 'C:\dotnetbuild\work\5965e5fe-719d-4ba9-a256-d3d60b6a7a70\Work\ed3631b2-b397-4473-8088-590754b65a5a\Exec\..\console.f216fa0c.log'
Uri 'https://helixexternalresults.blob.core.windows.net/dotnet-corefx-refs-pull-38469-merge-5965e5fe719d4ba9a2/System.Net.Http.Functional.Tests/console.f216fa0c.log?sv=2017-07-29&sr=c&sig=aQOkH8idk7iXCUKDlu5pl%2FduGfTgQ2gVifobNDu%2FLZ8%3D&se=2019-07-04T17%3A08%3A09Z&sp=rl'
Generated log list: <ul><li><a href='https://helixexternalresults.blob.core.windows.net/dotnet-corefx-refs-pull-38469-merge-5965e5fe719d4ba9a2/System.Net.Http.Functional.Tests/console.f216fa0c.log?sv=2017-07-29&sr=c&sig=aQOkH8idk7iXCUKDlu5pl%2FduGfTgQ2gVifobNDu%2FLZ8%3D&se=2019-07-04T17%3A08%3A09Z&sp=rl' target='_blank'>console.f216fa0c.log</a></li></ul>
Searching 'C:\dotnetbuild\work\5965e5fe-719d-4ba9-a256-d3d60b6a7a70\Work\ed3631b2-b397-4473-8088-590754b65a5a\Exec' for test results files
No results file found in any of the following formats: xunit, junit, trx
Main thread finished queueing batches
Worker 0: uploaded 1 results
Main thread exiting
 Mon 06/24/2019-17:30:28.02
2019-06-24 17:30:28,540: ERROR: xunit-reporter(69): main: Unable to report xunit results: no test results xml file found.

cc @stephentoub, @wfurt, @krwq

[stephentoub]

This is from WinHttpHandler.
cc: @davidsh

[davidsh]

When I go back to that test failure from the link above (and verify it against the PR it referenced), the failure is in Nano and it isn't WinHttpHandler. It is HTTP/2 related to SocketsHttpHandler.

Process terminated. Assertion failed.
This should only be called after we've transitioned _hasWaiter to true to enable this GetWaiterTask call.
   at System.Net.Http.Http2Connection.Http2Stream.GetWaiterTask(CancellationToken cancellationToken) in /_/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Stream.cs:line 680
   at System.Net.Http.Http2Connection.Http2Stream.ReadDataAsync(Memory`1 buffer, CancellationToken cancellationToken) in /_/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Stream.cs:line 603
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs:line 1019
   at System.Net.Http.Http2Connection.Http2Stream.ReadDataAsync(Memory`1 buffer, CancellationToken cancellationToken)
   at System.Net.Http.Http2Connection.Http2Stream.Http2ReadStream.ReadAsync(Memory`1 destination, CancellationToken cancellationToken) in /_/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Stream.cs:line 795
   at System.IO.Stream.CopyToAsyncInternal(Stream destination, Int32 bufferSize, CancellationToken cancellationToken) in /_/src/System.Private.CoreLib/shared/System/IO/Stream.cs:line 140
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs:line 1019
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[TStateMachine](TStateMachine& stateMachine) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs:line 223
   at System.IO.Stream.CopyToAsyncInternal(Stream destination, Int32 bufferSize, CancellationToken cancellationToken)
   at System.IO.Stream.CopyToAsync(Stream destination, Int32 bufferSize, CancellationToken cancellationToken) in /_/src/System.Private.CoreLib/shared/System/IO/Stream.cs:line 130
   at System.Net.Http.HttpConnectionResponseContent.SerializeToStreamAsync(Stream stream, TransportContext context, CancellationToken cancellationToken) in /_/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionResponseContent.cs:line 47
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs:line 1019
   at System.Net.Http.HttpConnectionResponseContent.SerializeToStreamAsync(Stream stream, TransportContext context, CancellationToken cancellationToken)
   at System.Net.Http.HttpContent.LoadIntoBufferAsync(Int64 maxBufferSize, CancellationToken cancellationToken) in /_/src/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 413
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) in /_/src/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 505
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/System.Private.CoreLib/shared/System/Threading/ExecutionContext.cs:line 172
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs:line 610
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining) in /_/src/System.Private.CoreLib/shared/System/Threading/Tasks/TaskContinuation.cs:line 794
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject) in /_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs:line 3321
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result) in /_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Future.cs:line 419
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(TResult result) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs:line 720
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in /_/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:line 84
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/System.Private.CoreLib/shared/System/Threading/ExecutionContext.cs:line 172
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs:line 610
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining) in /_/src/System.Private.CoreLib/shared/System/Threading/Tasks/TaskContinuation.cs:line 794
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject) in /_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs:line 3321
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result) in /_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Future.cs:line 419
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(TResult result) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs:line 720
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 552
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/System.Private.CoreLib/shared/System/Threading/ExecutionContext.cs:line 172
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs:line 610
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining) in /_/src/System.Private.CoreLib/shared/System/Threading/Tasks/TaskContinuation.cs:line 794
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject) in /_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs:line 3321
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result) in /_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Future.cs:line 419
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(TResult result) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs:line 720
   at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in /_/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 1466
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/System.Private.CoreLib/shared/System/Threading/ExecutionContext.cs:line 172
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs:line 610
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining) in /_/src/System.Private.CoreLib/shared/System/Threading/Tasks/TaskContinuation.cs:line 794
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject) in /_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs:line 3321
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result) in /_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Future.cs:line 419
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(TResult result) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs:line 720
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult() in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs:line 273
   at System.Net.Http.Http2Connection.Http2Stream.ReadResponseHeadersAsync(CancellationToken cancellationToken) in /_/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Stream.cs:line 500
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/System.Private.CoreLib/shared/System/Threading/ExecutionContext.cs:line 172
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs:line 610
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining) in /_/src/System.Private.CoreLib/shared/System/Threading/Tasks/TaskContinuation.cs:line 794
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject) in /_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs:line 3321
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result) in /_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Future.cs:line 419
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(TResult result) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs:line 720
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult() in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs:line 273
   at System.Net.Http.Http2Connection.Http2Stream.<>c__DisplayClass48_0.<<GetWaiterTask>g__GetWaiterTaskCore|0>d.MoveNext() in /_/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Stream.cs:line 717
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/System.Private.CoreLib/shared/System/Threading/ExecutionContext.cs:line 289
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs:line 618
   at System.Threading.ThreadPoolWorkQueue.Dispatch() in /_/src/System.Private.CoreLib/shared/System/Threading/ThreadPool.cs:line 663

[stephentoub]

the failure is in Nano and it isn't WinHttpHandler. It is HTTP/2 related to SocketsHttpHandler

@davidsh, maybe @ahsonkhan shared the wrong link (or more likely the tests were rerun and the results overwritten), but obviously this came from somewhere:

Process terminated. Assertion failed.
   at Interop.WinHttp.SafeWinHttpHandle.SetParentHandle(SafeWinHttpHandle parentHandle) in /_/src/Common/src/Interop/Windows/WinHttp/Interop.SafeWinHttpHandle.cs:line 35
   at System.Net.Http.WinHttpHandler.StartRequestAsync(WinHttpRequestState state) in /_/src/System.Net.Http.WinHttpHandler/src/System/Net/Http/WinHttpHandler.cs:line 813
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs:line 1019
   at System.Net.Http.WinHttpHandler.StartRequestAsync(WinHttpRequestState state)
   at System.Net.Http.WinHttpHandler.<>c.<SendAsync>b__101_0(Object s) in /_/src/System.Net.Http.WinHttpHandler/src/System/Net/Http/WinHttpHandler.cs:line 560
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/System.Private.CoreLib/shared/System/Threading/ExecutionContext.cs:line 289
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) in /_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs:line 2389
   at System.Threading.ThreadPoolWorkQueue.Dispatch() in /_/src/System.Private.CoreLib/shared/System/Threading/ThreadPool.cs:line 663

We should keep this issue focused on the WinHttpHandler failure and open a new issue for the SocketsHttpHandler failure; that assert is one I just added, and the fact that it triggered is troubling.

[stephentoub]

that assert is one I just added, and the fact that it triggered is troubling.

Ah, nevermind, the assert I added is bogus. I'll remove it (dotnet/corefx#38842).

[davidsh]

at Interop.WinHttp.SafeWinHttpHandle.SetParentHandle(SafeWinHttpHandle parentHandle) in /_/src/Common/src/Interop/Windows/WinHttp/Interop.SafeWinHttpHandle.cs:line 35

We haven't seen this failure ever that I'm aware of in WinHttpHandler based CI runs. But it is possible that it's related to handle lifetime issues we already have (#9228). Those are issues we've chosen not to focus on since SocketsHttpHandler is the default handler now. Unless we start to see this failure occur more often, I'm setting the milestone to 'Future' on this.

[stephentoub]

Happened again here:
dotnet/corefx#38804 (comment)

• Catastrophic failure on Windows.7.Amd64.Open-x64 for System.Net.Http.Functional.Tests:

Process terminated. Assertion failed.
   at Interop.WinHttp.SafeWinHttpHandle.SetParentHandle(SafeWinHttpHandle parentHandle) in /_/src/Common/src/Interop/Windows/WinHttp/Interop.SafeWinHttpHandle.cs:line 35

   at System.Net.Http.WinHttpHandler.StartRequestAsync(WinHttpRequestState state) in /_/src/System.Net.Http.WinHttpHandler/src/System/Net/Http/WinHttpHandler.cs:line 813

   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs:line 1019

   at System.Net.Http.WinHttpHandler.StartRequestAsync(WinHttpRequestState state)

   at System.Net.Http.WinHttpHandler.<>c.<SendAsync>b__101_0(Object s) in /_/src/System.Net.Http.WinHttpHandler/src/System/Net/Http/WinHttpHandler.cs:line 560

   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/System.Private.CoreLib/shared/System/Threading/ExecutionContext.cs:line 289

   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) in /_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs:line 2389

   at System.Threading.ThreadPoolWorkQueue.Dispatch() in /_/src/System.Private.CoreLib/shared/System/Threading/ThreadPool.cs:line 663
----- end Tue 06/25/2019  5:07:26.86 ----- exit code -2146232797 ----------------------------------------------------------

cc: @tmds

[ahsonkhan]

Failed again here (in release/3.0 branch - consistently): dotnet/corefx#39850

...
System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Authentication_Test.Credentials_DomainJoinedServerUsesKerberos_UseIpAddressAndHostHeader_Success [SKIP]
      Condition(s) not met: "IsDomainJoinedServerAvailable"
*** Error in `/home/helixbot/work/84e1ba26-c67b-471f-b2f2-da3d3bf31b3c/Payload/dotnet': double free or corruption (!prev): 0x00007fab300404c0 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x70bfb)[0x7fabf3271bfb]
/lib/x86_64-linux-gnu/libc.so.6(+0x76fc6)[0x7fabf3277fc6]
/lib/x86_64-linux-gnu/libc.so.6(+0x7780e)[0x7fabf327880e]
/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2(CRYPTO_free+0x1d)[0x7fab22a85edd]
/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2(lh_free+0x57)[0x7fab22b3d5b7]
/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2(+0x131cfa)[0x7fab22b3fcfa]
/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2(ERR_remove_thread_state+0x40)[0x7fab22b40890]
/usr/lib/x86_64-linux-gnu/libcurl.so.4(+0x265c2)[0x7fab23ba15c2]
/usr/lib/x86_64-linux-gnu/libcurl.so.4(curl_multi_cleanup+0x158)[0x7fab23bb4f58]
/home/helixbot/work/84e1ba26-c67b-471f-b2f2-da3d3bf31b3c/Payload/shared/Microsoft.NETCore.App/3.0.0/System.Net.Http.Native.so(HttpNative_MultiDestroy+0x15)[0x7fab23dfd175]
[0x7fab79ca8ab9]
======= Memory map: ========
00400000-00410000 r-xp 00000000 08:01 271614                             /home/helixbot/work/84e1ba26-c67b-471f-b2f2-da3d3bf31b3c/Payload/dotnet
0060f000-00610000 r--p 0000f000 08:01 271614                             /home/helixbot/work/84e1ba26-c67b-471f-b2f2-da3d3bf31b3c/Payload/dotnet
00610000-00611000 rw-p 00010000 08:01 271614                             /home/helixbot/work/84e1ba26-c67b-471f-b2f2-da3d3bf31b3c/Payload/dotnet
01812000-01d97000 rw-p 00000000 00:00 0                                  [heap]
...
7fabeabb1000-7fabead56000 r--s    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Authentication_Test.Credentials_ServerUsesWindowsAuthentication_Success [SKIP]
      Condition(s) not met: "IsWindowsServerAvailable"
 00000000 08:01 271898                     /home/helixbot/work/84e1ba26-c67b-471f-b2f2-da3d3bf31b3c/Work/21a5cc05-a3c5-485f-a569-b37745d05f0e/Exec/System.Net.Http.Functional.Tests.dll
7fabead56000-7fabead93000 r--s 00000000 08:01 271901                     /home/helixbot/work/84e1ba26-c67b-471f-b2f2-da3d3bf31b3c/Work/21a5cc05-a3c5-485f-a569-b37745d05f0e/Exec/xunit.execution.dotnet.dll
7fabead93000-7fabead9f000 r--s 00000000 08:01 397568                     /home/helixbot/work/84e1ba26-c67b-471f-b2f2-da3d3bf31b3c/Payload/shared/Microsoft.NETCore.App/3.0.0/System.Diagnostics.TraceSource.dll
...

7fabf4478000-7fabf4479000 ---p 00000000 00:00 0 
7fabf4479000-7fabf447c000 rw-p 00000000 00:00 0 
7fabf447c000-7fabf447d000 ---p 00000000 00:00 0 
7fabf447d000-7fabf447e000 ---p 00000000 00:00 0 
7fabf447e000-7fabf4481000 rw-p 00000000 00:00 0 
7fabf4481000-7fabf4482000 r--p 00023000 08:01 1709119                    /lib/x86_64-linux-gnu/ld-2.24.so
7fabf4482000-7fabf4483000 rw-p 00024000 08:01 1709119                    /lib/x86_64-linux-gnu/ld-2.24.so
7fabf4483000-7fabf4484000 rw-p 00000000 00:00 0 
7ffff5709000-7ffff572a000 rw-p 00000000 00:00 0                          [stack]
7ffff5737000-7ffff5739000 r--p 00000000 00:00 0                          [vvar]
7ffff5739000-7ffff573b000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
./RunTests.sh: line 161:  7576 Aborted                 (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Net.Http.Functional.Tests.runtimeconfig.json xunit.console.dll System.Net.Http.Functional.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=nonnetcoreapptests -notrait category=nonlinuxtests -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
~/work/84e1ba26-c67b-471f-b2f2-da3d3bf31b3c/Work/21a5cc05-a3c5-485f-a569-b37745d05f0e/Exec
----- end Mon Jul 29 04:56:07 UTC 2019 ----- exit code 134 ----------------------------------------------------------
exit code 134 means SIGABRT Abort. Managed or native assert, or runtime check such as heap corruption, caused call to abort(). Core dumped.
Looking around for any Linux dump...
... found no dump in /home/helixbot/work/84e1ba26-c67b-471f-b2f2-da3d3bf31b3c/Work/21a5cc05-a3c5-485f-a569-b37745d05f0e/Exec
+ export _commandExitCode=134
+ /bin/sh /home/helixbot/work/84e1ba26-c67b-471f-b2f2-da3d3bf31b3c/Payload/reporter/run.sh https://dev.azure.com/dnceng/ public 7932104 eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6Im9PdmN6NU1fN3AtSGpJS2xGWHo5M3VfVjBabyJ9.eyJuYW1laWQiOiJlNDI4NWM4Yy0zZmQ1LTQyNjctOWIxYy00MjE5NWM0N2E1NTMiLCJzY3AiOiJMb2NhdGlvblNlcnZpY2UuQ29ubmVjdCBSZWFkQW5kUHVibGlzaFRlc3Q6OWVlNmQ0NzgtZDI4OC00N2Y3LWFhY2MtZjZlNmQwODJhZTZkIFJlYWRBbmRVcGRhdGVCdWlsZEJ5VXJpOjllZTZkNDc4LWQyODgtNDdmNy1hYWNjLWY2ZTZkMDgyYWU2ZC9kb3RuZXQvY29yZWZ4LzE5NjpCdWlsZC9CdWlsZC8yODM4NzkiLCJhdWkiOiJhNmFkODI3Ny0xZTAwLTQ0M2MtOGFmNy0yN2EyNjQ0NjJmZWMiLCJzaWQiOiIyZTlhMjMyOS1jNWQzLTQ2OGUtYTcxNC1lMzhhYzJlZGRkYTgiLCJpc3MiOiJhcHAudnN0b2tlbi52aXN1YWxzdHVkaW8uY29tIiwiYXVkIjoiYXBwLnZzdG9rZW4udmlzdWFsc3R1ZGlvLmNvbXx2c286YjU1ZGU0ZWQtNGI1YS00MjE1LWE4ZTQtMGEwYTVmNzFlN2Q4IiwibmJmIjoxNTY0MzczNTY4LCJleHAiOjE1NjQzODU1Njh9.1WKuqZ427H5lDuwpau4ZUJemlCAO2yLCNczupYF2CGsUQxw-HtxFwNTEP8wAup6BjHt8j9JnMVCJTlLCNjlTeyJY2EXechH3IdJbXjWuS9dy6bmuQW2Tmm_AexADnFK8N09yRPfzP1Ut7SsneT4iJk6lwRcBxGz8FtxmJTXSuKsccth3RSz6zJlQchCu755wzyQVYLIYSNeNy3QMj0b0gjrAm2UcfVYjdrFyiet6HLlyLFGJFxvY8r7b8EwfBUpsGVLb7FYbSR2yhnm0UToUghQp71OPlAFbNWrfYiW7hS8gD4_HZieJ9ukq96_Y_u3_ekwJ9RqKiwTsNe3gAmsVig
+ dirname -- /home/helixbot/work/84e1ba26-c67b-471f-b2f2-da3d3bf31b3c/Payload/reporter/run.sh
+ CDPATH= cd -- /home/helixbot/work/84e1ba26-c67b-471f-b2f2-da3d3bf31b3c/Payload/reporter
+ pwd
+ script_path=/home/helixbot/work/84e1ba26-c67b-471f-b2f2-da3d3bf31b3c/Payload/reporter
+ ENV_PATH=/home/helixbot/.vsts-env
+ date -u +%FT%TZ
2019-07-29T04:56:08Z
+ [ ! -f /home/helixbot/.vsts-env/bin/python ]
+ /home/helixbot/.vsts-env/bin/python -c import vsts
+ echo vsts module already available
vsts module already available
+ /home/helixbot/.vsts-env/bin/python -c import future
+ echo future module already available
future module already available
+ date -u +%FT%TZ
2019-07-29T04:56:08Z
+ /home/helixbot/.vsts-env/bin/python /home/helixbot/work/84e1ba26-c67b-471f-b2f2-da3d3bf31b3c/Payload/reporter/run.py https://dev.azure.com/dnceng/ public 7932104 eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6Im9PdmN6NU1fN3AtSGpJS2xGWHo5M3VfVjBabyJ9.eyJuYW1laWQiOiJlNDI4NWM4Yy0zZmQ1LTQyNjctOWIxYy00MjE5NWM0N2E1NTMiLCJzY3AiOiJMb2NhdGlvblNlcnZpY2UuQ29ubmVjdCBSZWFkQW5kUHVibGlzaFRlc3Q6OWVlNmQ0NzgtZDI4OC00N2Y3LWFhY2MtZjZlNmQwODJhZTZkIFJlYWRBbmRVcGRhdGVCdWlsZEJ5VXJpOjllZTZkNDc4LWQyODgtNDdmNy1hYWNjLWY2ZTZkMDgyYWU2ZC9kb3RuZXQvY29yZWZ4LzE5NjpCdWlsZC9CdWlsZC8yODM4NzkiLCJhdWkiOiJhNmFkODI3Ny0xZTAwLTQ0M2MtOGFmNy0yN2EyNjQ0NjJmZWMiLCJzaWQiOiIyZTlhMjMyOS1jNWQzLTQ2OGUtYTcxNC1lMzhhYzJlZGRkYTgiLCJpc3MiOiJhcHAudnN0b2tlbi52aXN1YWxzdHVkaW8uY29tIiwiYXVkIjoiYXBwLnZzdG9rZW4udmlzdWFsc3R1ZGlvLmNvbXx2c286YjU1ZGU0ZWQtNGI1YS00MjE1LWE4ZTQtMGEwYTVmNzFlN2Q4IiwibmJmIjoxNTY0MzczNTY4LCJleHAiOjE1NjQzODU1Njh9.1WKuqZ427H5lDuwpau4ZUJemlCAO2yLCNczupYF2CGsUQxw-HtxFwNTEP8wAup6BjHt8j9JnMVCJTlLCNjlTeyJY2EXechH3IdJbXjWuS9dy6bmuQW2Tmm_AexADnFK8N09yRPfzP1Ut7SsneT4iJk6lwRcBxGz8FtxmJTXSuKsccth3RSz6zJlQchCu755wzyQVYLIYSNeNy3QMj0b0gjrAm2UcfVYjdrFyiet6HLlyLFGJFxvY8r7b8EwfBUpsGVLb7FYbSR2yhnm0UToUghQp71OPlAFbNWrfYiW7hS8gD4_HZieJ9ukq96_Y_u3_ekwJ9RqKiwTsNe3gAmsVig
Got args https://dev.azure.com/dnceng/ public 7932104 eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6Im9PdmN6NU1fN3AtSGpJS2xGWHo5M3VfVjBabyJ9.eyJuYW1laWQiOiJlNDI4NWM4Yy0zZmQ1LTQyNjctOWIxYy00MjE5NWM0N2E1NTMiLCJzY3AiOiJMb2NhdGlvblNlcnZpY2UuQ29ubmVjdCBSZWFkQW5kUHVibGlzaFRlc3Q6OWVlNmQ0NzgtZDI4OC00N2Y3LWFhY2MtZjZlNmQwODJhZTZkIFJlYWRBbmRVcGRhdGVCdWlsZEJ5VXJpOjllZTZkNDc4LWQyODgtNDdmNy1hYWNjLWY2ZTZkMDgyYWU2ZC9kb3RuZXQvY29yZWZ4LzE5NjpCdWlsZC9CdWlsZC8yODM4NzkiLCJhdWkiOiJhNmFkODI3Ny0xZTAwLTQ0M2MtOGFmNy0yN2EyNjQ0NjJmZWMiLCJzaWQiOiIyZTlhMjMyOS1jNWQzLTQ2OGUtYTcxNC1lMzhhYzJlZGRkYTgiLCJpc3MiOiJhcHAudnN0b2tlbi52aXN1YWxzdHVkaW8uY29tIiwiYXVkIjoiYXBwLnZzdG9rZW4udmlzdWFsc3R1ZGlvLmNvbXx2c286YjU1ZGU0ZWQtNGI1YS00MjE1LWE4ZTQtMGEwYTVmNzFlN2Q4IiwibmJmIjoxNTY0MzczNTY4LCJleHAiOjE1NjQzODU1Njh9.1WKuqZ427H5lDuwpau4ZUJemlCAO2yLCNczupYF2CGsUQxw-HtxFwNTEP8wAup6BjHt8j9JnMVCJTlLCNjlTeyJY2EXechH3IdJbXjWuS9dy6bmuQW2Tmm_AexADnFK8N09yRPfzP1Ut7SsneT4iJk6lwRcBxGz8FtxmJTXSuKsccth3RSz6zJlQchCu755wzyQVYLIYSNeNy3QMj0b0gjrAm2UcfVYjdrFyiet6HLlyLFGJFxvY8r7b8EwfBUpsGVLb7FYbSR2yhnm0UToUghQp71OPlAFbNWrfYiW7hS8gD4_HZieJ9ukq96_Y_u3_ekwJ9RqKiwTsNe3gAmsVig
Main thread starting workers
Worker 0: starting...
Worker 1: starting...
Worker 2: starting...
Worker 3: starting...
Worker 4: starting...
Worker 5: starting...
Worker 6: starting...
Worker 7: starting...
Worker 8: starting...
Worker 9: starting...
Beginning reading of test results.
Uploading results in batches of size 1000
Searching '/home/helixbot/work/84e1ba26-c67b-471f-b2f2-da3d3bf31b3c/Work/21a5cc05-a3c5-485f-a569-b37745d05f0e/Exec/..' for log files
Found log '/home/helixbot/work/84e1ba26-c67b-471f-b2f2-da3d3bf31b3c/Work/21a5cc05-a3c5-485f-a569-b37745d05f0e/Exec/../console.07da34c2.log'
Uri 'https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-corefx-refs-pull-39850-merge-84e1ba26c67b471fb2/System.Net.Http.Functional.Tests/console.07da34c2.log?sv=2017-07-29&sr=c&sig=fgegro4PBzHt0NM0%2FmMYq%2BC3tvrPiLQ7c%2BYzv5yu8wI%3D&se=2019-08-08T04%3A41%3A59Z&sp=rl'
Generated log list: <ul><li><a href='https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-corefx-refs-pull-39850-merge-84e1ba26c67b471fb2/System.Net.Http.Functional.Tests/console.07da34c2.log?sv=2017-07-29&sr=c&sig=fgegro4PBzHt0NM0%2FmMYq%2BC3tvrPiLQ7c%2BYzv5yu8wI%3D&se=2019-08-08T04%3A41%3A59Z&sp=rl' target='_blank'>console.07da34c2.log</a></li></ul>
Searching '/home/helixbot/work/84e1ba26-c67b-471f-b2f2-da3d3bf31b3c/Work/21a5cc05-a3c5-485f-a569-b37745d05f0e/Exec' for test results files
No results file found in any of the following formats: xunit, junit, trx
Main thread finished queueing batches
Worker 0: uploaded 1 results
Main thread exiting
+ date -u +%FT%TZ
2019-07-29T04:56:08Z
+ /usr/bin/python3 /home/helixbot/work/84e1ba26-c67b-471f-b2f2-da3d3bf31b3c/Payload/xunit-reporter.py
2019-07-29 04:56:09,151: ERROR: xunit-reporter(69): main: Unable to report xunit results: no test results xml file found.
+ exit 134

[stephentoub]

Failed again here

Something is wrong here... the test name says it's testing SocketsHttpHandler, but the stack trace shows it's CurlHandler.

[stephentoub]

Nevermind, that test name wasn't related; just confusing order of output.

@ahsonkhan, why do you believe that crash is related to this issue?

[ahsonkhan]

but the stack trace shows it's CurlHandler.

Where do you see the stack trace?

why do you believe that crash is related to this issue?

I got confused by the ordering in the output log (since it shows an error right after the skipped test: System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Authentication_Test.Credentials_DomainJoinedServerUsesKerberos_UseIpAddressAndHostHeader_Success [SKIP] and assumed it was related. I can't tell which test caused the crash from the log alone.

Should I open a new issue for this failure, then?

[stephentoub]

Where do you see the stack trace

======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x70bfb)[0x7fabf3271bfb]
/lib/x86_64-linux-gnu/libc.so.6(+0x76fc6)[0x7fabf3277fc6]
/lib/x86_64-linux-gnu/libc.so.6(+0x7780e)[0x7fabf327880e]
/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2(CRYPTO_free+0x1d)[0x7fab22a85edd]
/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2(lh_free+0x57)[0x7fab22b3d5b7]
/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2(+0x131cfa)[0x7fab22b3fcfa]
/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2(ERR_remove_thread_state+0x40)[0x7fab22b40890]
/usr/lib/x86_64-linux-gnu/libcurl.so.4(+0x265c2)[0x7fab23ba15c2]
/usr/lib/x86_64-linux-gnu/libcurl.so.4(curl_multi_cleanup+0x158)[0x7fab23bb4f58]
/home/helixbot/work/84e1ba26-c67b-471f-b2f2-da3d3bf31b3c/Payload/shared/Microsoft.NETCore.App/3.0.0/System.Net.Http.Native.so(HttpNative_MultiDestroy+0x15)[0x7fab23dfd175]

[stephentoub]

Failed again here:
https://dev.azure.com/dnceng/public/_build/results?buildId=377373&view=ms.vss-test-web.build-test-results-tab&runId=11586988&paneView=debug

Process terminated. Assertion failed.
   at Interop.WinHttp.SafeWinHttpHandle.SetParentHandle(SafeWinHttpHandle parentHandle) in /_/src/Common/src/Interop/Windows/WinHttp/Interop.SafeWinHttpHandle.cs:line 35
   at System.Net.Http.WinHttpHandler.StartRequestAsync(WinHttpRequestState state) in /_/src/System.Net.Http.WinHttpHandler/src/System/Net/Http/WinHttpHandler.cs:line 813
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs:line 42
   at System.Net.Http.WinHttpHandler.StartRequestAsync(WinHttpRequestState state)
   at System.Net.Http.WinHttpHandler.<>c.<SendAsync>b__101_0(Object s) in /_/src/System.Net.Http.WinHttpHandler/src/System/Net/Http/WinHttpHandler.cs:line 560
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/System.Private.CoreLib/shared/System/Threading/ExecutionContext.cs:line 289
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) in /_/src/System.Private.CoreLib/shared/System/Threading/Tasks/Task.cs:line 2315
   at System.Threading.ThreadPoolWorkQueue.Dispatch() in /_/src/System.Private.CoreLib/shared/System/Threading/ThreadPool.cs:line 662

[karelz]

Triage: Looks like a bug in WinHttpHandler (probably a race condition).