Work items crashing on arm64-mono_interpreter

[ViktorHofer]

https://dev.azure.com/dnceng/public/_build/results?buildId=670388&view=ms.vss-test-web.build-test-results-tab&runId=20765838&paneView=attachments&resultId=172282

Work items:

• System.Xml.XPath.XmlDocument.Tests: https://helix.dot.net/api/2019-06-17/jobs/06480c01-8731-4e95-b305-36626c6432b7/workitems/System.Xml.XPath.XmlDocument.Tests/files/console.40a7c797.log
• System.Net.WebSockets.Client.Tests: https://helix.dot.net/api/2019-06-17/jobs/06480c01-8731-4e95-b305-36626c6432b7/workitems/System.Net.WebSockets.Client.Tests/files/console.4977c118.log

Configuration: net5.0-Linux-Debug-arm64-mono_interpreter_release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-helix-arm64v8-bfcd90a-20200127194925

=================================================================
	Managed Stacktrace:
=================================================================
	  at <unknown> <0xffffffff>
	  at System.Reflection.RuntimeMethodInfo:InternalInvoke <0x0002e>
	  at System.Reflection.RuntimeMethodInfo:Invoke <0x00114>
	  at System.Reflection.MethodBase:Invoke <0x00014>
	  at Xunit.Sdk.TestInvoker`1:CallTestMethod <0x00014>
	  at <<InvokeTestMethodAsync>b__1>d:MoveNext <0x0017c>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0004a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder:Start <0x00004>
	  at <>c__DisplayClass48_1:<InvokeTestMethodAsync>b__1 <0x00062>
	  at <AggregateAsync>d__4:MoveNext <0x0003e>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0004a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder:Start <0x00004>
	  at Xunit.Sdk.ExecutionTimer:AggregateAsync <0x0006e>
	  at <>c__DisplayClass48_1:<InvokeTestMethodAsync>b__0 <0x00040>
	  at <RunAsync>d__9:MoveNext <0x00028>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0004a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder:Start <0x00004>
	  at Xunit.Sdk.ExceptionAggregator:RunAsync <0x0006e>
	  at <InvokeTestMethodAsync>d__48:MoveNext <0x000aa>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0004a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x00004>
	  at Xunit.Sdk.TestInvoker`1:InvokeTestMethodAsync <0x0006e>
	  at Xunit.Sdk.XunitTestInvoker:InvokeTestMethodAsync <0x00072>
	  at <<RunAsync>b__47_0>d:MoveNext <0x00298>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0004a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x00004>
	  at Xunit.Sdk.TestInvoker`1:<RunAsync>b__47_0 <0x00062>
	  at <RunAsync>d__10`1:MoveNext <0x00028>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0004a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x00004>
	  at Xunit.Sdk.ExceptionAggregator:RunAsync <0x00064>
	  at Xunit.Sdk.TestInvoker`1:RunAsync <0x00018>
	  at Xunit.Sdk.XunitTestRunner:InvokeTestMethodAsync <0x0003c>
	  at <InvokeTestAsync>d__4:MoveNext <0x000ae>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0004a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x00004>
	  at Xunit.Sdk.XunitTestRunner:InvokeTestAsync <0x0006e>
	  at <>c__DisplayClass43_0:<RunAsync>b__0 <0x00010>
	  at <RunAsync>d__10`1:MoveNext <0x00028>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0004a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x00004>
	  at Xunit.Sdk.ExceptionAggregator:RunAsync <0x00064>
	  at <RunAsync>d__43:MoveNext <0x00168>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0004a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x00004>
	  at Xunit.Sdk.TestRunner`1:RunAsync <0x00062>
	  at Xunit.Sdk.XunitTestCaseRunner:RunTestAsync <0x00066>
	  at <RunAsync>d__19:MoveNext <0x0014a>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0004a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x00004>
	  at Xunit.Sdk.TestCaseRunner`1:RunAsync <0x00062>
	  at Xunit.Sdk.XunitTestCase:RunAsync <0x00034>
	  at Microsoft.DotNet.XUnitExtensions.SkippedFactTestCase:<>n__0 <0x00016>
	  at <RunAsync>d__2:MoveNext <0x00058>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0004a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x00004>
	  at Microsoft.DotNet.XUnitExtensions.SkippedFactTestCase:RunAsync <0x0007a>
	  at Xunit.Sdk.XunitTestMethodRunner:RunTestCaseAsync <0x0002e>
	  at <RunTestCasesAsync>d__32:MoveNext <0x00080>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0004a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x00004>
	  at Xunit.Sdk.TestMethodRunner`1:RunTestCasesAsync <0x00062>
	  at <RunAsync>d__31:MoveNext <0x00096>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0004a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x00004>
	  at Xunit.Sdk.TestMethodRunner`1:RunAsync <0x00062>
	  at Xunit.Sdk.XunitTestClassRunner:RunTestMethodAsync <0x0003e>
	  at <RunTestMethodsAsync>d__38:MoveNext <0x001fa>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0004a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x00004>
	  at Xunit.Sdk.TestClassRunner`1:RunTestMethodsAsync <0x00062>
	  at <RunAsync>d__37:MoveNext <0x00158>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0004a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x00004>
	  at Xunit.Sdk.TestClassRunner`1:RunAsync <0x00062>
	  at Xunit.Sdk.XunitTestCollectionRunner:RunTestClassAsync <0x00040>
	  at <RunTestClassesAsync>d__28:MoveNext <0x000e6>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0004a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x00004>
	  at Xunit.Sdk.TestCollectionRunner`1:RunTestClassesAsync <0x00062>
	  at <RunAsync>d__27:MoveNext <0x00158>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0004a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x00004>
	  at Xunit.Sdk.TestCollectionRunner`1:RunAsync <0x00062>
	  at Xunit.Sdk.XunitTestAssemblyRunner:RunTestCollectionAsync <0x00032>
	  at <>c__DisplayClass14_2:<RunTestCollectionsAsync>b__2 <0x0003e>
	  at System.Threading.Tasks.Task`1:InnerInvoke <0x0001c>
	  at <>c:<.cctor>b__276_0 <0x00008>
	  at System.Threading.ExecutionContext:RunInternal <0x00088>
	  at System.Threading.Tasks.Task:ExecuteWithThreadLocal <0x00118>
	  at System.Threading.Tasks.Task:ExecuteEntry <0x00062>
	  at <>c:<.cctor>b__8_0 <0x0000c>
	  at Xunit.Sdk.MaxConcurrencySyncContext:RunOnSyncContext <0x0001e>
	  at <>c__DisplayClass11_0:<WorkerThreadProc>b__0 <0x00022>
	  at System.Action`1:invoke_void_T <0x0007a>
	  at System.Threading.ExecutionContext:RunInternal <0x00088>
	  at System.Threading.ExecutionContext:Run <0x0001c>
	  at System.Object:lambda_method2 <0x00012>
	  at Xunit.Sdk.ExecutionContextHelper:Run <0x00022>
	  at Xunit.Sdk.MaxConcurrencySyncContext:WorkerThreadProc <0x000b2>
	  at <>c:<QueueUserWorkItem>b__5_0 <0x00018>
	  at System.Threading.Tasks.Task:InnerInvoke <0x00044>
	  at <>c:<.cctor>b__276_0 <0x00008>
	  at System.Threading.ExecutionContext:RunInternal <0x00088>
	  at System.Threading.Tasks.Task:ExecuteWithThreadLocal <0x00118>
	  at System.Threading.Tasks.Task:ExecuteEntryUnsafe <0x00042>
	  at <>c:<.cctor>b__10_0 <0x0000e>
	  at System.Threading.Thread:StartCallback <0x00094>
	  at System.Object:runtime_invoke_direct_void__this__ <0x0008a>
=================================================================
./RunTests.sh: line 161:    23 Aborted                 "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Net.WebSockets.Client.Tests.runtimeconfig.json --depsfile System.Net.WebSockets.Client.Tests.deps.json xunit.console.dll System.Net.WebSockets.Client.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/root/helix/work/workitem
----- end Wed Jun 3 21:20:14 UTC 2020 ----- exit code 134 ----------------------------------------------------------
exit code 134 means SIGABRT Abort. Managed or native assert, or runtime check such as heap corruption, caused call to abort(). Core dumped.
Looking around for any Linux dump...
... found no dump in /root/helix/work/workitem

cc @steveisok @marek-safar

[Dotnet-GitSync-Bot]

I couldn't figure out the best area label to add to this issue. Please help me learn by adding exactly one area label.

[akoeplinger]

Failures from the last 200 builds:

Build	Kind	Console	Core Dump
683099	PR 37625	console.log
684850	PR 37795	console.log
683850	PR 37718	console.log
685470	PR 36163	console.log

Command:

runfo search-helix -d runtime -c 200 -pr -m -v "runtime_invoke_direct_void__this__"

[ViktorHofer]

@akoeplinger would you mind doing another runfo?

[akoeplinger]

@ViktorHofer I don't get any reports for that string but runfo hit issues on a bunch of builds:

Unable to search helix logs for 698956:  The SSL connection could not be established, see inner exception.
Unable to search helix logs for 699474:  The SSL connection could not be established, see inner exception.
Unable to search helix logs for 699175:  Response status code does not indicate success: 503 (Service Unavailable).
Unable to search helix logs for 698519:  The operation was canceled.
Unable to search helix logs for 698070:  Response status code does not indicate success: 503 (Service Unavailable).
Unable to search helix logs for 697177:  Response status code does not indicate success: 500 (Internal Server Error).
Unable to search helix logs for 696757:  The operation was canceled.
Unable to search helix logs for 694874:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 700110:  The operation was canceled.
Unable to search helix logs for 699555:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 699127:  The SSL connection could not be established, see inner exception.
Unable to search helix logs for 698572:  The operation was canceled.
Unable to search helix logs for 698026:  The SSL connection could not be established, see inner exception.
Unable to search helix logs for 697188:  Response status code does not indicate success: 500 (Internal Server Error).
Unable to search helix logs for 696765:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 700358:  The SSL connection could not be established, see inner exception.
Unable to search helix logs for 700063:  Response status code does not indicate success: 503 (Service Unavailable).
Unable to search helix logs for 699542:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 699105:  The operation was canceled.
Unable to search helix logs for 698500:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 698001:  The operation was canceled.
Unable to search helix logs for 697172:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 696753:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 700407:  The SSL connection could not be established, see inner exception.
Unable to search helix logs for 699516:  The SSL connection could not be established, see inner exception.
Unable to search helix logs for 699111:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 697399:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 697903:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 697166:  The operation was canceled.
Unable to search helix logs for 696736:  The SSL connection could not be established, see inner exception.
Unable to search helix logs for 699510:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 700091:  The SSL connection could not be established, see inner exception.
Unable to search helix logs for 699528:  The SSL connection could not be established, see inner exception.
Unable to search helix logs for 699044:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 698111:  The operation was canceled.
Unable to search helix logs for 697882:  The operation was canceled.
Unable to search helix logs for 697146:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 696697:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 700391:  The operation was canceled.
Unable to search helix logs for 700015:  The operation was canceled.
Unable to search helix logs for 699508:  Response status code does not indicate success: 503 (Service Unavailable).
Unable to search helix logs for 699033:  Response status code does not indicate success: 500 (Internal Server Error).
Unable to search helix logs for 698494:  The SSL connection could not be established, see inner exception.
Unable to search helix logs for 697817:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 697138:  The SSL connection could not be established, see inner exception.
Unable to search helix logs for 696671:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 700346:  The SSL connection could not be established, see inner exception.
Unable to search helix logs for 700033:  Response status code does not indicate success: 500 (Internal Server Error).
Unable to search helix logs for 699536:  The operation was canceled.
Unable to search helix logs for 698969:  The operation was canceled.
Unable to search helix logs for 698485:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 697653:  Response status code does not indicate success: 500 (Internal Server Error).
Unable to search helix logs for 697103:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 696701:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 700279:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 700004:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 699487:  The SSL connection could not be established, see inner exception.
Unable to search helix logs for 698913:  The SSL connection could not be established, see inner exception.
Unable to search helix logs for 698482:  Response status code does not indicate success: 500 (Internal Server Error).
Unable to search helix logs for 697659:  Response status code does not indicate success: 500 (Internal Server Error).
Unable to search helix logs for 697068:  The SSL connection could not be established, see inner exception.
Unable to search helix logs for 696651:  The SSL connection could not be established, see inner exception.
Unable to search helix logs for 700338:  Response status code does not indicate success: 500 (Internal Server Error).
Unable to search helix logs for 699988:  The SSL connection could not be established, see inner exception.
Unable to search helix logs for 699481:  Response status code does not indicate success: 503 (Service Unavailable).
Unable to search helix logs for 698468:  The operation was canceled.
Unable to search helix logs for 697629:  The operation was canceled.
Unable to search helix logs for 697028:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 696640:  The operation was canceled.
Unable to search helix logs for 698977:  The SSL connection could not be established, see inner exception.
Unable to search helix logs for 699995:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 699482:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 698884:  The operation was canceled.
Unable to search helix logs for 698464:  Response status code does not indicate success: 500 (Internal Server Error).
Unable to search helix logs for 697578:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 696986:  Response status code does not indicate success: 500 (Internal Server Error).
Unable to search helix logs for 696421:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 700339:  Response status code does not indicate success: 500 (Internal Server Error).
Unable to search helix logs for 699984:  Response status code does not indicate success: 500 (Internal Server Error).
Unable to search helix logs for 699416:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 698861:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 698561:  The operation was canceled.
Unable to search helix logs for 697005:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 696597:  Response status code does not indicate success: 503 (Service Unavailable).
Unable to search helix logs for 700318:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 699918:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 699449:  The SSL connection could not be established, see inner exception.
Unable to search helix logs for 698699:  Response status code does not indicate success: 500 (Internal Server Error).
Unable to search helix logs for 698412:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 697471:  Response status code does not indicate success: 500 (Internal Server Error).
Unable to search helix logs for 696990:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 696610:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 699910:  Response status code does not indicate success: 503 (Service Unavailable).
Unable to search helix logs for 699470:  Response status code does not indicate success: 503 (Service Unavailable).
Unable to search helix logs for 698810:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 698460:  The SSL connection could not be established, see inner exception.
Unable to search helix logs for 697474:  The SSL connection could not be established, see inner exception.
Unable to search helix logs for 696864:  The SSL connection could not be established, see inner exception.
Unable to search helix logs for 696589:  Response status code does not indicate success: 503 (Service Unavailable).
Unable to search helix logs for 699618:  The SSL connection could not be established, see inner exception.
Unable to search helix logs for 699872:  Response status code does not indicate success: 500 (Internal Server Error).
Unable to search helix logs for 699407:  The SSL connection could not be established, see inner exception.
Unable to search helix logs for 698819:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 698442:  Response status code does not indicate success: 500 (Internal Server Error).
Unable to search helix logs for 697433:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 696563:  The SSL connection could not be established, see inner exception.
Unable to search helix logs for 696568:  The operation was canceled.
Unable to search helix logs for 700290:  Response status code does not indicate success: 500 (Internal Server Error).
Unable to search helix logs for 699878:  Response status code does not indicate success: 503 (Service Unavailable).
Unable to search helix logs for 699401:  The SSL connection could not be established, see inner exception.
Unable to search helix logs for 698783:  Response status code does not indicate success: 503 (Service Unavailable).
Unable to search helix logs for 698398:  The operation was canceled.
Unable to search helix logs for 697426:  The operation was canceled.
Unable to search helix logs for 696968:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 696540:  Response status code does not indicate success: 500 (Internal Server Error).
Unable to search helix logs for 699864:  The operation was canceled.
Unable to search helix logs for 699269:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 696661:  The operation was canceled.
Unable to search helix logs for 698423:  Response status code does not indicate success: 500 (Internal Server Error).
Unable to search helix logs for 697408:  Response status code does not indicate success: 500 (Internal Server Error).
Unable to search helix logs for 696944:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 696534:  The operation was canceled.
Unable to search helix logs for 699761:  The operation was canceled.
Unable to search helix logs for 699851:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 699385:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 697959:  Response status code does not indicate success: 503 (Service Unavailable).
Unable to search helix logs for 697407:  Response status code does not indicate success: 503 (Service Unavailable).
Unable to search helix logs for 696905:  Response status code does not indicate success: 503 (Service Unavailable).
Unable to search helix logs for 696427:  Response status code does not indicate success: 500 (Internal Server Error).
Unable to search helix logs for 700272:  Response status code does not indicate success: 503 (Service Unavailable).
Unable to search helix logs for 699748:  The operation was canceled.
Unable to search helix logs for 699361:  The SSL connection could not be established, see inner exception.
Unable to search helix logs for 698295:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 697392:  The operation was canceled.
Unable to search helix logs for 696926:  The operation was canceled.
Unable to search helix logs for 696364:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 700250:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 699746:  The SSL connection could not be established, see inner exception.
Unable to search helix logs for 697941:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 695883:  The operation was canceled.
Unable to search helix logs for 697358:  The operation was canceled.
Unable to search helix logs for 696892:  The operation was canceled.
Unable to search helix logs for 696508:  The SSL connection could not be established, see inner exception.
Unable to search helix logs for 700208:  The SSL connection could not be established, see inner exception.
Unable to search helix logs for 699730:  The SSL connection could not be established, see inner exception.
Unable to search helix logs for 699019:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 698717:  Response status code does not indicate success: 503 (Service Unavailable).
Unable to search helix logs for 698260:  The operation was canceled.
Unable to search helix logs for 697329:  The SSL connection could not be established, see inner exception.
Unable to search helix logs for 696780:  The SSL connection could not be established, see inner exception.
Unable to search helix logs for 696492:  The operation was canceled.
Unable to search helix logs for 700240:  Response status code does not indicate success: 503 (Service Unavailable).
Unable to search helix logs for 699702:  Response status code does not indicate success: 503 (Service Unavailable).
Unable to search helix logs for 699277:  The operation was canceled.
Unable to search helix logs for 698693:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 698238:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 696844:  The SSL connection could not be established, see inner exception.
Unable to search helix logs for 696880:  The SSL connection could not be established, see inner exception.
Unable to search helix logs for 696476:  The SSL connection could not be established, see inner exception.
Unable to search helix logs for 699696:  Response status code does not indicate success: 500 (Internal Server Error).
Unable to search helix logs for 698681:  Response status code does not indicate success: 503 (Service Unavailable).
Unable to search helix logs for 697534:  The SSL connection could not be established, see inner exception.
Unable to search helix logs for 697270:  Response status code does not indicate success: 503 (Service Unavailable).
Unable to search helix logs for 696868:  Response status code does not indicate success: 500 (Internal Server Error).
Unable to search helix logs for 696465:  The operation was canceled.
Unable to search helix logs for 700222:  The operation was canceled.
Unable to search helix logs for 699587:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 699213:  The SSL connection could not be established, see inner exception.
Unable to search helix logs for 698618:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 698181:  The operation was canceled.
Unable to search helix logs for 697282:  The SSL connection could not be established, see inner exception.
Unable to search helix logs for 696799:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 696450:  The operation was canceled.
Unable to search helix logs for 700129:  The operation was canceled.
Unable to search helix logs for 699605:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 698889:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 698590:  The operation was canceled.
Unable to search helix logs for 698089:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 697273:  The operation was canceled.
Unable to search helix logs for 696805:  The operation was canceled.
Unable to search helix logs for 696435:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 699893:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 699615:  The operation was canceled.
Unable to search helix logs for 691240:  The SSL connection could not be established, see inner exception.
Unable to search helix logs for 698530:  The operation was canceled.
Unable to search helix logs for 697235:  Response status code does not indicate success: 429 ().
Unable to search helix logs for 696918:  The operation was canceled.
Unable to search helix logs for 696432:  Response status code does not indicate success: 429 ().

[ViktorHofer]

ok thanks, cc @jaredpar