Need retry on Http Client timeouts for Helix Arcade SDK (waiting for jobs to finish)

[robertborr]

@dotnet-mc-bot commented on Thu Mar 07 2019

There were a set of failures during this build. Here is a summary of these:

• https://dnceng.visualstudio.com/internal/_build/results?buildId=118107
  • Agent: NetCoreInternal-Int-Pool 2
  • Error log: F:\workspace_work\0\s\src\Microsoft.DotNet.Helix\Sdk\tools\Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(0,0): error : TaskCanceledException: The operation was canceled. [F:\workspace_work\0\s\tests\UnitTests.proj]...

@robertborr

---

@robertborr commented on Fri Mar 08 2019

Build FAILED.

F:\workspace_work\1\s\src\Microsoft.DotNet.Helix\Sdk\tools\Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : TaskCanceledException: The operation was canceled. [F:\workspace_work\1\s\tests\UnitTests.proj]
F:\workspace_work\1\s\src\Microsoft.DotNet.Helix\Sdk\tools\Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken) [F:\workspace_work\1\s\tests\UnitTests.proj]
F:\workspace_work\1\s\src\Microsoft.DotNet.Helix\Sdk\tools\Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) [F:\workspace_work\1\s\tests\UnitTests.proj]
F:\workspace_work\1\s\src\Microsoft.DotNet.Helix\Sdk\tools\Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) [F:\workspace_work\1\s\tests\UnitTests.proj]
F:\workspace_work\1\s\src\Microsoft.DotNet.Helix\Sdk\tools\Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : at Microsoft.Rest.RetryAfterDelegatingHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) [F:\workspace_work\1\s\tests\UnitTests.proj]
F:\workspace_work\1\s\src\Microsoft.DotNet.Helix\Sdk\tools\Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : at Microsoft.Rest.RetryDelegatingHandler.<>c__DisplayClass11_0.<b__1>d.MoveNext() [F:\workspace_work\1\s\tests\UnitTests.proj]
F:\workspace_work\1\s\src\Microsoft.DotNet.Helix\Sdk\tools\Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : --- End of stack trace from previous location where exception was thrown --- [F:\workspace_work\1\s\tests\UnitTests.proj]
F:\workspace_work\1\s\src\Microsoft.DotNet.Helix\Sdk\tools\Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : at Microsoft.Rest.RetryDelegatingHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) [F:\workspace_work\1\s\tests\UnitTests.proj]
F:\workspace_work\1\s\src\Microsoft.DotNet.Helix\Sdk\tools\Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) [F:\workspace\_work\1\s\tests\UnitTests.proj] F:\workspace\_work\1\s\src\Microsoft.DotNet.Helix\Sdk\tools\Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : at Microsoft.DotNet.Helix.Client.Aggregate.JobSummaryInternalAsync(IImmutableList1 groupBy, Int32 maxResultSets, String filterBuild, String filterCreator, String filterName, String filterSource, String filterType, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.Helix/Client/CSharp/generated-code/Aggregate.cs:line 581 [F:\workspace_work\1\s\tests\UnitTests.proj]
F:\workspace_work\1\s\src\Microsoft.DotNet.Helix\Sdk\tools\Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : at Microsoft.DotNet.Helix.Client.Aggregate.JobSummaryAsync(IImmutableList1 groupBy, Int32 maxResultSets, String filterBuild, String filterCreator, String filterName, String filterSource, String filterType, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.Helix/Client/CSharp/generated-code/Aggregate.cs:line 493 [F:\workspace\_work\1\s\tests\UnitTests.proj] F:\workspace\_work\1\s\src\Microsoft.DotNet.Helix\Sdk\tools\Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : at Microsoft.DotNet.Helix.Sdk.CheckHelixJobStatus.MissionControlTestProcessingDoneAsync(String jobName) in /_/src/Microsoft.DotNet.Helix/Sdk/CheckHelixJobStatus.cs:line 68 [F:\workspace\_work\1\s\tests\UnitTests.proj] F:\workspace\_work\1\s\src\Microsoft.DotNet.Helix\Sdk\tools\Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : at Microsoft.DotNet.Helix.Sdk.CheckHelixJobStatus.CheckHelixJobAsync(String jobName) in /_/src/Microsoft.DotNet.Helix/Sdk/CheckHelixJobStatus.cs:line 56 [F:\workspace\_work\1\s\tests\UnitTests.proj] F:\workspace\_work\1\s\src\Microsoft.DotNet.Helix\Sdk\tools\Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : at Microsoft.DotNet.Helix.Sdk.CheckHelixJobStatus.ExecuteCore() in /_/src/Microsoft.DotNet.Helix/Sdk/CheckHelixJobStatus.cs:line 28 [F:\workspace\_work\1\s\tests\UnitTests.proj] F:\workspace\_work\1\s\src\Microsoft.DotNet.Helix\Sdk\tools\Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : at Microsoft.DotNet.Helix.Sdk.HelixTask.Execute() in /_/src/Microsoft.DotNet.Helix/Sdk/HelixTask.cs:line 43 [F:\workspace\_work\1\s\tests\UnitTests.proj] F:\workspace\_work\1\s\src\Microsoft.DotNet.Helix\Sdk\tools\Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : IOException: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request. [F:\workspace\_work\1\s\tests\UnitTests.proj] F:\workspace\_work\1\s\src\Microsoft.DotNet.Helix\Sdk\tools\Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error) [F:\workspace\_work\1\s\tests\UnitTests.proj] F:\workspace\_work\1\s\src\Microsoft.DotNet.Helix\Sdk\tools\Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token) [F:\workspace\_work\1\s\tests\UnitTests.proj] F:\workspace\_work\1\s\src\Microsoft.DotNet.Helix\Sdk\tools\Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : at System.Net.Security.SslStreamInternal.<FillBufferAsync>g__InternalFillBufferAsync|38_0[TReadAdapter](TReadAdapter adap, ValueTask1 task, Int32 min, Int32 initial) [F:\workspace_work\1\s\tests\UnitTests.proj]
F:\workspace_work\1\s\src\Microsoft.DotNet.Helix\Sdk\tools\Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : at System.Net.Security.SslStreamInternal.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer) [F:\workspace_work\1\s\tests\UnitTests.proj]
F:\workspace_work\1\s\src\Microsoft.DotNet.Helix\Sdk\tools\Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken) [F:\workspace_work\1\s\tests\UnitTests.proj]
F:\workspace_work\1\s\src\Microsoft.DotNet.Helix\Sdk\tools\Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : SocketException: The I/O operation has been aborted because of either a thread exit or an application request [F:\workspace_work\1\s\tests\UnitTests.proj]
F:\workspace_work\1\s\src\Microsoft.DotNet.Helix\Sdk\tools\Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : [F:\workspace_work\1\s\tests\UnitTests.proj]
F:\workspace_work\1\s\src\Microsoft.DotNet.Helix\Sdk\tools\Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(67,5): error : [F:\workspace_work\1\s\tests\UnitTests.proj]
0 Warning(s)
1 Error(s)

---

@robertborr commented on Fri Mar 08 2019

@JohnTortugo manually started this build and it appears that the problem is fixed in later builds

---

@robertborr commented on Fri Mar 08 2019

non repeatable networing issue. the component making the request is in generated code and does not properly handle HttpClient timeouts. Since its Arcade code, mark toolserror, and move to arcade

[MattGal]

@markwilkie any reason this is closed with no comment? @safern and @tannergooding are hitting this, and it's a simple fix (catch the timeout exception and retry).

[MattGal]

Reopening to discuss @ triage today

[markwilkie]

@MattGal - my bad sorry...

[markwilkie]

Next steps @MattGal ?

[MattGal]

I was hoping filing issues here was just the backlog for Arcade work. It's not biting us very often, but this sort of problem is very common.

[markwilkie]

ACK

[MattGal]

Helix arcade SDK has retries to all Helix API calls now.