Helix submission failing with ArgumentException: Provided Job List Uri is not accessible

[ericstj]

Installing Microsoft.DotNet.Arcade.Sdk 1.0.0-beta.19171.6.
  Zipping directory "D:\a\1\s\artifacts\bin\testhost\netcoreapp-Windows_NT-Debug-x64\" to "D:\a\1\s\artifacts\helix\runtime\test-runtime-netcoreapp-Windows_NT-Debug-x64.zip".
  Results will be available from https://mc.dot.net/#/user/dotnet-bot/pr~2Fdotnet~2Fcorefx~2Frefs~2Fpull~2F36369~2Fmerge/test~2Ffunctional~2Fcli~2F/20190326.13
  Sent Helix Job 2a1ded34-f2c3-485d-a499-8c3bfaf1529c
C:\Users\VssAdministrator\.nuget\packages\microsoft.dotnet.helix.sdk\2.0.0-beta.19171.6\tools\Microsoft.DotNet.Helix.Sdk.MonoQueue.targets(51,5): error : ArgumentException: Provided Job List Uri is not accessible [D:\a\1\s\eng\sendtohelix.proj]
C:\Users\VssAdministrator\.nuget\packages\microsoft.dotnet.helix.sdk\2.0.0-beta.19171.6\tools\Microsoft.DotNet.Helix.Sdk.MonoQueue.targets(51,5): error :    at Microsoft.DotNet.Helix.Client.HelixApi.HandleFailedRequest(RestApiException ex) in /_/src/Microsoft.DotNet.Helix/Client/CSharp/HelixApi.cs:line 29 [D:\a\1\s\eng\sendtohelix.proj]
C:\Users\VssAdministrator\.nuget\packages\microsoft.dotnet.helix.sdk\2.0.0-beta.19171.6\tools\Microsoft.DotNet.Helix.Sdk.MonoQueue.targets(51,5): error :    at Microsoft.DotNet.Helix.Client.Job.NewInternalAsync(JobCreationRequest body, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.Helix/Client/CSharp/generated-code/Job.cs:line 259 [D:\a\1\s\eng\sendtohelix.proj]
C:\Users\VssAdministrator\.nuget\packages\microsoft.dotnet.helix.sdk\2.0.0-beta.19171.6\tools\Microsoft.DotNet.Helix.Sdk.MonoQueue.targets(51,5): error :    at Microsoft.DotNet.Helix.Client.Job.NewAsync(JobCreationRequest body, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.Helix/Client/CSharp/generated-code/Job.cs:line 190 [D:\a\1\s\eng\sendtohelix.proj]
C:\Users\VssAdministrator\.nuget\packages\microsoft.dotnet.helix.sdk\2.0.0-beta.19171.6\tools\Microsoft.DotNet.Helix.Sdk.MonoQueue.targets(51,5): error :    at Microsoft.DotNet.Helix.Client.HelixApi.RetryAsync[T](Func`1 function, Action`1 logRetry, Func`2 isRetryable) [D:\a\1\s\eng\sendtohelix.proj]
C:\Users\VssAdministrator\.nuget\packages\microsoft.dotnet.helix.sdk\2.0.0-beta.19171.6\tools\Microsoft.DotNet.Helix.Sdk.MonoQueue.targets(51,5): error :    at Microsoft.DotNet.Helix.Client.JobDefinition.SendAsync(Action`1 log) in /_/src/Microsoft.DotNet.Helix/JobSender/JobDefinition.cs:line 206 [D:\a\1\s\eng\sendtohelix.proj]
C:\Users\VssAdministrator\.nuget\packages\microsoft.dotnet.helix.sdk\2.0.0-beta.19171.6\tools\Microsoft.DotNet.Helix.Sdk.MonoQueue.targets(51,5): error :    at Microsoft.DotNet.Helix.Sdk.SendHelixJob.ExecuteCore() in /_/src/Microsoft.DotNet.Helix/Sdk/SendHelixJob.cs:line 234 [D:\a\1\s\eng\sendtohelix.proj]
C:\Users\VssAdministrator\.nuget\packages\microsoft.dotnet.helix.sdk\2.0.0-beta.19171.6\tools\Microsoft.DotNet.Helix.Sdk.MonoQueue.targets(51,5): error :    at Microsoft.DotNet.Helix.Sdk.HelixTask.Execute() in /_/src/Microsoft.DotNet.Helix/Sdk/HelixTask.cs:line 43 [D:\a\1\s\eng\sendtohelix.proj]
C:\Users\VssAdministrator\.nuget\packages\microsoft.dotnet.helix.sdk\2.0.0-beta.19171.6\tools\Microsoft.DotNet.Helix.Sdk.MonoQueue.targets(51,5): error : RestApiException: The response contained an invalid status code 400 Bad Request [D:\a\1\s\eng\sendtohelix.proj]
C:\Users\VssAdministrator\.nuget\packages\microsoft.dotnet.helix.sdk\2.0.0-beta.19171.6\tools\Microsoft.DotNet.Helix.Sdk.MonoQueue.targets(51,5): error :  [D:\a\1\s\eng\sendtohelix.proj]

Hitting this on multiple PRs in CoreFx:
dotnet/corefx#36358 (comment)
https://dev.azure.com/dnceng/public/_build/results?buildId=133711&view=logs&jobId=29e7c093-c33b-5b8e-a00b-0e954875c839&taskId=f1035801-25f3-595c-4079-a03464e8f0ad&lineStart=18&lineEnd=31&colStart=1&colEnd=187

Appears to be a server side regression?

/cc @alexperovich @safern @MattGal

[MattGal]

We're having a separate problem from this right now, but I don't think this is a server-side regression (the code that checks this hasn't changed in years) rather this is likely caused by mis-generation of the job list URI to have an excessively short timeout.

[ericstj]

I see. My assumption about server side was that things started failing with us unrelated to any update to client package, so really just a guess.

[MattGal]

It's very curious too, since the job list for your builds are actually publicly accessible too (so mis-generated SAS is not likely). I think it makes the most sense to get the other issue resolved (https://github.com/dotnet/core-eng/issues/5662) before going deeper on this though. That's rolling out shortly.

[MattGal]

that issue is resolved, if you see this going forward we're going to need to dig deeper since the binlogs from the builds you sent do show valid parse-able job lists.

[markwilkie]

Closing per @MattGal comment