slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."

[jeffschwMSFT]

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
2902149	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
1290845	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#65400
1290167	dotnet/maui	Controls Layout	Log
2902006	dotnet-dotnet-monitor	Test MacOS x64 Release
1290710	dotnet/maui	Controls SafeAreaEdges,Shadow	Log
2901965	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
1290600	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#52641
1290551	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#124202
1290549	dotnet/runtime	osx-x64 Release NativeAOT	Log	dotnet/runtime#124202
2901834	dotnet-maui	Controls (v18.4)	Log
1290497	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log
2901774	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
1290248	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#123962
1290307	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log
1290190	dotnet/aspnetcore	Tests: macOS	Log
2901617	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
2901593	dotnet-performance	Performance maui_scenarios_android Default Mono windows 22H2 x64 Galaxy main	Log
2901539	dotnet-maui	Controls Layout	Log
1288586	dotnet/maui	Controls Layout	Log	dotnet/maui#32797
2901483	dotnet-performance	Performance maui_scenarios_android Default CoreCLR_Debug windows 22H2 x64 Pixel main	Log
1289726	dotnet/maui	Controls Layout	Log	dotnet/maui#33951
1289316	dotnet/maui	Controls Layout	Log	dotnet/maui#33826
1289447	dotnet/runtime	ios-arm64 Release AllSubsets_CoreCLR_Smoke	Log	dotnet/runtime#124269
2901253	dotnet-runtime	SuperPMI collect aspnet2 run osx arm64 checked	Log
1289428	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono	Log
2901247	dotnet-maui	Controls Layout	Log
1289303	dotnet/maui	Controls Layout	Log	dotnet/maui#33765
1289354	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#123932
2901232	dotnet-maui	Controls ViewBaseTests,Window	Log
2901226	dotnet-runtime	SuperPMI collect benchmarks run osx arm64 checked	Log
1289278	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#124242
2901198	dotnet-runtime	Performance ios_scenarios iOSCoreCLR Interpreter iOSStripSymbols osx x64 perfiphone12mini net11.0	Log
2901216	dotnet-performance	Performance powershell windows 22H2 arm64 Ampere main	Log
1289248	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log	dotnet/runtime#124220
2901173	dotnet-runtime	coreclr Pri1 Runtime Tests Run osx arm64 checked	Log
2901194	dotnet-performance	Performance maui_scenarios_ios Default Mono osx 15 x64 iPhoneMini12 main	Log
2901189	dotnet-performance	Performance nativeaot_scenarios windows 22H2 arm64 Ampere 8.0	Log
2901136	dotnet-runtime	coreclr Pri1 Runtime Tests Run osx arm64 checked	Log
1289101	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log
1289064	dotnet/maui	Controls Layout	Log	dotnet/maui#33978
2901089	dotnet-sdk	Darwin	Log	#57700
2901088	dotnet-sdk	Darwin	Log	#57702
2900921	dotnet-sdk	AoT: macOS (x64)	Log	#57694
1289018	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#124202
1289016	dotnet/runtime	osx-x64 Release NativeAOT	Log	dotnet/runtime#124202
2900998	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
1288468	dotnet/runtime	ios-arm64 Release AllSubsets_NativeAOT	Log
1288797	dotnet/runtime	Libraries Test Run checked coreclr osx x64 Debug	Log	dotnet/runtime#124025
2900922	dotnet-sdk	TestBuild: macOS (x64)	Log	#57693
2900863	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
1288473	dotnet/runtime	Libraries Test Run release coreclr osx x64 Release	Log
1287966	dotnet/runtime	Libraries Test Run release coreclr osx x64 Debug	Log	dotnet/runtime#124048
1288596	dotnet/aspnetcore	Tests: macOS	Log
2900805	dotnet-sdk	Darwin	Log	#57700
2900804	dotnet-sdk	Darwin	Log	#57702
1288490	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#124202
1288486	dotnet/runtime	osx-x64 Release NativeAOT	Log	dotnet/runtime#124202
2900723	dotnet-performance	Performance bepuphysics ubuntu 2204 arm64 Ampere 9.0	Log
1288471	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
1288402	dotnet/runtime	osx-x64 Release Libraries_Release_CoreCLR	Log
1288117	dotnet/runtime	Libraries Test Run release coreclr osx x64 Debug	Log	dotnet/runtime#124236
1288373	dotnet/runtime	osx-x64 Release Libraries_Release_CoreCLR	Log
1288364	dotnet/runtime	osx-x64 Debug PALTests	Log
1288173	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Minijit_RuntimeTests minijit	Log
1288153	dotnet/runtime	ios-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#124237
1288048	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#124177
2900406	dotnet-runtime	Performance ios_scenarios iOSCoreCLR R2R iOSStripSymbols osx x64 perfiphone12mini net11.0	Log
2900404	dotnet-runtime	Performance ios_scenarios iOSCoreCLR Interpreter iOSStripSymbols osx x64 perfiphone12mini net11.0	Log
1287884	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log
1287814	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#123990
1287808	dotnet/aspnetcore	Tests: macOS	Log
1287764	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Minijit_RuntimeTests minijit	Log	dotnet/runtime#124142
1287560	dotnet/maui	Controls SoftInput,Stepper,Switch,SwipeView	Log	dotnet/maui#33945
1287693	dotnet/maui	Controls CollectionView	Log
1287654	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#123933
1287651	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log	dotnet/runtime#124220
1287531	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log
1287508	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#122638
1287485	dotnet/maui	Controls Layout	Log
1287438	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#123992
1287257	dotnet/maui	Controls Editor,Effects,FlyoutPage,Focus,Fonts,Frame,Gestures,GraphicsView	Log	dotnet/maui#33346
1287230	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#52932
1287352	dotnet/runtime	osx-x64 Debug PALTests	Log
1287350	dotnet/runtime	osx-x64 Release Libraries_Release_CoreCLR	Log
2899869	dotnet-sdk	TestBuild: macOS (x64)	Log	#57694
1287216	dotnet/aspnetcore	Tests: macOS	Log
2899878	dotnet-sdk	Darwin	Log	#57700
2899877	dotnet-sdk	Darwin	Log	#57702
2899870	dotnet-sdk	TestBuild: macOS (x64)	Log	#57714
1287171	dotnet/maui	Controls Layout	Log
1287170	dotnet/maui	Controls Layout	Log
1286665	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log
2899795	dotnet-dotnet-monitor	Test MacOS x64 Release	Log
1286972	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#52792
1286956	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log
2899761	dotnet-sdk	TestBuild: macOS (x64)	Log	#57694
2899760	dotnet-sdk	AoT: macOS (x64)	Log	#57714
1286881	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#65377
2899633	dotnet-maui	Controls (v18.4) SearchBar,Shape,Slider
1286053	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#124179
Displaying 100 of 1045 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
39	213	1045

[lewing]

🤔

[nagilson]

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

[ivanpovazan]

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

• On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

• While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

[garath]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

[garath]

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.

[ivanpovazan]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

We are still experiencing the problem.

Should we try to change the queue to osx.13.arm64?

@dotnet/dnceng

[ilyas1974]

Looks at the failing builds, I'm noticing the jobs that are timing out in the hosted pool (Azure Pipelines) are still using the older hardware. I would recommend moving the workloads to the mac-latest-internal or mac-14-arm64 agent specifications. These have the latest mac hardware associated with them.