slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."

[jeffschwMSFT]

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Repository	Step Name	Console log	Pull Request
1505059	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#67248
1505026	dotnet/aspnetcore	Tests: macOS	Log
1505023	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#66901
1504972	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR	Log
1504894	dotnet/aspnetcore	Tests: macOS	Log
1504858	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#130558
3020258	dotnet-maui	RunOniOS_BlazorRelease x64	Log
1504795	dotnet/aspnetcore	Tests: macOS	Log
1504782	dotnet/aspnetcore	Tests: macOS	Log
1504725	dotnet/maui	Controls Layout	Log	dotnet/maui#35606
1504722	dotnet/maui	Controls CollectionView	Log	dotnet/maui#36429
1504716	dotnet/maui	Controls Shell	Log	dotnet/maui#36511
1504701	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#129992
1504668	dotnet/runtime	Libraries Test Run release coreclr osx arm64 Debug	Log	dotnet/runtime#130393
1504673	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#130156
1504678	dotnet/aspnetcore	Tests: macOS	Log
3020113	dotnet-performance	Performance mlnet windows 22H2 arm64 Ampere main	Log
1504581	dotnet/aspnetcore	Tests: macOS	Log
1504549	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Minijit_RuntimeTests minijit	Log	dotnet/runtime#130478
1504535	dotnet/runtime	coreclr Pri1 Runtime Tests Run R2R_CG2 osx x64 checked	Log
1504532	dotnet/maui	Controls CollectionView	Log
1504533	dotnet/runtime	osx-x64 Release NativeAOT_Libs	Log
1504508	dotnet/maui	Controls CollectionView	Log	dotnet/maui#36121
1504491	dotnet/maui	Controls CollectionView	Log
1504438	dotnet/roslyn	Build_macOS_Debug	Log
1504433	dotnet/aspnetcore	Tests: macOS	Log
1504386	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR	Log
1504309	dotnet/maui	Controls CollectionView	Log
1504303	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#67736
1504275	dotnet/runtime	osx-x64 release CoreCLR_Libraries	Log	dotnet/runtime#130505
1504048	dotnet/runtime	Libraries Test Run checked coreclr osx arm64 Debug	Log	dotnet/runtime#129457
1504241	dotnet/aspnetcore	Tests: macOS	Log
1504021	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#130156
1504144	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#130526
1504123	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#129543
1504112	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#130332
1504009	dotnet/maui	Controls CollectionView	Log	dotnet/maui#36423
1503764	dotnet/runtime	Libraries Test Run checked coreclr osx arm64 Debug	Log	dotnet/runtime#130272
1503964	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#129634
1503879	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#130525
1503806	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#130513
1503732	dotnet/maui	Controls Shell	Log
1503758	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#130446
1503538	dotnet/runtime	osx-arm64 Debug Installer_Build_And_Test	Log	dotnet/runtime#130499
1503683	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#130373
1503609	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#130504
3019507	dotnet-maui	Controls Layout	Log
1503598	dotnet/aspnetcore	Tests: macOS	Log
3019513	dotnet-runtime	Performance ios_scenarios iOSCoreCLR R2R iOSStripSymbols osx x64 perfiphone17 net11.0	Log
3019488	dotnet-maui	RunOniOS_MauiReleaseTrimFull_CoreCLR ARM64	Log
1503351	dotnet/runtime	osx-arm64 release Installer_Build_And_Test	Log	dotnet/runtime#130420
3019433	dotnet-maui	RunOniOS_MauiNativeAOT x64	Log
1503537	dotnet/runtime	tvos-arm64 Release AllSubsets_CoreCLR	Log	dotnet/runtime#130469
1503536	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR	Log	dotnet/runtime#130468
1503556	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#55202
1503524	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#130468
3019430	dotnet-maui	RunOniOS_MauiRelease ARM64	Log
1503495	dotnet/maui	Controls CollectionView	Log
3019460	dotnet-maui	Controls ScrollView	Log
3019450	dotnet-maui	Controls Shell	Log
1503500	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Minijit_RuntimeTests minijit	Log	dotnet/runtime#130481
1503510	dotnet/aspnetcore	Tests: macOS	Log
1503490	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#67733
1503478	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#67324
1501899	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#126367
1503362	dotnet/maui	Controls Layout	Log
3019347	dotnet-maui	RunOniOS_MauiRelease x64	Log
3019339	dotnet-runtime	Libraries Test Run release mono osx x64 Debug	Log	#62798
1503266	dotnet/maui	Controls Image,ImageButton,IndicatorView,InputTransparent,IsEnabled,IsVisible	Log	dotnet/maui#36328
1503290	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#130444
3019331	dotnet-runtime	ios-arm64 Release AllSubsets_NativeAOT	Log	#62802
1503232	dotnet/aspnetcore	Tests: macOS	Log
1503219	dotnet/runtime	osx-arm64 checked	Log
3019312	dotnet-runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	#62799
1502840	dotnet/runtime	browser-wasm osx Release LibraryTests_SmokeMac	Log	dotnet/runtime#130122
1503073	dotnet/runtime	Libraries Test Run checked coreclr osx arm64 Debug	Log	dotnet/runtime#130474
3019269	dotnet-maui	RunOniOS_MauiNativeAOT x64	Log
1503182	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#67679
1503023	dotnet/runtime	osx-x64 release Installer_Build_And_Test	Log	dotnet/runtime#130472
3019255	dotnet-maui	RunOniOS_BlazorRelease ARM64	Log
1503169	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log	dotnet/runtime#130421
3019286	dotnet-performance	Performance roslyn ubuntu 2204 arm64 Ampere main	Log
1503159	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#130482
1503132	dotnet/runtime	ios-arm64 Release AllSubsets_CoreCLR_Smoke	Log	dotnet/runtime#130478
3019237	dotnet-maui	RunOniOS_MauiNativeAOT x64	Log
1502987	dotnet/runtime	osx-x64 release CdacDumpTest	Log	dotnet/runtime#128436
3019232	dotnet-maui	RunOniOS_MauiDebug x64	Log
1503095	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#130477
3019238	dotnet-maui	Controls Shell	Log
1503077	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#130476
1503045	dotnet/maui	Controls Shell	Log
3019233	dotnet-maui	Controls CollectionView	Log
1503036	dotnet/maui	Controls Layout	Log
1503048	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#130469
1502995	dotnet/maui	Controls Shell	Log	dotnet/maui#31755
1503010	dotnet/aspnetcore	Tests: macOS	Log
1502967	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#130467
1502952	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#130462
3019200	dotnet-runtime	ios-arm64 Release AllSubsets_NativeAOT	Log	#62798
1501416	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#130361
Displaying 100 of 1316 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
7	347	1316

[lewing]

🤔

[nagilson]

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

[ivanpovazan]

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

• On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

• While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

[garath]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

[garath]

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.

[ivanpovazan]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

We are still experiencing the problem.

Should we try to change the queue to osx.13.arm64?

@dotnet/dnceng

[ilyas1974]

Looks at the failing builds, I'm noticing the jobs that are timing out in the hosted pool (Azure Pipelines) are still using the older hardware. I would recommend moving the workloads to the mac-latest-internal or mac-14-arm64 agent specifications. These have the latest mac hardware associated with them.