slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."

[jeffschwMSFT]

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
1238805	dotnet/maui	RunOniOS_MauiNativeAOT macOS ($(HostedMacImage))	Log	dotnet/maui#33251
1238803	dotnet/aspnetcore	Test: Blazor E2E tests on Linux	Log	dotnet/aspnetcore#64712
1238762	dotnet/maui	Controls ListView	Log	dotnet/maui#33284
1238719	dotnet/maui	Controls CollectionView	Log	dotnet/maui#33159
1238631	dotnet/maui	Controls CollectionView	Log	dotnet/maui#30009
1238622	dotnet/maui	Controls CollectionView	Log	dotnet/maui#29028
1238681	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log
1238645	dotnet/maui	Controls CollectionView	Log	dotnet/maui#32734
1238648	dotnet/maui	Controls Navigation	Log	dotnet/maui#33134
1238618	dotnet/maui	RunOniOS macOS	Log	dotnet/maui#28474
1238642	dotnet/maui	Controls Layout	Log	dotnet/maui#32349
1238628	dotnet/maui	Controls Layout	Log	dotnet/maui#29550
1238616	dotnet/maui	Controls CollectionView	Log	dotnet/maui#26053
1238604	dotnet/maui	Controls CollectionView	Log	dotnet/maui#33251
1238540	dotnet/maui	Controls Shell	Log	dotnet/maui#29755
1238581	dotnet/maui	Controls CollectionView	Log	dotnet/maui#33251
1238562	dotnet/sdk	TemplateEngine: macOS (x64)	Log	dotnet/sdk#52157
2866377	dotnet-performance	Performance mlnet ubuntu 2204 arm64 Ampere 8.0	Log
1238411	dotnet/maui	Controls CollectionView	Log	dotnet/maui#33278
1238469	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log
1238455	dotnet/maui	Controls CollectionView	Log	dotnet/maui#33263
1238338	dotnet/maui	Controls CollectionView	Log	dotnet/maui#33275
1238363	dotnet/maui	Controls Layout	Log
1238308	dotnet/maui	macOS Unit Tests	Log	dotnet/maui#33271
1238291	dotnet/maui	Controls Layout	Log	dotnet/maui#30242
1238228	dotnet/maui	Controls Layout	Log	dotnet/maui#33228
1238213	dotnet/maui	Controls CollectionView	Log	dotnet/maui#33074
1238184	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log
1238171	dotnet/maui	macOS Unit Tests	Log	dotnet/maui#33185
1238169	dotnet/maui	macOS Unit Tests	Log
1238157	dotnet/maui	Controls RadioButton,RefreshView	Log	dotnet/maui#33086
1238129	dotnet/maui	macOS Unit Tests	Log	dotnet/maui#33257
1238101	dotnet/maui	Controls Layout	Log	dotnet/maui#33251
1237889	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log
1237809	dotnet/maui	Controls Layout	Log	dotnet/maui#33252
1237768	dotnet/maui	macOS Unit Tests	Log	dotnet/maui#31840
1237734	dotnet/maui	macOS Unit Tests	Log
1237709	dotnet/maui	Controls Layout	Log
1237653	dotnet/maui	macOS Unit Tests	Log	dotnet/maui#33257
1237543	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log
1237531	dotnet/maui	Build macOS (Release)	Log	dotnet/maui#33254
1237335	dotnet/sdk	TemplateEngine: macOS (x64)	Log	dotnet/sdk#52157
1237300	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log
1237195	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log
1237105	dotnet/runtime	tvos-arm64 Release AllSubsets_CoreCLR_Smoke	Log
2864498	dotnet-runtime	Performance ios_scenarios iOSCoreCLR Interpreter iOSStripSymbols osx x64 perfiphone12mini net11.0	Log
1236845	dotnet/maui	Controls CollectionView	Log
1236816	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log
1236571	dotnet/maui	Controls CollectionView	Log	dotnet/maui#33246
1236377	dotnet/maui	Controls (v18.4) Navigation	Log
1236428	dotnet/maui	Build Device Tests (CoreCLR)	Log
1236275	dotnet/maui	Build macOS	Log	dotnet/maui#33236
1236276	dotnet/maui	Controls CollectionView	Log	dotnet/maui#33236
1236116	dotnet/maui	Controls CollectionView	Log	dotnet/maui#33091
1236097	dotnet/maui	Controls Editor,Effects,FlyoutPage,Focus,Fonts,Frame,Gestures,GraphicsView	Log	dotnet/maui#33228
1236152	dotnet/runtime	tvos-arm64 Release AllSubsets_CoreCLR_Smoke	Log
1235980	dotnet/sdk	TemplateEngine: macOS (x64)	Log	dotnet/sdk#52157
1236041	dotnet/maui	Controls CollectionView	Log	dotnet/maui#33151
1235945	dotnet/aspnetcore	Test: Blazor E2E tests on Linux	Log	dotnet/aspnetcore#64742
1235681	dotnet/maui	Controls (v18.4) ListView
2863834	dotnet-performance	Performance imagesharp ubuntu 2204 arm64 Ampere main	Log
1235781	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log
1235416	dotnet/maui	Controls CollectionView	Log	dotnet/maui#33215
1231816	dotnet/maui	Run DeviceTests iOS	Log	dotnet/maui#33004
1235189	dotnet/aspnetcore	Test: Blazor E2E tests on Linux	Log	dotnet/aspnetcore#64742
2863474	dotnet-performance	Performance mlnet ubuntu 2204 arm64 Ampere 8.0	Log
1235083	dotnet/runtime	ios-arm64 Release AllSubsets_NativeAOT_Smoke	Log	dotnet/runtime#122090
1234986	dotnet/sdk	TemplateEngine: macOS (x64)	Log	dotnet/sdk#52157
1235041	dotnet/runtime	iossimulator-arm64 Release AllSubsets_Mono_RuntimeTests	Log
1234799	dotnet/sdk	TemplateEngine: macOS (x64)	Log	dotnet/sdk#52157
1234845	dotnet/aspnetcore	Test: Blazor E2E tests on Linux	Log	dotnet/aspnetcore#64742
2863108	dotnet-runtime	Performance ios_scenarios iOSNativeAOT NativeAOT iOSStripSymbols osx x64 perfiphone12mini net11.0	Log
1234598	dotnet/runtime	tvos-arm64 Release AllSubsets_Mono	Log
1234368	dotnet/aspnetcore	Test: Blazor E2E tests on Linux	Log	dotnet/aspnetcore#64742
1234091	dotnet/maui	Controls CollectionView	Log	dotnet/maui#33199
1233977	dotnet/maui	Controls Editor,Effects,FlyoutPage,Focus,Fonts,Frame,Gestures,GraphicsView	Log
1233961	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log
1233937	dotnet/maui	RunOnAndroid macOS	Log	dotnet/maui#31895
1233925	dotnet/maui	macOS Unit Tests	Log	dotnet/maui#33028
1233908	dotnet/maui	Controls Editor,Effects,FlyoutPage,Focus,Fonts,Frame,Gestures,GraphicsView	Log	dotnet/maui#33192
1233710	dotnet/aspnetcore	Test: Blazor E2E tests on Linux	Log	dotnet/aspnetcore#64742
1233658	dotnet/maui	Controls CollectionView	Log	dotnet/maui#33177
2862466	dotnet-runtime	Performance ios_scenarios iOSCoreCLR Interpreter iOSStripSymbols osx x64 perfiphone12mini net11.0	Log
1233488	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
1233381	dotnet/maui	Controls CollectionView	Log	dotnet/maui#33127
1233292	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#51934
2862246	dotnet-runtime	coreclr Pri1 Runtime Tests Run osx arm64 checked	Log
1233092	dotnet/maui	Controls Editor,Effects,FlyoutPage,Focus,Fonts,Frame,Gestures,GraphicsView	Log	dotnet/maui#33183
1232836	dotnet/maui	Controls ScrollView	Log	dotnet/maui#33173
1232841	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
1231402	dotnet/maui	Controls CollectionView	Log	dotnet/maui#32674
1232765	dotnet/maui	Controls Editor,Effects,FlyoutPage,Focus,Fonts,Frame,Gestures,GraphicsView	Log	dotnet/maui#33161
1232697	dotnet/maui	Controls Layout	Log	dotnet/maui#33157
1232261	dotnet/runtime	tvos-arm64 Release AllSubsets_Mono	Log
1232258	dotnet/maui	Controls Shell	Log	dotnet/maui#33168
1231815	dotnet/maui	Controls CollectionView	Log	dotnet/maui#33004
1228398	dotnet/maui	macOS Unit Tests	Log	dotnet/maui#33028
1231163	dotnet/runtime	iossimulator-arm64 Release AllSubsets_Mono_RuntimeTests	Log
1230918	dotnet/maui	Controls Editor,Effects,FlyoutPage,Focus,Fonts,Frame,Gestures,GraphicsView	Log	dotnet/maui#33152
1230448	dotnet/maui	Controls Editor,Effects,FlyoutPage,Focus,Fonts,Frame,Gestures,GraphicsView	Log	dotnet/maui#33144
Displaying 100 of 497 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
20	73	497

[lewing]

🤔

[nagilson]

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

[ivanpovazan]

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

• On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

• While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

[garath]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

[garath]

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.

[ivanpovazan]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

We are still experiencing the problem.

Should we try to change the queue to osx.13.arm64?

@dotnet/dnceng

[ilyas1974]

Looks at the failing builds, I'm noticing the jobs that are timing out in the hosted pool (Azure Pipelines) are still using the older hardware. I would recommend moving the workloads to the mac-latest-internal or mac-14-arm64 agent specifications. These have the latest mac hardware associated with them.