Reduce Apple mobile CI matrix by kotlarmilos · Pull Request #125441 · dotnet/runtime

kotlarmilos · 2026-03-11T13:52:24Z

Reduce the Apple mobile CI test matrix from 47 → 32 jobs (saving 15 jobs) to mitigate macOS queue congestion and align with the default scenarios.

What runs where

`runtime.yml` — ios + maccatalyst arm64 smoke (6 jobs)

Platform	Runtime	Test type	Config
ios_arm64	Mono	Library smoke	Release
ios_arm64	NativeAOT	Library smoke	Release
ios_arm64	CoreCLR	Library smoke	Release
maccatalyst_arm64	Mono	Library smoke	Release
maccatalyst_arm64	NativeAOT	Library smoke	Release
maccatalyst_arm64	CoreCLR	Library smoke	Release

`runtime-extra-platforms-ioslike.yml` — tvos + ios Debug smoke (7 jobs)

Platform	Runtime	Test type	Config
tvos_arm64	Mono	Library tests	Release
tvos_arm64	Mono	Runtime tests	Release
tvos_arm64	NativeAOT	Library smoke	Release
tvos_arm64	CoreCLR	Library tests	Release
tvos_arm64	CoreCLR	Runtime tests	Release
ios_arm64, tvos_arm64	CoreCLR	Library smoke	Debug

`runtime-extra-platforms-ioslikesimulator.yml` — iossimulator Debug + Release smoke (8 jobs)

Platform	Runtime	Test type	Config
iossimulator_x64	Mono	Library tests	Debug
iossimulator_x64, iossimulator_arm64	Mono	Runtime tests	Debug
iossimulator_x64	NativeAOT	Library smoke	Debug
iossimulator_x64	CoreCLR	Library tests	Debug
iossimulator_x64, iossimulator_arm64	CoreCLR	Runtime tests	Debug
iossimulator_arm64	CoreCLR	Library smoke	Release

`runtime-extra-platforms-maccatalyst.yml` — maccatalyst full + Debug smoke (11 jobs)

Platform	Runtime	Test type	Config
maccatalyst_x64, maccatalyst_arm64	Mono	Library tests	Release
maccatalyst_arm64	Mono	Runtime tests	Release
maccatalyst_arm64	Mono	Library smoke (AppSandbox)	Release
maccatalyst_arm64	NativeAOT	Library smoke	Release
maccatalyst_arm64	NativeAOT	Library smoke (AppSandbox)	Release
maccatalyst_x64, maccatalyst_arm64	CoreCLR	Library tests	Release
maccatalyst_arm64	CoreCLR	Runtime tests	Release
maccatalyst_arm64	CoreCLR	Library smoke (AppSandbox)	Release
maccatalyst_arm64	CoreCLR	Library smoke	Debug

Other changes

Re-enabled 3 tvOS test suites that were previously excluded due to mysterious crashes (System.IO.Hashing.Tests, System.IO.MemoryMappedFiles.Tests, System.Runtime.Numerics.Tests)
Entries reordered consistently: Mono → NativeAOT → CoreCLR, library → runtime within each group

dotnet-policy-service · 2026-03-11T13:54:03Z

Tagging subscribers to this area: @akoeplinger, @matouskozak, @simonrozsival
See info in area-owners.md if you want to be subscribed.

Copilot

Pull request overview

Reduces macOS CI load by shrinking the Apple mobile test matrix (iOS/tvOS simulators/devices and MacCatalyst), aiming to keep coverage focused on common developer/user scenarios while removing redundant jobs.

Changes:

runtime.yml: drop tvOS device from mobile smoke jobs and remove the MacCatalyst Mono smoke entry.
runtime-extra-platforms-ioslike.yml: shift iOS-like device coverage to tvOS-only.
runtime-extra-platforms-ioslikesimulator.yml: focus simulator coverage on iOS simulator only, switch to Debug, and remove NativeAOT simulator runtime tests.
runtime-extra-platforms-maccatalyst.yml: remove MacCatalyst x64 coverage (arm64 only).

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 7 comments.

File	Description
eng/pipelines/runtime.yml	Removes tvOS device and MacCatalyst smoke jobs from the default runtime pipeline’s Apple-mobile matrix.
eng/pipelines/extra-platforms/runtime-extra-platforms-ioslike.yml	Removes iOS device runs so iOS-like device testing is tvOS-only in extra-platforms.
eng/pipelines/extra-platforms/runtime-extra-platforms-ioslikesimulator.yml	Narrows simulator jobs to iOS simulator, switches to Debug, and drops NativeAOT simulator runtime tests.
eng/pipelines/extra-platforms/runtime-extra-platforms-maccatalyst.yml	Drops MacCatalyst x64 from extra-platforms, leaving arm64 only.

You can also share your feedback on Copilot code review. Take the survey.

eng/pipelines/extra-platforms/runtime-extra-platforms-ioslikesimulator.yml

eng/pipelines/extra-platforms/runtime-extra-platforms-maccatalyst.yml

eng/pipelines/extra-platforms/runtime-extra-platforms-ioslike.yml

eng/pipelines/runtime.yml

vitek-karas · 2026-03-11T14:04:28Z

I didn't look at the existing coverage, so maybe this is already covered:

I think we should have at least some basic testing of Release builds running on simulators (simulator is different enough platform from iOS, such that it warrants at least basic coverage)
Similarly we should have at least basic coverage for running Debug builds on real devices (while not as common, it IS a scenario for users) - this should include both iOS and tvOS devices.
macCatalyst - we should have some coverage for both Debug and Release (since both are scenarios for that target)

Reduce the Apple mobile test matrix from 47 to 24 jobs (-23 per run), restructured around real user scenarios: - runtime.yml: iOS-only smoke tests (8 -> 3 jobs) Remove tvos_arm64 from smoke tests, remove maccatalyst Mono_Smoke - ioslike.yml: tvOS devices only in Release (12 -> 6 jobs) Remove ios_arm64 (covered by runtime.yml smoke) - ioslikesimulator.yml: iossimulator only in Debug (15 -> 8 jobs) Remove tvossimulator, change Release to Debug, drop NativeAOT - maccatalyst.yml: arm64 only (12 -> 7 jobs) Remove maccatalyst_x64 (Intel Macs EOL) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

matouskozak · 2026-03-12T08:49:09Z

eng/pipelines/extra-platforms/runtime-extra-platforms-ioslike.yml

    isExtraPlatformsBuild: ${{ parameters.isExtraPlatformsBuild }}
    isiOSLikeOnlyBuild: ${{ parameters.isiOSLikeOnlyBuild }}
    platforms:
-      - ios_arm64


I'm almost certain that we run more tests in extra-platforms on iOS-arm64 than we do in the runtime pipeline (smoky-only). Do you think that tvOS provides sufficient parity with iOS so we're fine with dropping this?

In extra-platforms ios-arm64, the following test suites run: System.IO.Hashing.Tests, System.IO.MemoryMappedFiles.Tests, and System.Runtime.Numerics.Tests. These tests were previously crashing on tvOS. I assume this was an infrastructure issue rather than something platform-specific, so let's try enabling them on tvOS again.

eng/pipelines/extra-platforms/runtime-extra-platforms-maccatalyst.yml

eng/pipelines/runtime.yml

…urations for clarity

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

You can also share your feedback on Copilot code review. Take the survey.

… exclusions from tests

…ous crashes on tvOS

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

You can also share your feedback on Copilot code review. Take the survey.

eng/pipelines/extra-platforms/runtime-extra-platforms-maccatalyst.yml

src/libraries/tests.proj

- Restore MacCatalyst Mono + CoreCLR + NativeAOT smoke in runtime.yml - Add NativeAOT library smoke to ioslikesimulator and maccatalyst - Replace NativeAOT runtime tests with library tests in maccatalyst - Add CoreCLR AppSandbox library tests for maccatalyst - Change maccatalyst CoreCLR x64+arm64 from smoke to full library tests - Use x64 only for library tests in ioslikesimulator - Reorder entries: Mono > NativeAOT > CoreCLR, library > runtime Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…uce-apple-ci-matrix # Conflicts: # eng/pipelines/extra-platforms/runtime-extra-platforms-ioslike.yml # eng/pipelines/extra-platforms/runtime-extra-platforms-maccatalyst.yml

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

You can also share your feedback on Copilot code review. Take the survey.

Copilot · 2026-03-12T11:16:02Z

eng/pipelines/extra-platforms/runtime-extra-platforms-maccatalyst.yml

+      - maccatalyst_x64
+      - maccatalyst_arm64


This new AllSubsets_CoreCLR job includes maccatalyst_x64, which contradicts the PR description rationale that "Intel Macs are EOL; arm64 is the sole active architecture" and the stated goal of removing maccatalyst_x64 from entries. All other modified entries in this file have x64 removed. If x64 coverage is intentionally kept here for CoreCLR full library tests, the PR description should be updated to reflect this. Otherwise, consider removing maccatalyst_x64 from this entry as well.

Copilot · 2026-03-12T11:16:02Z

eng/pipelines/runtime.yml

+                    eq(variables['librariesContainsChange'], true),
+                    eq(variables['coreclrContainsChange'], true),
+                    eq(variables['isRollingBuild'], true))


The indentation of eq() calls inside the condition block is inconsistent with the surrounding pattern. Other condition blocks in this file (e.g., lines 1303-1306) indent the eq() calls inside or() with two extra spaces. Here, the eq() calls are at the same indentation level as or(. While this doesn't affect functionality (it's inside a >- scalar), it would be more consistent to indent them.

Suggested change

eq(variables['librariesContainsChange'], true),

eq(variables['coreclrContainsChange'], true),

eq(variables['isRollingBuild'], true))

eq(variables['librariesContainsChange'], true),

eq(variables['coreclrContainsChange'], true),

eq(variables['isRollingBuild'], true))

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

You can also share your feedback on Copilot code review. Take the survey.

src/libraries/tests.proj

@@ -249,14 +249,6 @@
    <ProjectExclusions Include="$(MSBuildThisFileDirectory)System.Runtime/tests/System.Reflection.Tests/System.Reflection.Tests.csproj" />


kotlarmilos · 2026-03-13T11:28:27Z

/azp run runtime-ioslike

azure-pipelines · 2026-03-13T11:28:38Z

Azure Pipelines successfully started running 1 pipeline(s).

Copilot AI review requested due to automatic review settings March 11, 2026 13:52

dotnet-policy-service bot assigned kotlarmilos Mar 11, 2026

github-actions bot added the area-Infrastructure-mono label Mar 11, 2026

Copilot started reviewing on behalf of kotlarmilos March 11, 2026 13:53 View session

kotlarmilos requested review from akoeplinger, matouskozak and vitek-karas March 11, 2026 13:54

kotlarmilos changed the title ~~Reduce Apple mobile CI matrix to alleviate macOS queue congestion~~ Reduce Apple mobile CI matrix Mar 11, 2026

Copilot AI reviewed Mar 11, 2026

View reviewed changes

kotlarmilos force-pushed the reduce-apple-ci-matrix branch from 88bd5b0 to ce99125 Compare March 11, 2026 14:48

This was referenced Mar 11, 2026

Cannot find 'arm64-v8a' device dotnet/dnceng#2284

Open

[wasm] OpenQA.Selenium.WebDriverTimeoutException: timeout: Timed out receiving message from renderer #117486

Open

XHarness DevToolsActivePort file doesn't exist #124438

Closed

matouskozak reviewed Mar 12, 2026

View reviewed changes

Rename build parameters for iOS/tvOS simulator and MacCatalyst config…

25affcc

…urations for clarity

Copilot AI review requested due to automatic review settings March 12, 2026 10:01

Copilot started reviewing on behalf of kotlarmilos March 12, 2026 10:03 View session

Copilot AI reviewed Mar 12, 2026

View reviewed changes

kotlarmilos added 2 commits March 12, 2026 11:15

Refactor MacCatalyst CI configurations and remove unnecessary project…

f0439a4

… exclusions from tests

Remove project exclusion for System.Formats.Cbor.Tests due to mysteri…

0bfee88

…ous crashes on tvOS

Copilot AI review requested due to automatic review settings March 12, 2026 10:16

Copilot started reviewing on behalf of kotlarmilos March 12, 2026 10:17 View session

Copilot AI reviewed Mar 12, 2026

View reviewed changes

eng/pipelines/extra-platforms/runtime-extra-platforms-maccatalyst.yml Outdated Show resolved Hide resolved

src/libraries/tests.proj Show resolved Hide resolved

kotlarmilos and others added 4 commits March 12, 2026 11:24

Merge branch 'main' into reduce-apple-ci-matrix

8607492

Add maccatalyst_x64 platform to MacCatalyst CI jobs

31bac4d

Merge remote-tracking branch 'origin/reduce-apple-ci-matrix' into red…

bedaf5d

…uce-apple-ci-matrix # Conflicts: # eng/pipelines/extra-platforms/runtime-extra-platforms-ioslike.yml # eng/pipelines/extra-platforms/runtime-extra-platforms-maccatalyst.yml

Copilot AI review requested due to automatic review settings March 12, 2026 11:07

Copilot started reviewing on behalf of kotlarmilos March 12, 2026 11:08 View session

Copilot AI reviewed Mar 12, 2026

View reviewed changes

This was referenced Mar 12, 2026

iOS.Device test WorkItemExecutions #122874

Open

XHarness package install failure on iOS due to devicectl NSPOSIXErrorDomain error 49 #123796

Open

MsQuic fails with QUIC_STATUS_OUT_OF_MEMORY on AzureLinux #123216

Open

kotlarmilos added 2 commits March 13, 2026 08:55

Merge branch 'main' into reduce-apple-ci-matrix

caea0f5

Add support for ArchiveTests in maccatalyst with native AOT runtime

580f048

Copilot AI review requested due to automatic review settings March 13, 2026 11:21

Copilot started reviewing on behalf of kotlarmilos March 13, 2026 11:22 View session

Copilot AI reviewed Mar 13, 2026

View reviewed changes

src/libraries/tests.proj

@@ -249,14 +249,6 @@

<ProjectExclusions Include="$(MSBuildThisFileDirectory)System.Runtime/tests/System.Reflection.Tests/System.Reflection.Tests.csproj" />

This was referenced Mar 13, 2026

"We stopped hearing from agent Azure Pipelines 32. Verify the agent machine is running and has a healthy network connection" dotnet/dnceng#1886

Open

tracing/runtimeeventsource/nativeruntimeeventsource/nativeruntimeeventsource failing in CI #90605

Open

		@@ -249,14 +249,6 @@
		<ProjectExclusions Include="$(MSBuildThisFileDirectory)System.Runtime/tests/System.Reflection.Tests/System.Reflection.Tests.csproj" />

Conversation

kotlarmilos commented Mar 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What runs where

runtime.yml — ios + maccatalyst arm64 smoke (6 jobs)

runtime-extra-platforms-ioslike.yml — tvos + ios Debug smoke (7 jobs)

runtime-extra-platforms-ioslikesimulator.yml — iossimulator Debug + Release smoke (8 jobs)

runtime-extra-platforms-maccatalyst.yml — maccatalyst full + Debug smoke (11 jobs)

Other changes

Uh oh!

dotnet-policy-service bot commented Mar 11, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vitek-karas commented Mar 11, 2026

Uh oh!

matouskozak Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

kotlarmilos Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

kotlarmilos commented Mar 13, 2026

Uh oh!

azure-pipelines bot commented Mar 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kotlarmilos commented Mar 11, 2026 •

edited

Loading

`runtime.yml` — ios + maccatalyst arm64 smoke (6 jobs)

`runtime-extra-platforms-ioslike.yml` — tvos + ios Debug smoke (7 jobs)

`runtime-extra-platforms-ioslikesimulator.yml` — iossimulator Debug + Release smoke (8 jobs)

`runtime-extra-platforms-maccatalyst.yml` — maccatalyst full + Debug smoke (11 jobs)