From f37f00f0a502dc34eead9dd5c824db365558c95c Mon Sep 17 00:00:00 2001 From: Andy Gocke Date: Thu, 2 Oct 2025 15:48:55 -0700 Subject: [PATCH 1/9] Define platform testing policy --- docs/infra/test-platforms.md | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100644 docs/infra/test-platforms.md diff --git a/docs/infra/test-platforms.md b/docs/infra/test-platforms.md new file mode 100644 index 00000000000000..54b00f2aa42fb6 --- /dev/null +++ b/docs/infra/test-platforms.md @@ -0,0 +1,36 @@ + +# Platform Testing Organization + +## Overview + +.NET ships on many platforms. Testing all platforms all the time is too expensive. This document defines the testing policy for each branch of the product. + +## Policy + +We want to mix and match platform versions and .NET versions to produce good platform coverage without too much a cost. This means we want to catch breaks on each platform as quickly as possible, and prioritize catching the type of platform breaks that are most likely to affect the specific version being tested. + +* `main` - PRs run on the *latest supported* platform. +* `servicing` - PRs run on the *oldest supported* platform. + +The above policy only applies to PRs. Scheduled or incidental runs can be queued against other platform definitions, if deemed necessary. + +## Details + +### Platform version + +There are two platform versions we want to test on, depending on the .NET support lifecycle: + +1. Latest supported +2. Oldest supported + +We assume that all supported platform versions in between have sufficient coverage based on the latest and the oldest. We currently have no defined strategy for pre-release versions. + +### .NET lifecycle + +We currently regularly maintain three .NET versions: + +1. `main`, i.e. the next release +2. The previous release +3. The release before the previous release + +The last two versions (which have already been released) are called "servicing" releases. From 3206c037f6406fd2d933abe08527609d52d3631e Mon Sep 17 00:00:00 2001 From: Andy Gocke Date: Thu, 2 Oct 2025 15:54:00 -0700 Subject: [PATCH 2/9] Update docs/infra/test-platforms.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- docs/infra/test-platforms.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/infra/test-platforms.md b/docs/infra/test-platforms.md index 54b00f2aa42fb6..e2a06d2cd70e4c 100644 --- a/docs/infra/test-platforms.md +++ b/docs/infra/test-platforms.md @@ -7,7 +7,7 @@ ## Policy -We want to mix and match platform versions and .NET versions to produce good platform coverage without too much a cost. This means we want to catch breaks on each platform as quickly as possible, and prioritize catching the type of platform breaks that are most likely to affect the specific version being tested. +We want to mix and match platform versions and .NET versions to produce good platform coverage without too much cost. This means we want to catch breaks on each platform as quickly as possible, and prioritize catching the type of platform breaks that are most likely to affect the specific version being tested. * `main` - PRs run on the *latest supported* platform. * `servicing` - PRs run on the *oldest supported* platform. From cc1672463102b328db53228874691a3f6497ad20 Mon Sep 17 00:00:00 2001 From: Andy Gocke Date: Fri, 3 Oct 2025 10:35:14 -0700 Subject: [PATCH 3/9] Enhance platform definition and testing policy Expanded the definition of a platform to include architectures, OSes, OS flavors, and crypto stacks. Updated the testing policy to clarify versioning strategy for platform coverage. --- docs/infra/test-platforms.md | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/docs/infra/test-platforms.md b/docs/infra/test-platforms.md index e2a06d2cd70e4c..164ba738e744e5 100644 --- a/docs/infra/test-platforms.md +++ b/docs/infra/test-platforms.md @@ -3,10 +3,21 @@ ## Overview -.NET ships on many platforms. Testing all platforms all the time is too expensive. This document defines the testing policy for each branch of the product. +.NET ships on many platforms. For our purposes, a platform consists of a combination of: + +* Architectures (x86, arm64, et al.) +* OSes, including libc (Windows, Linux, Linux-musl, Browser, et al.) +* OS flavors (Windows Client, Windows Server, Windows Nano, Azure Linux, Ubuntu, Debian, Alpine, Chrome Browser, Mozilla Browser, et al.) +* Crypto stack (OpenSSL, Android, SymCrypt, et al.) + +Each of the above combinations is considered a single "platform". New versions are not considered new platforms, but different versions of the same platform. Only if a new version modifies one of the above elements would it be considered a new platform. + +Testing all platforms all the time is too expensive. This document defines the testing policy for each branch of the product. ## Policy +### Versions + We want to mix and match platform versions and .NET versions to produce good platform coverage without too much cost. This means we want to catch breaks on each platform as quickly as possible, and prioritize catching the type of platform breaks that are most likely to affect the specific version being tested. * `main` - PRs run on the *latest supported* platform. From aa413a618f51b4f0bf6b3bd1bad08860555db7aa Mon Sep 17 00:00:00 2001 From: Andy Gocke Date: Fri, 3 Oct 2025 10:40:39 -0700 Subject: [PATCH 4/9] Clarify testing policy for platform versions --- docs/infra/test-platforms.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/infra/test-platforms.md b/docs/infra/test-platforms.md index 164ba738e744e5..fa7b1b4515975b 100644 --- a/docs/infra/test-platforms.md +++ b/docs/infra/test-platforms.md @@ -12,7 +12,7 @@ Each of the above combinations is considered a single "platform". New versions are not considered new platforms, but different versions of the same platform. Only if a new version modifies one of the above elements would it be considered a new platform. -Testing all platforms all the time is too expensive. This document defines the testing policy for each branch of the product. +Testing all versions of all platforms all the time is too expensive. This document defines the testing policy for each branch of the product. ## Policy From 180c83cfcae931b46195c639a61965539342e042 Mon Sep 17 00:00:00 2001 From: Andy Gocke Date: Sun, 5 Oct 2025 19:32:21 -0700 Subject: [PATCH 5/9] Add open question about version testing triggers Added an open question regarding area paths triggering additional version testing. --- docs/infra/test-platforms.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/infra/test-platforms.md b/docs/infra/test-platforms.md index fa7b1b4515975b..d4eac284870a50 100644 --- a/docs/infra/test-platforms.md +++ b/docs/infra/test-platforms.md @@ -25,6 +25,8 @@ We want to mix and match platform versions and .NET versions to produce good pla The above policy only applies to PRs. Scheduled or incidental runs can be queued against other platform definitions, if deemed necessary. +- [ ] **Open question** Should certain area paths trigger additional version testing? + ## Details ### Platform version From 78c0deac8257009b7cbe6c28dbc52bd7bb88af3c Mon Sep 17 00:00:00 2001 From: Andy Gocke Date: Sun, 5 Oct 2025 20:00:59 -0700 Subject: [PATCH 6/9] Update test-platforms.md with servicing release policy Added instructions for handling servicing releases in PR configuration. --- docs/infra/test-platforms.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/infra/test-platforms.md b/docs/infra/test-platforms.md index d4eac284870a50..b622b42c2e3fb0 100644 --- a/docs/infra/test-platforms.md +++ b/docs/infra/test-platforms.md @@ -47,3 +47,8 @@ We currently regularly maintain three .NET versions: 3. The release before the previous release The last two versions (which have already been released) are called "servicing" releases. + +Because the policy differs between main and servicing, whenever the main branch is snapped to a servicing release, the PR configuration needs to change. In particular, the following tasks have to be done: + +1. [ ] Determine the **oldest** supported version of each test platform. The policy to determine this version is currently out of scope of this document. Refer to https://dotnet.microsoft.com/en-us/platform/support/policy/dotnet-core +2. [ ] Change PR definitions in CI to use oldest versions. From be8d59eeb47acd5a3803a2f1563f7dca8e5a40e1 Mon Sep 17 00:00:00 2001 From: Andy Gocke Date: Thu, 16 Oct 2025 15:05:38 -0700 Subject: [PATCH 7/9] Update docs/infra/test-platforms.md Co-authored-by: Jan Kotas --- docs/infra/test-platforms.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/infra/test-platforms.md b/docs/infra/test-platforms.md index b622b42c2e3fb0..6ac811eeee3738 100644 --- a/docs/infra/test-platforms.md +++ b/docs/infra/test-platforms.md @@ -52,3 +52,4 @@ Because the policy differs between main and servicing, whenever the main branch 1. [ ] Determine the **oldest** supported version of each test platform. The policy to determine this version is currently out of scope of this document. Refer to https://dotnet.microsoft.com/en-us/platform/support/policy/dotnet-core 2. [ ] Change PR definitions in CI to use oldest versions. +3. [ ] Stabilize CI for oldest OS versions. From 6441bbcbe4887c3a536d98e862d2d5b3ccb2f8e1 Mon Sep 17 00:00:00 2001 From: Andy Gocke Date: Wed, 22 Oct 2025 09:50:36 -0700 Subject: [PATCH 8/9] Update docs/infra/test-platforms.md Co-authored-by: Jeff Handley --- docs/infra/test-platforms.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/infra/test-platforms.md b/docs/infra/test-platforms.md index 6ac811eeee3738..ab84aa0cde585b 100644 --- a/docs/infra/test-platforms.md +++ b/docs/infra/test-platforms.md @@ -40,13 +40,13 @@ We assume that all supported platform versions in between have sufficient covera ### .NET lifecycle -We currently regularly maintain three .NET versions: +We regularly maintain three .NET versions: -1. `main`, i.e. the next release -2. The previous release -3. The release before the previous release +1. `main`, i.e. the next release. +2. The previous release. +3. The release before the previous release. -The last two versions (which have already been released) are called "servicing" releases. +The last two versions (which have already been released) are called "servicing" releases. During the Release Candidate (RC1, RC2, GA) phase of the next release, its release branches are treated the same as "servicing" releases. Because the policy differs between main and servicing, whenever the main branch is snapped to a servicing release, the PR configuration needs to change. In particular, the following tasks have to be done: From 5e52397b76feded8ba173e52a537b553f173498d Mon Sep 17 00:00:00 2001 From: Andy Gocke Date: Tue, 28 Oct 2025 12:40:11 -0700 Subject: [PATCH 9/9] Update docs/infra/test-platforms.md Co-authored-by: Jeff Handley --- docs/infra/test-platforms.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/infra/test-platforms.md b/docs/infra/test-platforms.md index ab84aa0cde585b..9166d362262696 100644 --- a/docs/infra/test-platforms.md +++ b/docs/infra/test-platforms.md @@ -50,6 +50,7 @@ The last two versions (which have already been released) are called "servicing" Because the policy differs between main and servicing, whenever the main branch is snapped to a servicing release, the PR configuration needs to change. In particular, the following tasks have to be done: -1. [ ] Determine the **oldest** supported version of each test platform. The policy to determine this version is currently out of scope of this document. Refer to https://dotnet.microsoft.com/en-us/platform/support/policy/dotnet-core -2. [ ] Change PR definitions in CI to use oldest versions. -3. [ ] Stabilize CI for oldest OS versions. +1. [ ] Determine the **oldest** supported version of each test platform. The policy to determine this version is currently out of scope of this document. Refer to https://dotnet.microsoft.com/en-us/platform/support/policy/dotnet-core and https://github.com/dotnet/core/blob/main/os-lifecycle-policy.md. +2. [ ] Change PR definitions in CI to add coverage for the **oldest supported** version alongside the **newest supported** version. +3. [ ] Change the daily scheduled builds to run on the **every supported** version if changes have been committed (including platform definition changes). +4. [ ] Stabilize CI for each supported version.