Don't run integration tests in release jobs by Copilot · Pull Request #7568 · microsoft/testfx

Copilot · 2026-03-17T09:29:08Z

Integration tests spawn child processes and run dotnet build internally (already covering both Debug and Release configurations). Running them under both Debug/Release CI matrix entries is redundant—only the test project itself gets rebuilt, not meaningful additional coverage.

Changes

azure-pipelines.yml — uses dotnet test --test-modules with a glob pattern to run only unit tests in Release builds:
- Debug: keeps the original dotnet test command unchanged (runs all tests including integration)
- Release: uses dotnet test --test-modules "artifacts/bin/*UnitTests*/$(_BuildConfig)/**/*UnitTests.dll" to run only unit test modules, skipping integration tests
- Integration test binlog copy/publish steps conditioned on eq(variables._BuildConfig, 'Debug') so they're skipped in Release
- Applied to Windows, Linux, and MacOS jobs

No solution filters or eng/Build.props changes are needed—the build always compiles the full solution. On Linux/Mac, only DLLs built from NonWindowsTests.slnf exist in artifacts/bin/, so the glob pattern naturally excludes Windows-only tests as well.

The official pipeline (azure-pipelines-official.yml) is not affected—its Release builds continue running integration tests as the sole configuration.

🔒 GitHub Advanced Security automatically protects Copilot coding agent pull requests. You can protect all pull requests by enabling Advanced Security for your repositories. Learn more about Advanced Security.

Add solution filters (TestFx.NoIntegrationTests.slnf and NonWindowsTests.NoIntegrationTests.slnf) that exclude integration test projects and their test assets. Modify azure-pipelines.yml to use ExcludeIntegrationTests property and the new solution filters for Release matrix entries, so that integration tests are only built and run under Debug jobs. Update eng/Build.props to select the appropriate solution filter based on the ExcludeIntegrationTests property. Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>

Evangelink · 2026-03-17T12:13:01Z

@copilot Publish integration tests binlogs step fails on release for all OSes as we don't run the tests

… excluded The 'Copy binlogs' and 'Publish integration tests binlogs' steps fail on Release jobs because the testsuite binlog directory doesn't exist when integration tests are excluded. Add condition to only run these steps when _ExcludeIntegrationTests is 'false' (Debug builds). Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>

Copilot · 2026-03-17T12:18:01Z

@copilot Publish integration tests binlogs step fails on release for all OSes as we don't run the tests

Fixed in ecc4627. Added condition: and(always(), eq(variables._ExcludeIntegrationTests, 'false')) to the "Copy binlogs" and "Publish integration tests binlogs" steps for all three OS jobs (Windows, Linux, MacOS), so they're skipped on Release builds where integration tests don't run.

Evangelink · 2026-03-17T12:31:17Z

@copilot can you please do a different PR where you explore doing the same functional fix but not by creating more slnf but instead by updating the dotnet test command to use:
--test-modules

Filters test modules using file globbing in .NET. Only tests belonging to those test modules will run. For more information and examples on how to use file globbing in .NET, see File globbing.

More info at https://learn.microsoft.com/en-us/dotnet/core/tools/dotnet-test-mtp

Copilot · 2026-03-17T12:35:19Z

@copilot can you please do a different PR where you explore doing the same functional fix but not by creating more slnf but instead by updating the dotnet test command to use:
--test-...

I'm unable to create new PRs — I can only work within the current PR branch. Would you like me to rework this PR to use the --test-modules approach instead of the .slnf files? That would involve reverting the solution filter additions and updating the dotnet test command in azure-pipelines.yml to use --test-modules to exclude integration test assemblies.

Evangelink · 2026-03-17T13:29:09Z

@copilot ok let's reword this PR then

… tests in Release Replace the solution filter approach with dotnet test --test-modules glob pattern. For Release builds, runs only unit test modules via: --test-modules "artifacts/bin/*UnitTests*/$(_BuildConfig)/**/*UnitTests.dll" For Debug builds, keeps original test commands unchanged. Removes TestFx.NoIntegrationTests.slnf and NonWindowsTests.NoIntegrationTests.slnf. Reverts eng/Build.props to original (no ExcludeIntegrationTests logic needed). Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>

Copilot · 2026-03-17T13:45:03Z

@copilot ok let's reword this PR then

Reworked in c6ff763. Replaced the .slnf approach with --test-modules:

Deleted TestFx.NoIntegrationTests.slnf and NonWindowsTests.NoIntegrationTests.slnf
Reverted eng/Build.props to original (no changes needed)
Release test step now uses:
```
dotnet test --test-modules "artifacts/bin/*UnitTests*/$(_BuildConfig)/**/*UnitTests.dll"
```
This glob matches all unit test DLLs while naturally excluding integration test modules. On Linux/Mac, only DLLs built from NonWindowsTests.slnf exist in artifacts/bin/, so the pattern also excludes Windows-only tests.
Debug test step unchanged — runs all tests (unit + integration) as before
Integration test binlog steps conditioned on eq(variables._BuildConfig, 'Debug')

Evangelink · 2026-03-17T14:27:00Z

@Youssef1313 I was reviewing the tests and I wonder if we should only skip acceptance tests but still run the other integration tests (specifically for MSTest).

Copilot

Pull request overview

Updates the unofficial CI pipeline to avoid running integration tests in Release matrix legs (keeping them in Debug only), reducing redundant work while still running unit test coverage.

Changes:

Conditioned existing dotnet test steps to run only for Debug.
Added dotnet test --test-modules in Release to execute only unit test assemblies from artifacts/bin/....
Conditioned integration-test binlog copy/publish steps to run only for Debug on Windows/Linux/macOS jobs.

nohwnd

I've spent a minute figuring out why you run the tests twice, hardcoding the config into the paths would make it much easier to see that one step is for debug and other is for release, or at least moving the condition up.

Also do you need to run debug tests at all ?

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

…on-tests-redundancy # Conflicts: # azure-pipelines.yml

In Release mode, if the build fails before the test step runs, the TestResults directory won't exist and the publish step would add a misleading failure. Adding continueOnError: true prevents this from failing the build.

Copilot

Copilot's findings

Comments suppressed due to low confidence (4)

azure-pipelines.yml:304

continueOnError: true on publishing the Test Results artifact can mask failures to upload results/logs, making CI issues harder to diagnose. If not explicitly desired, consider removing this so artifact publish failures surface as job failures.

            continueOnError: true

azure-pipelines.yml:219

The Release-only dotnet test --test-modules command is missing --no-build. Because the preceding cibuild.sh already produced the test binaries, adding --no-build would avoid redundant compilation and keep behavior aligned with the Debug test step.

          - script: dotnet test --test-modules "artifacts/bin/*UnitTests*/$(_BuildConfig)/**/*UnitTests.dll" --results-directory $(BUILD.SOURCESDIRECTORY)/artifacts/TestResults/$(_BuildConfig) --no-progress --output detailed

azure-pipelines.yml:289

The Release-only dotnet test --test-modules step doesn’t include --no-build. Since the build step already ran for this config, consider adding --no-build to prevent an extra build/evaluation pass and to keep Debug/Release behavior consistent.

          - script: dotnet test --test-modules "artifacts/bin/*UnitTests*/$(_BuildConfig)/**/*UnitTests.dll" --results-directory $(BUILD.SOURCESDIRECTORY)/artifacts/TestResults/$(_BuildConfig) --no-progress --output detailed

azure-pipelines.yml:234

continueOnError: true on the Test Results artifact publish step means the pipeline can pass even if test results fail to upload, reducing observability for CI failures. Unless this is a deliberate reliability tradeoff, consider removing continueOnError here.

            continueOnError: true

Files reviewed: 1/1 changed files
Comments generated: 2

…OnError - Add --no-build to all three Release dotnet test --test-modules commands to avoid unnecessary rebuilds (Windows, Linux, macOS) - Remove continueOnError: true from Publish Test Results steps to avoid masking real infrastructure failures

--test-modules only accepts a single argument. Use *.exe glob which covers both net4x (native exe) and net8+/net9+ (apphost exe) outputs. Linux/macOS already use a single --test-modules with *.dll pattern.

Copilot

Copilot's findings

Files reviewed: 1/1 changed files
Comments generated: 1

Initial plan

1ac24f7

Copilot AI assigned Copilot and Evangelink Mar 17, 2026

Copilot started work on behalf of Evangelink March 17, 2026 09:29 View session

Copilot AI linked an issue Mar 17, 2026 that may be closed by this pull request

Don't run integration tests in release jobs #7555

Closed

Copilot AI changed the title ~~[WIP] Fix redundancy in integration tests for release jobs~~ Don't run integration tests in release jobs Mar 17, 2026

Copilot AI requested a review from Evangelink March 17, 2026 09:51

Copilot finished work on behalf of Evangelink March 17, 2026 09:51

Copilot started work on behalf of Evangelink March 17, 2026 12:13 View session

Copilot finished work on behalf of Evangelink March 17, 2026 12:18

Copilot started work on behalf of Evangelink March 17, 2026 12:31 View session

Copilot finished work on behalf of Evangelink March 17, 2026 12:35

Copilot started work on behalf of Evangelink March 17, 2026 13:29 View session

Copilot finished work on behalf of Evangelink March 17, 2026 13:46

Evangelink marked this pull request as ready for review March 20, 2026 09:00

Copilot AI review requested due to automatic review settings March 20, 2026 09:00

Evangelink enabled auto-merge March 20, 2026 09:00

Copilot started reviewing on behalf of Evangelink March 20, 2026 09:00 View session

Copilot AI reviewed Mar 20, 2026

View reviewed changes

Comment thread azure-pipelines.yml Outdated

nohwnd reviewed Mar 20, 2026

View reviewed changes

drognanar approved these changes Mar 20, 2026

View reviewed changes

Update azure-pipelines.yml

953e5fd

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

This was referenced Mar 31, 2026

[repo-status] Daily Repo Status — March 31, 2026 #7643

Closed

[repo-status] Daily Repo Status — April 2, 2026 #7666

Closed

[repo-status] Daily Repo Status – April 5, 2026 #7679

Closed

[repo-status] Daily Repo Status – April 6, 2026 #7681

Closed

Evangelink added 2 commits April 21, 2026 09:28

Merge remote-tracking branch 'origin/main' into copilot/fix-integrati…

67f7d42

…on-tests-redundancy # Conflicts: # azure-pipelines.yml

Add continueOnError to Publish Test Results steps

61486e1

In Release mode, if the build fails before the test step runs, the TestResults directory won't exist and the publish step would add a misleading failure. Adding continueOnError: true prevents this from failing the build.

Copilot AI review requested due to automatic review settings April 21, 2026 08:30

Copilot started reviewing on behalf of Evangelink April 21, 2026 08:31 View session

Copilot AI reviewed Apr 21, 2026

View reviewed changes

Comment thread azure-pipelines.yml Outdated

Comment thread azure-pipelines.yml Outdated

Evangelink added 2 commits April 21, 2026 10:38

Fix --test-modules: use single *.exe glob for Windows Release

7f8bb73

--test-modules only accepts a single argument. Use *.exe glob which covers both net4x (native exe) and net8+/net9+ (apphost exe) outputs. Linux/macOS already use a single --test-modules with *.dll pattern.

Copilot AI review requested due to automatic review settings April 21, 2026 09:21

Copilot AI reviewed Apr 21, 2026

View reviewed changes

Comment thread azure-pipelines.yml

Evangelink merged commit 1a9f275 into main Apr 21, 2026
13 of 14 checks passed

Evangelink deleted the copilot/fix-integration-tests-redundancy branch April 21, 2026 12:18

Conversation

Copilot AI commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Uh oh!

Evangelink commented Mar 17, 2026

Uh oh!

Copilot AI commented Mar 17, 2026

Uh oh!

Evangelink commented Mar 17, 2026

Uh oh!

Copilot AI commented Mar 17, 2026

Uh oh!

Evangelink commented Mar 17, 2026

Uh oh!

Copilot AI commented Mar 17, 2026

Uh oh!

Evangelink commented Mar 17, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

nohwnd left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot's findings

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot's findings

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Copilot AI commented Mar 17, 2026 •

edited

Loading