Fix CI flakiness: MSB4216 task host failures, dotnet-watch hangs, NuGet errors by mmitche · Pull Request #53424 · dotnet/sdk

mmitche · 2026-03-12T20:04:26Z

Summary

This PR addresses intermittent CI failures identified through systematic analysis of recent builds.

Remaining Fixes (after merge from main)

Root Cause 1 - MSB4216 task host failures on macOS Helix

Impact: Tests using NuGet package tasks with TaskHostFactory fail intermittently on macOS.
Fix: Export DOTNET_HOST_PATH in RunTestsOnHelix.sh and RunTestsOnHelix.cmd.

Root Cause 2 - GZipCompress file lock races

Impact: Parallel.For in GZipCompress races with antivirus/file indexer.
Fix: Added retry with exponential backoff for file access.

Root Cause 3 - Noisy NuGet source removal errors

Impact: Helix test setup produces confusing errors removing non-existent NuGet sources.
Fix: Suppress errors from dotnet nuget remove source commands.

Root Cause 4 - DefaultRequestDispatcherTest timing issues

Impact: Test makes timing assumptions that fail under load.
Fix: Increased timeouts for CI environment.

Root Cause 5 - Missing runtimeconfig.json in test assets

Impact: Test tool projects fail to locate runtime configuration.
Fix: Added MSBuild target in test assets Directory.Build.targets.

Changes Removed

The dotnet-watch Aspire race condition fix has been removed from this PR — it was superseded by @tmat's proper fix in #53271 (now merged to main).

Validation

Starting fresh validation run after merge from main. Target: 25 consecutive passing builds.

Copilot

Pull request overview

This PR hardens Helix/CI test execution for Hot Reload / dotnet-watch scenarios by reducing hang risk and improving the reliability of Helix environment setup.

Changes:

Replace “effectively infinite” DCP/Aspire timeouts used by watch-based tests with bounded (5-minute) values to prevent multi-hour Helix hangs.
Improve test process cleanup by closing stdin prior to termination and adding a bounded wait for process exit.
Update Helix test entrypoint scripts to set DOTNET_HOST_PATH and make NuGet source removal resilient when sources are absent.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
test/Microsoft.DotNet.HotReload.Test.Utilities/WatchableApp.cs	Switches DCP/Aspire-related environment timeouts to bounded values for CI.
test/Microsoft.DotNet.HotReload.Test.Utilities/AwaitableProcess.cs	Adjusts disposal/termination behavior to reduce hangs during cleanup.
build/RunTestsOnHelix.sh	Sets `DOTNET_HOST_PATH` and makes `dotnet nuget remove source` tolerant of missing sources.
build/RunTestsOnHelix.cmd	Sets `DOTNET_HOST_PATH` and suppresses errors when removing non-existent NuGet sources.

You can also share your feedback on Copilot code review. Take the survey.

Copilot · 2026-03-12T20:08:07Z

+            catch (OperationCanceledException)
+            {
+                Logger.Log($"Process {Id} did not exit within 30 seconds after Kill()");


If the process still hasn’t exited after the 30s timeout, DisposeAsync only logs and continues. That can leave a runaway dotnet-watch process (or child processes) running in the Helix work item, causing resource leaks and cross-test interference while also hiding the failure signal. Consider failing the test/cleanup in this case (or at least making the cleanup path retry/force termination and surface the problem), rather than just logging and proceeding to dispose the Process handle.

Suggested change

catch (OperationCanceledException)

{

Logger.Log($"Process {Id} did not exit within 30 seconds after Kill()");

catch (OperationCanceledException ex)

{

Logger.Log($"Process {Id} did not exit within 30 seconds after Kill()");

throw new TimeoutException($"Process {Id} did not exit within 30 seconds after Kill().", ex);

Copilot · 2026-03-12T20:08:08Z

+            try
+            {
+                Process.StandardInput.Close();
+            }
+            catch


The broad catch around closing StandardInput/Kill() swallows all exceptions without logging. Since this code was added to address a platform-specific hang, swallowing the exception makes it hard to diagnose when stdin can’t be closed (or why). Consider at least logging the exception details (or narrowing the caught exception types) so cleanup failures are actionable in CI logs.

tmat · 2026-03-13T02:29:33Z

            _isDisposed = true;

            // wait for all in-flight process initialization to complete:
+            // If no session initialization is in-flight (_pendingSessionInitializationCount == 0),


Not entirely correct either. Ok to merge, I'll follow up with better fix.

@tmat This is an automatic fix by the AI for flakiness. Don't merge this...when it's gotten 25 passing runs we'll take a second pass over this to smooth it out.

Yeah, I figured. It looks very AI like. It's great it found the issue. I'll work on a better fix over the weekend.

tmat · 2026-03-13T02:34:13Z

watch changes lgtm

mmitche · 2026-03-13T12:44:20Z

🎯 Milestone: 5 Consecutive Passes

Validation Results

Build	Jobs	Result
1333280	18/18 ✅	Passed
1333378	18/18 ✅	Passed
1333489	18/18 ✅	Passed
1333567	18/18 ✅	Passed
1333595	18/18 ✅	Passed

Root Causes Fixed

MSB4216 TaskHostFactory — DOTNET_HOST_PATH not set in Helix scripts
dotnet-watch Aspire hang — Per-operation timeout was inheriting the 2-hour Helix work-item timeout instead of being capped at 5 minutes. Also fixed semaphore deadlock in AspireServiceFactory.DisposeAsync, added stdin close before process kill, and set DCP timeout environment variables.
GZipCompress file lock — Parallel.For races with antivirus/file indexer; added retry with exponential backoff
NuGet source removal noise — Suppressed stderr from removing non-existent NuGet sources
Missing runtimeconfig.json — Added MSBuild target to include runtimeconfig.json in NuGet packages for test tool projects

Baseline vs Current

Before: ~57% failure rate (17/30 builds failed on main)
After: 5/5 consecutive perfect builds (100% pass rate so far)

Continuing validation toward 25 consecutive passes target.

mmitche · 2026-03-13T13:44:05Z

@akoeplinger This is an automated run...let it go for a while..don't merge or approve. I'll have it go through another pass and also reivew at the end of when it thinks it gets a stable run.

On Linux, Console.ReadKey() blocks indefinitely when stdin is inherited from a parent test process. Aspire launcher processes (server, resources) each create a PhysicalConsole that starts a LongRunning task calling Console.ReadKey(), which never unblocks when the test process kills the child. Check Console.IsInputRedirected before starting the keyboard listener to avoid the hang. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Resolved conflict: kept main's file-scoped namespace cleanup and numpad key handling, applied our Console.IsInputRedirected guard to prevent Aspire test hangs from Console.ReadKey() on inherited stdin. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

marcpopMSFT · 2026-03-25T20:14:22Z

    public PhysicalConsole(TestFlags testFlags)
    {
        Console.OutputEncoding = Encoding.UTF8;
-        _ = testFlags.HasFlag(TestFlags.ReadKeyFromStdin) ? ListenToStandardInputAsync() : ListenToConsoleKeyPressAsync();


@tmat to review this change.

marcpopMSFT · 2026-03-25T20:14:47Z

@@ -20,6 +20,10 @@ public class GZipCompress : Task
        [Required]
        public string OutputDirectory { get; set; }



@dotnet/aspnet-blazor-eng to review the changes to this file.

marcpopMSFT · 2026-03-25T20:17:15Z

Test changes look reasonable to me. Pinged a few folks on the product changes as they are in areas outside of my knoweldge.

@mmitche seems like it didn't get 25 passing runs in a row though and wasn't continuing to try to fix tests. Thoughts on next steps we should take here?

mmitche · 2026-03-26T14:13:25Z

Test changes look reasonable to me. Pinged a few folks on the product changes as they are in areas outside of my knoweldge.

@mmitche seems like it didn't get 25 passing runs in a row though and wasn't continuing to try to fix tests. Thoughts on next steps we should take here?

Yeah it looks like it gave up and decided that the additional failures were not its fault. Well, duh. I think getting this PR in, trimming it down to the fixes we think are good to go, and then I want to take a different approach with the next round of experimentation.

mmitche · 2026-03-26T14:16:44Z

@marcpopMSFT What I'd say is that anything we're unsure about here, let's just revert. We can then let the agent party on the results.

akoeplinger · 2026-03-30T11:06:25Z

@@ -20,6 +20,10 @@ public class GZipCompress : Task
        [Required]
        public string OutputDirectory { get; set; }



Copilot AI review requested due to automatic review settings March 12, 2026 20:04

github-actions Bot added the Area-Infrastructure label Mar 12, 2026

Copilot started reviewing on behalf of mmitche March 12, 2026 20:05 View session

Copilot AI reviewed Mar 12, 2026

View reviewed changes

mmitche force-pushed the fix/ci-flakiness/main branch from 83d044f to ae5112f Compare March 12, 2026 20:21

mmitche requested a review from a team as a code owner March 12, 2026 20:21

mmitche force-pushed the fix/ci-flakiness/main branch 2 times, most recently from a35fa75 to d9f4358 Compare March 12, 2026 23:54

mmitche requested review from a team and tmat as code owners March 12, 2026 23:54

build-analysis Bot mentioned this pull request Mar 13, 2026

dotnet-watch tests failing #40006

Open

mmitche force-pushed the fix/ci-flakiness/main branch from d9f4358 to 9949b40 Compare March 13, 2026 01:26

mmitche requested a review from a team as a code owner March 13, 2026 01:26

mmitche force-pushed the fix/ci-flakiness/main branch from 9949b40 to ed04d76 Compare March 13, 2026 02:28

tmat reviewed Mar 13, 2026

View reviewed changes

mmitche force-pushed the fix/ci-flakiness/main branch 3 times, most recently from 2eb12cb to bcae687 Compare March 13, 2026 12:43

mmitche force-pushed the fix/ci-flakiness/main branch from bcae687 to 1644c42 Compare March 13, 2026 12:45

akoeplinger reviewed Mar 13, 2026

View reviewed changes

Comment thread build/SetupHelixEnvironment.cmd Outdated

Comment thread build/SetupHelixEnvironment.cmd Outdated

mmitche force-pushed the fix/ci-flakiness/main branch 7 times, most recently from 4cb1ca9 to ea8fad3 Compare March 14, 2026 00:12

mmitche and others added 11 commits March 20, 2026 07:12

ci: trigger validation run 22

84d258f

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

ci: validation run 23

b2da329

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

ci: validation run 24

06c9f29

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

ci: retrigger validation run 24

6b850a8

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

ci: validation run 25

99f60c7

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

ci: validation run 26

fa48b4b

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

ci: validation run 27

22505b5

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

ci: validation run 28

e99bb84

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

ci: validation run 29

5e36300

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

build-analysis Bot mentioned this pull request Mar 21, 2026

Could Not Resolve SDK - repo testfx issue #51530

Open

ci: validation run 30

3436132

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

marcpopMSFT self-requested a review March 24, 2026 20:23

marcpopMSFT reviewed Mar 25, 2026

View reviewed changes

Merge remote-tracking branch 'origin/main' into fix/ci-flakiness/main

4faeada

akoeplinger reviewed Mar 30, 2026

View reviewed changes

mmitche added 2 commits March 30, 2026 10:14

Revert some changes

40173db

Revert some changes

2fd15a9

MichaelSimons approved these changes Apr 2, 2026

View reviewed changes

MichaelSimons and others added 2 commits April 2, 2026 14:57

Merge branch 'main' into fix/ci-flakiness/main

fdb456c

Merge branch 'main' into fix/ci-flakiness/main

7f4a370

mmitche merged commit 96683a0 into main Apr 3, 2026
25 checks passed

mmitche deleted the fix/ci-flakiness/main branch April 3, 2026 15:55

dotnet-maestro Bot mentioned this pull request Apr 7, 2026

[main] Source code updates from dotnet/sdk dotnet/dotnet#5918

Merged

MichaelSimons mentioned this pull request Apr 10, 2026

Reapply "Migrate test infrastructure from xUnit v2 to xUnit v3 (#52930)" #53809

Merged

		@@ -20,6 +20,10 @@ public class GZipCompress : Task
		[Required]
		public string OutputDirectory { get; set; }

Conversation

mmitche commented Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Remaining Fixes (after merge from main)

Root Cause 1 - MSB4216 task host failures on macOS Helix

Root Cause 2 - GZipCompress file lock races

Root Cause 3 - Noisy NuGet source removal errors

Root Cause 4 - DefaultRequestDispatcherTest timing issues

Root Cause 5 - Missing runtimeconfig.json in test assets

Changes Removed

Validation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tmat commented Mar 13, 2026

Uh oh!

mmitche commented Mar 13, 2026

🎯 Milestone: 5 Consecutive Passes

Validation Results

Root Causes Fixed

Baseline vs Current

Uh oh!

Uh oh!

Uh oh!

mmitche commented Mar 13, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

marcpopMSFT commented Mar 25, 2026

Uh oh!

mmitche commented Mar 26, 2026

Uh oh!

mmitche commented Mar 26, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

mmitche commented Mar 12, 2026 •

edited

Loading