Add retry gate for MSB3030 copy race in parallel CI builds#66440
Merged
wtgodbe merged 1 commit intodotnet:mainfrom Apr 23, 2026
Merged
Add retry gate for MSB3030 copy race in parallel CI builds#66440wtgodbe merged 1 commit intodotnet:mainfrom
wtgodbe merged 1 commit intodotnet:mainfrom
Conversation
github-actions
Bot
added
the
needs-area-label
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a CI-only MSBuild workaround to reduce flaky MSB3030 copy failures caused by parallel builds racing on shared dependency outputs.
Changes:
- Introduces a
RoslynCodeTaskFactoryinline task that waits/retries for missing source files to appear. - Hooks a new target
BeforeTargets="_CopyFilesMarkedCopyLocal"to run the wait gate during CI builds.
Parallel builds race on shared project outputs: when a dependency's CopyFilesToOutputDirectory overwrites a DLL/PDB, consumers running _CopyFilesMarkedCopyLocal can see 'file not found' (MSB3030). The SDK's Copy task does not retry on FileNotFoundException. Add a BeforeTargets gate that waits up to 2.5s for missing source files before the stock Copy task runs. Only active in CI builds (ContinuousIntegrationBuild=true) to avoid overhead in local dev. Proper fix: dotnet/msbuild#13599 Tracked as: dotnet#62807 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
wtgodbe
force-pushed
the
wtgodbe/fix-copy-race
branch
from
7998fc8 to
86ec5e2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Parallel CI builds (
/m) race on shared project outputs. When a dependency'sCopyFilesToOutputDirectoryoverwrites a DLL/PDB, consumer projects'_CopyFilesMarkedCopyLocalcan see the source file as transiently missing and fail with MSB3030 ("Could not copy because it was not found").The SDK's
Copytask does not retry onFileNotFoundException— only onIOException. This causes ~25 flaky failures per month (Known Build Error #62807).Fix
Adds a
BeforeTargets="_CopyFilesMarkedCopyLocal"gate ineng/Workarounds.targetsthat waits for missing source files before the stockCopytask runs. Uses aRoslynCodeTaskFactoryinline task that retriesFile.Existsup to 5 times with 500ms delay (2.5s max).Only active in CI (
ContinuousIntegrationBuild=true) — no overhead for local dev builds.Root Cause (from binlog analysis)
Hosting.Server.Abstractions) are evaluated 6+ times andCopyFilesToOutputDirectoryruns twicebin/output via_CopyFilesMarkedCopyLocalCopyFilesToOutputDirectorybriefly makes the file unavailable during overwriteProper Fix
Filed dotnet/msbuild#13599 requesting the
Copytask retry onFileNotFoundException, not justIOException. Once that's fixed upstream, this workaround can be removed.Relates to #62807