[wip proposal] roadmap: Fast Up-To-Date Checks for CLI builds#13529
Closed
[wip proposal] roadmap: Fast Up-To-Date Checks for CLI builds#13529
Conversation
Agent-Logs-Url: https://github.com/dotnet/msbuild/sessions/2f9b89c3-bc21-4f9c-ad1d-bfe86fa532bc Co-authored-by: JanProvaznik <25267098+JanProvaznik@users.noreply.github.com>
Copilot created this pull request from a session on behalf of
JanProvaznik
April 12, 2026 15:49
View session
JanProvaznik
changed the title
Member
|
Oops this was not supposed to be a pr, but an investigation in agent mode, but after all why not, please tell me if this is a fundamentally bad idea. It bothers me very much that in VS we can FUTDC in milliseconds to do a no-op build for big solutions and in dotnet build it takes seconds to do no-op even for medium size solutions. There is much to be gained. |
Complete rewrite of the Fast Up-To-Date Check roadmap document based on deep investigation across MSBuild engine, SDK CLI layer, VS project-system source, ProjectCachePlugin system, and prior art. Key findings: - ProjectCachePlugins are consulted AFTER evaluation (can't skip the dominant 50-70% cost) - Best insertion point: BuildManager.ExecuteSubmission() line 1580, before the cache/build fork - obj/ path can be discovered by convention for ~95-98% of projects via BaseIntermediateOutputPath (always 'obj\') - 15 correctness scenarios analyzed with severity ratings - Prior art mapped: RAR-as-service abandoned, Static Graph design envisioned persistent cache, SDK has zero eval capability Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Member
|
The spec document has been rewritten with a deep investigation. See the full analysis in the updated spec. Related discussions across repos: Core problem: sdk#7850, msbuild#2015, msbuild#12954, msbuild#1979 RAR-as-service (abandoned): #3139, #5536, #6193, #11741 (revival) Project cache plugins: #5936, #8726, #9122 ("MSBuild has no project-level skip") Graph/eval/server: #3696 (static graph design), #4025 (NuGetSdkResolver overhead), #9462 (concurrent builds), #10035 (daemon lifetime) VS FUTDC bugs relevant to CLI: project-system#4261 (git branch switch false positive), project-system#4100 (custom targets invisible), project-system#9477 (T4 not detected), project-system#8908 (multi-target copy bug), msbuild#3762 (DisableFastUpToDateCheck cascade) |
Same data format for both disk-based (Phase 1) and server in-memory (Phase 4) paths. Server flushes to disk on shutdown, restores on cold start — mirroring VS's OnBeforeCloseSolution persistence pattern. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Compares obj/, .msbuild/, .vs/, user-level cache, and artifacts/ directories. Key factors: dotnet clean semantics, .gitignore, pre-evaluation discoverability, multi-tool compatibility. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ions Major additions: - Section 7: Red-team blocking flaws (BuildResult semantic gap, fingerprint circularity, graph/solution build bypass, blast radius) - Section 2.1: How CPS actually feeds FUTDC (IVsAsyncFileChangeEx for file watching, Dataflow for snapshots, File.GetLastWriteTimeUtc at check time - no Windows-specific deps in the check itself) - Section 9: Revised recommendations after red-team (evaluation caching in server mode as safer alternative to evaluation skipping) - Open questions table with blocking/non-blocking classification Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ects Key reframing: for projects with downstream consumers, this is not a fast up-to-date check but a result-cache architecture. Target-set identity, configuration identity, completeness, and invalidation model all need formal specification. Leaf-only MVP could be viable. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Five specific engine constraints from code review: - ResultsCache.SatisfyRequest needs InitialTargets/DefaultTargets - TargetResult.Items must be fully populated with metadata - All-or-nothing target cache semantics - Non-graph builds have dynamic target requests - Graph builds know full target closure (natural fit) Updated open questions with answered items and new questions about graph topology caching. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Key reframe: VS FUTDC is a per-project gate within VS Solution Build Manager's project iteration loop. dotnet build passes the whole .sln to MSBuild as one invocation - there's no per-project loop. If the SDK iterated projects itself (like VS SBM), it could do FUTDC per-project without ANY engine changes. This sidesteps all red-team blocking issues: no synthetic BuildResult, no ResultsCache semantic gap, no target-result completeness problem. .slnx makes solution parsing feasible at the SDK layer. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
dotnet run file.cs already implements 3-tier BuildLevel (None/Csc/All) with SDK-layer up-to-date checking, proving the pattern works outside VS. Measured MSBuild overhead avoided: ~870ms Windows, ~500ms Linux. FUTDC code in project-system is VS-coupled (MEF, CPS, IVsServices) but core algorithm (ItemHashing, TimestampCache, 5-stage logic) is portable. Data acquisition layer would need reimplementation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
CPS = VS's project system. Dataflow = async pipeline keeping snapshots warm. Replaced all instances with plain English. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Final architecture: MSBuild server as CLI's equivalent of VS's project system. Five layers: 1. Evaluation cache (ProjectInstanceCache alongside ProjectRootElementCache) 2. Graph cache (persistent ProjectGraph across builds) 3. Per-node project skip (IsNodeUpToDate in ExecuteGraphBuildScheduler) 4. Disk persistence (obj/.futdc for cold-start recovery only) 5. --superfast switch (bundles graph+server+caching+skip+MT tasks) Correct by construction: uses real cached evaluations (not heuristic fingerprints), so no circularity. Same timestamp-based invalidation as existing ProjectRootElementCache. Reactive validation (timestamp checks at build time) vs VS's proactive file watchers. ~3250 LOC total. Zero SDK changes needed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…t skip Core implementation of dotnet build --superfast: - ProjectInstanceCache: server-persistent evaluation cache with MSBuildAllProjects timestamp-based invalidation - ProjectGraphCache: server-persistent graph cache with project file timestamp-based invalidation - Per-node skip in ExecuteGraphBuildScheduler: IsNodeUpToDate checks input timestamps vs output timestamps, skips up-to-date nodes - --superfast/-sf switch in XMake.cs that implies --graph mode - BuildParameters.SuperFast property - impl-plan.md with full implementation details Build succeeded with 0 errors. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
End-to-end verified: - 'dotnet msbuild TestSuperFast.sln -superfast' on 2-project solution - No-op build: 2/2 projects skipped in 2.2s (graph load ~0.9s) - Touch Lib source: both rebuild correctly (dependency cascade) - Touch App source: Lib skipped, App rebuilds (1/2 skipped) - Subsequent no-op: 2/2 skipped again Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Working prototype verified: - 2/2 projects skipped in 2.2s (no-op) - Correct dependency cascade on source changes - Correct partial skip on leaf-only changes Known gaps documented: incomplete synthetic BuildResult, missing glob/env var/source generator tracking, no memory management, no server mode enforcement. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Handle multi-targeting outer builds (IsCrossTargetingBuild) - Add UpToDateCheckInput/Output/Built checks - Add CopyToOutputDirectory, AdditionalFiles, EditorConfigFiles - Early --superfast arg scan to enable server mode - Remove test projects from repo (use Q:\test-superfast instead) Verified: 6/6 projects skipped on no-op in ~1.4s with MSBuild.exe Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Server mode enforcement via early arg scan causes handshake mismatches when custom MSBuild is deployed alongside stock SDK. Removed for now - the caches work within a single invocation via graph build mode. Verified results: - 6/6 projects skipped on no-op: 0.4s (with -tl:off) - 1/1 console project skip: 1.2s (via dotnet msbuild) - Correct partial rebuild on source touch - Correct full rebuild on output deletion Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- --superfast forces MSBUILDUSESERVER=1 for cross-invocation caching - 2-second server connect timeout (fast fallback to in-process) - Graceful exception handling: if server fails, falls back to in-process graph build (caches still work within invocation) - MSBUILDSERVERCONNECTTIMEOUT env var for configurable timeout - DeepCopy cached ProjectInstance to prevent mutation corruption Server mode works correctly when MSBuild versions match (production). During dev with DLL patching, fast fallback ensures <5s instead of 20s. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ation Graph cache now correctly deep-copies each node's ProjectInstance when returning a cached graph. Target execution mutates ProjectInstance during build (adds items, changes properties). Without DeepCopy, the second build sees stale mutated state (duplicate framework references). Also made ProjectGraphNode.ProjectInstance have internal setter to enable replacing the instance with a clone. Verified on Q:\cons\cons.csproj: - Build 1: 1/1 skipped, 3.0s (graph constructed + eval cached) - Build 2: 1/1 skipped, 0.39s (graph cached + eval cached) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Member
|
there is much prior art on this and vibecoding this publicly is low value, will reopen when it's actually discussable |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
VS achieves fast inner-loop builds via a Fast Up-To-Date Check (FUTDC) in
dotnet/project-systemthat skips MSBuild entirely in ~1-10ms. CLI users (dotnet build) get none of this — every no-op build pays full evaluation (~50-70% of time), RAR, and target-walking costs. This document analyzes the FUTDC architecture and proposes how to bring equivalent capabilities to MSBuild.Changes Made
New spec document at
documentation/specs/fast-up-to-date-check-roadmap.md:dotnet/project-systemsourceTargetUpToDateChecker,/questionflag,IIncrementalTask,CopyUpToDateMarker, results cache — with source file referencesobj/.msbuild.futdc)obj/path discovery without evaluation, multi-targeting, extensibility, environment sensitivity, concurrencyTesting
Documentation only — no code changes.
Notes
The document references existing MSBuild docs (
Persistent-Problems.md,Build-Scenarios.md,specs/question.md) and links to specific source files in bothdotnet/msbuildanddotnet/project-system. The quote fromPersistent-Problems.mdpreserves the original typo ("produced" → should be "produce") since it's a direct citation.