From d91f7d1a66ea268ef0cb140b50cd96d144d11266 Mon Sep 17 00:00:00 2001 From: Mihai Codoban Date: Wed, 8 Apr 2020 16:17:31 -0700 Subject: [PATCH 01/14] Update static graph docs --- .../static-graph-implementation-details.md | 63 +++++++++++++++++++ documentation/specs/static-graph.md | 62 +++++++++--------- 2 files changed, 91 insertions(+), 34 deletions(-) create mode 100644 documentation/specs/static-graph-implementation-details.md diff --git a/documentation/specs/static-graph-implementation-details.md b/documentation/specs/static-graph-implementation-details.md new file mode 100644 index 00000000000..eae8d97682b --- /dev/null +++ b/documentation/specs/static-graph-implementation-details.md @@ -0,0 +1,63 @@ +- [Single project isolated builds: implementation details](#single-project-isolated-builds-implementation-details) + - [Input / Output cache implementation](#input--output-cache-implementation) + - [Isolation implementation](#isolation-implementation) + - [How isolation exemption complicates everything :(](#how-isolation-exemption-complicates-everything) + +# Single project isolated builds: implementation details + + +Single project isolated builds can be achieved by providing MSBuild with input and output cache files. + +The input cache files should contain the cached results all the targets that a project calls on its references. When a project executes, it will naturally build its references via [MSBuild task](https://docs.microsoft.com/en-us/visualstudio/msbuild/msbuild-task) calls. In isolated builds, the engine, instead of executing these tasks, will serve them from the provided input caches. In an isolated project build, only the current project should be building targets. Any other referenced projects should be provided form the input caches. + +The output cache file tells MSBuild where it should serialize the results of the current project. This output cache would become an input cache for all other projects that depend on the current project. +The output cache file can be omitted in which case the build would just reuse prior results but not write out any new results. This could be useful when one wants to replay a build from previous caches. + +The presence of either input or output caches turns on [isolated build constraints](static-graph.md##single-project-isolated-builds). + +## Input / Output cache implementation + +The cache files contain the serialized state of MSBuild's [ConfigCache](https://github.com/Microsoft/msbuild/blob/master/src/Build/BackEnd/Components/Caching/ConfigCache.cs) and [ResultsCache](https://github.com/Microsoft/msbuild/blob/master/src/Build/BackEnd/Components/Caching/ResultsCache.cs). These two caches have been traditionally used by the engine to cache build results. For example, it is these caches which ensure that a target is only built once per build submission. The `ConfigCache` entries are instances of [BuildRequestConfiguration](https://github.com/microsoft/msbuild/blob/37c5a9fec416b403212a63f95f15b03dbd5e8b5d/src/Build/BackEnd/Shared/BuildRequestConfiguration.cs#L25). The `ResultsCache` entries are instances of [BuildResult](https://github.com/microsoft/msbuild/blob/37c5a9fec416b403212a63f95f15b03dbd5e8b5d/src/Build/BackEnd/Shared/BuildResult.cs#L34), which compose instances of [TargetResult](https://github.com/microsoft/msbuild/blob/37c5a9fec416b403212a63f95f15b03dbd5e8b5d/src/Build/BackEnd/Shared/TargetResult.cs#L22). + +One can view the two caches as the following mapping: `(project path, global properties) -> results`. `(project path, global properties)` is represented by a `BuildRequestConfiguration`, and the results are represented by `BuildResult` and `TargetResult`. + + +The input and output cache files have the same lifetime as the `ConfigCache` and the `ResultsCache`. The `ConfigCache` and the `ResultsCache` are owned by the [BuildManager](https://github.com/Microsoft/msbuild/blob/master/src/Build/BackEnd/BuildManager/BuildManager.cs), and their lifetimes are one `BuildManager.BeginBuild` / `BuildManager.EndBuild` session. On commandline builds, since MSBuild.exe uses one BuildManager with one BeginBuild / EndBuild session, the cache lifetime is the same as the entire process lifetime. When other processes (e.g. Visual Studio's devenv.exe) perform msbuild builds via the `BuildManager` APIs, there can be multiple build sessions in the same process. + + + +When MSBuild is loading input cache files, it has to merge multiple incoming instances of `ConfigCache` and `ResultsCache` into one instance of each. The [CacheAggregator](https://github.com/microsoft/msbuild/blob/37c5a9fec416b403212a63f95f15b03dbd5e8b5d/src/Build/BackEnd/BuildManager/CacheAggregator.cs#L13) is responsible for stitching together the pairs of deserialized `ConfigCache`/`ResultsCache` entries from each input cache file. +The following constraints are enforced during cache aggregation: +- For each input cache, `ConfigCache.Entries.Size == ResultsCache.Entries.Size` +- For each input cache, there is exactly one mapping from ConfigCache to ResultsCache (on `BuildResult.ConfigurationId` == `BuildRequestConfiguration.ConfigurationId`) +- Colliding configurations (defined as tuples of `(project path, global properties)`) get their corresponding BuildResult entries merged at the level of TargetResult entries. TargetResult conflicts are handled via the "first one wins" strategy. This is in line with vanilla msbuild's behaviour where a target tuple of `(project path, global properties, target)` gets executed only once. + +The output cache file **only contains results for additional work performed in the current BeginBuild / EndBuild session**. Entries from input caches are not transferred to the output cache. + + +Entries that make it into the output cache file are separated from entries serialized from input cache files via the use of [ConfigCacheWithOverride](https://github.com/microsoft/msbuild/blob/master/src/Build/BackEnd/Components/Caching/ConfigCacheWithOverride.cs) and [ResultsCacheWithOverride](https://github.com/microsoft/msbuild/blob/master/src/Build/BackEnd/Components/Caching/ResultsCacheWithOverride.cs). These are composite caches. Each contains two underlying caches: a cache where input caches files are loaded into (called the override cache), and a cache where new results are written into (called the current cache). Cache reads are satisified from both underlying caches (override cache is queried first, current cache is queried second). Writes are only written to the current cache, never into the override cache (isolation exemption complicates this, more on that [further down](#how-isolation-exemption-complicates-everything)). The output cache file only contains the serialized current cache, and not the override cache, thus ensuring that only newly built results are serialized in the output cache file. It is illegal for both the current cache and override cache to contain entries for the same project configuration, a constraint that is checked by the two override caches on each cache read. + +## Isolation implementation + +[Isolation constraints](static-graph.md##single-project-isolated-builds) are implemented in the Scheduler and the TaskBuilder. [TaskBuilder.ExecuteInstantiatedTask](https://github.com/microsoft/msbuild/blob/37c5a9fec416b403212a63f95f15b03dbd5e8b5d/src/Build/BackEnd/Components/RequestBuilder/TaskBuilder.cs#L743) ensures all `MSBuild` tasks are made on projects declared in `ProjectReference`. [Scheduler.CheckIfCacheMissOnReferencedProjectIsAllowedAndErrorIfNot](https://github.com/microsoft/msbuild/blob/37c5a9fec416b403212a63f95f15b03dbd5e8b5d/src/Build/BackEnd/Components/Scheduler/Scheduler.cs#L1818) ensures all `MSBuild` tasks are cache hits. + +### How isolation exemption complicates everything :( + +Project references [can be exempt](static-graph.md#exempting-references-from-isolation-constraints) from isolation constraints via the `GraphIsolationExemptReference` item. + +The `Scheduler` knows to skip isolation constraints on an exempt `BuildRequest` because the [ProjectBuilder compares](https://github.com/microsoft/msbuild/blob/37c5a9fec416b403212a63f95f15b03dbd5e8b5d/src/Build/BackEnd/Components/RequestBuilder/RequestBuilder.cs#L349) each new `BuildRequest` against the `GraphIsolationExemptReference` items defined in the calling project, and if exempt, sets `BuildRequest.SkipStaticGraphIsolationConstraints`. When a `BuildRequest` is marked as exempt, the `Scheduler` also marks its corresponding `BuildRequestConfiguration` as exempt as well, which aids in further identification of exempt projects outside the `Scheduler`. + +The build results for the exempt project are also included in the current cache (according to the above mentioned rule that newly built results are serialized in the output cache file). This complicates the way caches interact, due to multiple reasons: +1. the same project can be exempt by several references, thus potentially colliding when multiple output cache files containing the same exempt project get aggregated. For example, given the graph `{A->B, A->C}`, where both `B` and `C` build the exempt project `D`, `D` will appear in the output caches of both `B` and `C`. When `A` aggregates these two caches, it will encounter duplicate entries for `D`, and will need to merge the results. +2. a project can be both exempt and present in the graph at the same time. For example, given the graph `{A->B}`, both `A` and `B` are in the graph, but `A` can also mark `B` as exempt (meaning that `A` contains both a `ProjectReference` item to `B`, and a `GraphIsolationExemptReference` item to `B`). The fact that `B` is in the graph means that `A` will receive an input cache containing B's build results. There are two subcases here: + 1. `A` builds targets from `B` that already exist in the input cache file from `B`. In this case, all the builds of `B` will be cache hits, and no target results from `B` will make it into `A`'s output cache, since nothing new was built. + 2. `A` builds targets from `B` that do not exist in the input cache file from `B`. If `B` weren't exempt from isolation constraints, this scenario would lead to a build break, as cache misses are illegal under isolation. With `B` being exempt, the new builds of `B` will get included in `A`'s output cache. The results from `B`'s cache file won't get included in `A`'s output cache file, as they weren't built by `A`. +3. A project, which is not in the graph, can be exempt by two parent/child projects from the graph. For example, given the graph `{A->B}`, both `A` and `B` can exempt project `D` (meaning that neither `A` nor `B` have a `ProjectReference` to `D`, but both `A` and `B` have a `GraphIsolationExemptReference` to `D`). The fact that `B` is in the graph means that `A` will receive an input cache containing `B`'s build results. Since `B` builds targets from `D`, it means that `B`'s output cache file also contains target results from `D`. There are two subcases here: + 1. `A` builds targets from `D` that already exist in the input cache file from `B`. This is handled in the same way as the above case `2.1.` + 2. `A` builds targets from `D` that do not exist in the input cache file from `B`, meaning that `A` builds additional targets from `D` which `B` didn't build. This is handled in the same way as teh above case `2.2.` + + + +Cases `2.2.` and `3.2.` complicate the requirement that the output cache should only contain newly built targets, and complicate the desirable goal that the override cache should never be mutated. In these cases initial entries (`BuildRequestConfiguration` for the `ConfigCache` and `BuildResult` / `TargetResult` for the `ResultsCache`) are already loaded in the override cache from previous builds, but then additional new builds on the isolation exempt entries need to be migrated / promoted to the current cache. This promotion is achieved differently for configs and build results: +- `ConfigCache` entries [are moved](https://github.com/cdmihai/msbuild/blob/37c5a9fec416b403212a63f95f15b03dbd5e8b5d/src/Build/BackEnd/Components/Caching/ConfigCacheWithOverride.cs#L178) from the override cache into the current cache whenever a corresponding `BuildResult` is written into `BuildResultsWithOverride` (`BuildResult.ConfigurationId` == `BuildRequestConfiguration.ConfigurationId`). +- `BuildResult` / `TargetResult` entries are trickier. Sadly, the engine has a deep dependency on mutating existing results entries, so it's not possible to migrate result entries like config entries. Once the engine has obtained a reference to a result entry from the override cache, it will mutate it. In this particular case, [the BuildResultsWithOverride cache waives the requirement of non overlapping caches](https://github.com/cdmihai/msbuild/blob/37c5a9fec416b403212a63f95f15b03dbd5e8b5d/src/Build/BackEnd/Components/Caching/ResultsCacheWithOverride.cs#L139), and the new results are written both in the override caches (alongside the result entries deserialized from input caches), and also in the current cache. Thus, the override results cache will contain all build results, both old and new, while the current cache will contain only the new build results executed by the current build session. \ No newline at end of file diff --git a/documentation/specs/static-graph.md b/documentation/specs/static-graph.md index 8b899667790..ee683a2c3bb 100644 --- a/documentation/specs/static-graph.md +++ b/documentation/specs/static-graph.md @@ -12,9 +12,9 @@ - [Isolated builds](#isolated-builds) - [Isolated graph builds](#isolated-graph-builds) - [Single project isolated builds](#single-project-isolated-builds) - - [Single project isolated builds: implementation details](#single-project-isolated-builds-implementation-details) - [APIs](#apis) - [Command line](#command-line) + - [Exempting references from isolation constraints](#exempting-references-from-isolation-constraints) - [I/O Tracking](#io-tracking) - [Detours](#detours) - [Isolation requirement](#isolation-requirement) @@ -297,56 +297,50 @@ Because referenced projects and their entry targets are guaranteed to be in the ### Isolated graph builds When building a graph in isolated mode, the graph is used to traverse and build the projects in the right order, but each individual project is built in isolation. The build result cache will just be in memory exactly as it is today, but on cache miss it will error. This enforces that both the graph and target mappings are complete and correct. -Furthermore, running in this mode enforces that each (project, global properties) pair is executed only once and must execute all targets needed by all projects which reference that node. This gives it a concrete start and end time, which leads to some perf optimizations, like garbage collecting all project state (except the build results) once it finishes building. This can greatly reduce the memory overhead for large builds. +Furthermore, running in this mode enforces that each (project, global properties) pair is executed only once and must execute all targets needed by all projects which reference that node. This gives it a concrete start and end time, which leads to some potential perf optimizations, like garbage collecting all project state (except the build results) once it finishes building. This can greatly reduce the memory overhead for large builds. This discrete start and end time also allows for easy integration with [I/O Tracking](#io-tracking) to observe all inputs and outputs for a project. Note however that I/O during target execution, particular target execution which may not normally happen as part of a project's individual build execution, would be attributed to the project reference project rather the project with the project reference. This differs from today's behavior, but seems like a desirable difference anyway. ### Single project isolated builds When building a single project in isolation, all project references' build results must be provided to the project externally. Specifically, the results will need to be [deserialized](#deserialization) from files and loaded into the build result cache in memory. -Because of this, single project isolated builds is quite restrictive and is not intended to be used directly by end-users. Instead the scenario is intended for higher-order build engines which support caching and [distribution](#distribution). - -There is also the possibility for these higher-order build engines and even Visual Studio to enable extremely fast incremental builds for a project. For example, when all project references' build results are provided (and validated as up to date by that higher-order build engine), there is no need to evaluate or execute any targets on any other project. - -These incremental builds can even be extended to multiple projects by keeping a project graph in memory as well as the last build result for each node and whether that build result is valid. The higher-order build engine can then itself traverse the graph and do single project isolated builds for projects which are not currently up to date. - -### Single project isolated builds: implementation details +When MSBuild runs in isolation mode, it fails the build when it detects: +1. `MSBuild` task calls which cannot be served from the cache. Cache misses are illegal. +2. `MSBuild` task calls to project files which were not defined in the `ProjectReference` item. - -Single project builds can be achieved by providing MSBuild with input and output cache files. +Because of this, single project isolated builds is quite restrictive and is not intended to be used directly by end-users. Instead the scenario is intended for higher-order build engines which support caching and [distribution](#distribution). -The input cache files contain the cached results of all of the current project's references. This way, when the current project executes, it will naturally build its references via [MSBuild task](https://docs.microsoft.com/en-us/visualstudio/msbuild/msbuild-task) calls. The engine, instead of executing these tasks, will serve them from the provided input caches. +There is also the possibility for these higher-order build engines and even Visual Studio to enable faster incremental builds for a project. For example, when a project's references' build results are provided via file caches (and validated as up to date by that higher-order build engine), there is no need to evaluate or execute any targets for any reference. -The output cache file tells MSBuild where it should serialize the results of the current project. This output cache would become an input cache for all other projects that depend on the current project. -The output cache file can be ommited in which case the build would just reuse prior results but not write out any new results. This could be useful when one wants to replay a build from previous caches. +These incremental builds could be extended to the entire graph by keeping a project graph in memory as well as the last build result cache files for each node and whether a node's results are up to date. The higher-order build engine can then itself traverse the graph and do single project isolated builds only for projects which are not currently up to date. -The presence of either input or output caches turns on the isolated build constraints. The engine will fail the build on: -- `MSBuild` task calls to project files which were not defined in the `ProjectReference` item at evaluation time. -- `MSBuild` task calls which cannot be served from the cache +#### APIs +Cache file information is provided via [BuildParameters](https://github.com/Microsoft/msbuild/blob/2d4dc592a638b809944af10ad1e48e7169e40808/src/Build/BackEnd/BuildManager/BuildParameters.cs#L746-L764). Input caches are applied in `BuildManager.BeginBuild`. Output cache files are written in `BuildManager.EndBuild`. Thus, the scope of the caches are one BuildManager BeginBuild/EndBuild session. - -These cache files contain the serialized state of MSBuild's [ConfigCache](https://github.com/Microsoft/msbuild/blob/master/src/Build/BackEnd/Components/Caching/ConfigCache.cs) and [ResultsCache](https://github.com/Microsoft/msbuild/blob/master/src/Build/BackEnd/Components/Caching/ResultsCache.cs). These two caches have been traditionally used by the engine to cache build results. +Isolation constraints are turned on via [BuildParameters.IsolateProjects](https://github.com/microsoft/msbuild/blob/b111470ae61eba02c6102374c2b7d62aebe45f5b/src/Build/BackEnd/BuildManager/BuildParameters.cs#L742). Isolation constraints are also automatically turned on if either input or output cache files are used. -Cache structure: `(project path, global properties) -> results` +#### Command line +Caches are provided to MSBuild.exe via the multi value `/inputResultsCaches` and the single value `/outputResultsCache`. +Isolation constraints are turned on via `/isolate` (they are also implicitly activated when either input or output caches are used). - -The caches are applicable for the entire duration of the MSBuild.exe process. The input and output caches have the same lifetime as the `ConfigCache` and the `ResultsCache`. The `ConfigCache` and the `ResultsCache` are owned by the [BuildManager](https://github.com/Microsoft/msbuild/blob/master/src/Build/BackEnd/BuildManager/BuildManager.cs), and their lifetimes are one `BuildManager.BeginBuild` / `BuildManager.EndBuild` session. Since MSBuild.exe uses one BuildManager with one BeginBuild / EndBuild session, the cache lifetime is the same as the entire process lifetime. +#### Exempting references from isolation constraints +In certain situations one may want to exempt a reference from isolation constraints. A few potential cases: +- debugging / onboarding to isolation constraints +- exempting references whose project files are generated at build times with random names (for example, each WPF project, before the Build target, generates and builds a helper .csproj with a random file name) +- relaxing constraints for MSBuild task calling patterns that static graph cannot express (for exemple, if a project is calculating references, or the targets to call on references, at runtime via an arbitrary algorithm) - -The following are cache file constraints enforced by the engine. +A project is exempt from isolation constraints by adding its full path to the `GraphIsolationExemptReference` item. For example, if project A.csproj references project B.csproj, the following snippet exempts B.csproj from isolation constraints while A.csproj is built: +```xml + + + +``` -Input cache files constraints: -- A ConfigCache / ResultsCache mapping must be unique between all input caches (multiple input caches cannot provide information for the same cache entry) -- For each input cache, `ConfigCache.Entries.Size == ResultsCache.Entries.Size` -- For each input cache, there is exactly one mapping from ConfigCache to ResultsCache +A reference is exempt only in projects that add the reference in `GraphIsolationExemptReference`. If multiple projects need to exempt the same reference, all of them need to add the reference to `GraphIsolationExemptReference`. -Output cache file constraints: -- the output cache file contains results only for additional work performed in the current BeginBuild / EndBuild session. Entries from input caches are not transferred to the output cache. +For now, self builds (a project building itself with different global properties) are also exempt from isolation constraints, but this behaviour is of dubious value and might be changed in the future. -#### APIs -Caches are provided via [BuildParameters](https://github.com/Microsoft/msbuild/blob/2d4dc592a638b809944af10ad1e48e7169e40808/src/Build/BackEnd/BuildManager/BuildParameters.cs#L746-L764). They are applied in `BuildManager.BeginBuild` -#### Command line -Caches are provided to MSBuild.exe via the multi value `/inputResultsCaches` and the single value `/outputResultsCache`. +Details on how isolation and cache files are implemented in MSBuild can be found [here](./static-graph-implementation-details.md). ## I/O Tracking To help facilitate caching of build outputs by a higher-order build engine, MSBuild needs to track all I/O that happens as part of a build. From d39951116656235efe4255e81fbe2f858e31b4bc Mon Sep 17 00:00:00 2001 From: Mihai Codoban Date: Wed, 15 Apr 2020 18:35:15 -0700 Subject: [PATCH 02/14] multitargeting -> crosstargeting --- documentation/specs/static-graph.md | 50 ++++++++++++++--------------- 1 file changed, 25 insertions(+), 25 deletions(-) diff --git a/documentation/specs/static-graph.md b/documentation/specs/static-graph.md index ee683a2c3bb..34d45ba7e8a 100644 --- a/documentation/specs/static-graph.md +++ b/documentation/specs/static-graph.md @@ -3,10 +3,10 @@ - [Motivations](#motivations) - [Project Graph](#project-graph) - [Build dimensions](#build-dimensions) - - [Multitargeting](#multitargeting) + - [Crosstargeting](#crosstargeting) - [Building a project graph](#building-a-project-graph) - [Inferring which targets to run for a project within the graph](#inferring-which-targets-to-run-for-a-project-within-the-graph) - - [Multitargeting details](#multitargeting-details) + - [Crosstargeting details](#crosstargeting-details) - [Underspecified graphs](#underspecified-graphs) - [Public API](#public-api) - [Isolated builds](#isolated-builds) @@ -51,37 +51,37 @@ The graph also supports multiple entry points, so this enables scenarios where p For example, if project A had a project reference to project B with `GlobalPropertiesToRemove=Platform`, and we wanted to build project A for x86 and x64 so used both as entry points, the graph would consist of 3 nodes: project A with `Platform=x86`, project A with `Platform=x64`, and project B with no global properties set. -#### Multitargeting +#### Crosstargeting -Multitargeting refers to projects that specify multiple build dimensions applicable to themselves. For example, `Microsoft.Net.Sdk` based projects can target multiple target frameworks (e.g. `net472;netcoreapp2.2`). As discussed, build dimensions are expressed as global properties. Let's call the global properties that define the multitargeting set as the multitargeting global properties. +Crosstargeting refers to projects that specify multiple build dimensions applicable to themselves. For example, `Microsoft.Net.Sdk` based projects can target multiple target frameworks (e.g. `net472;netcoreapp2.2`). As discussed, build dimensions are expressed as global properties. Let's call the global properties that define the crosstargeting set as the crosstargeting global properties. -Multitargeting is implemented by having a project reference itself multiple times, once for each combination of multitargeting global properties. This leads to multiple evaluations of the same project, with different global properties. These evaluations can be classified in two groups -1. Multiple inner builds. Each inner build is evaluated with one set of multitargeting global properties (e.g. the `TargetFramework=net472` inner build, or the `TargetFramework=netcoreapp2.2` inner build). -2. One outer build. This evaluation does not have any multitargeting global properties set. It can be viewed as a proxy for the inner builds. Other projects query the outer build in order to learn the set of valid multitargeting global properties (the set of valid inner builds). When the outer build is also the root of the project to project graph, the outer build multicasts the entry target (i.e. `Build`, `Clean`, etc) to all inner builds. +Crosstargeting is implemented by having a project reference itself multiple times, once for each combination of crosstargeting global properties. This leads to multiple evaluations of the same project, with different global properties. These evaluations can be classified in two groups +1. Multiple inner builds. Each inner build is evaluated with one set of crosstargeting global properties (e.g. the `TargetFramework=net472` inner build, or the `TargetFramework=netcoreapp2.2` inner build). +2. One outer build. This evaluation does not have any crosstargeting global properties set. It can be viewed as a proxy for the inner builds. Other projects query the outer build in order to learn the set of valid crosstargeting global properties (the set of valid inner builds). When the outer build is also the root of the project to project graph, the outer build multicasts the entry target (i.e. `Build`, `Clean`, etc) to all inner builds. -In order for the graph to represent inner and outer builds as nodes, it imposes a contract on what multitargeting means, and requires the multitargeting supporting SDKs to implement this contract. +In order for the graph to represent inner and outer builds as nodes, it imposes a contract on what crosstargeting means, and requires the crosstargeting supporting SDKs to implement this contract. -Multitargeting supporting SDKs MUST implement the following properties and semantics: -- `InnerBuildProperty`. It contains the property name that defines the multitargeting build dimension. +Crosstargeting supporting SDKs MUST implement the following properties and semantics: +- `InnerBuildProperty`. It contains the property name that defines the crosstargeting build dimension. - `InnerBuildPropertyValues`. It contains the property name that holds the possible values for the `InnerBuildProperty`. - Project classification: - *Outer build*, when `$($(InnerBuildProperty))` is empty AND `$($(InnerBuildPropertyValues))` is not empty. - *Dependent inner build*, when both `$($(InnerBuildProperty))` and `$($(InnerBuildPropertyValues))` are non empty. These are inner builds that were generated from an outer build. - *Standalone inner build*, when `$($(InnerBuildProperty))` is not empty and `$($(InnerBuildPropertyValues))` is empty. These are inner builds that were not generated from an outer build. - - *Non multitargeting build*, when both `$($(InnerBuildProperty))` and `$($(InnerBuildPropertyValues))` are empty. + - *Non crosstargeting build*, when both `$($(InnerBuildProperty))` and `$($(InnerBuildPropertyValues))` are empty. - Node edges - - When project A references multitargeting project B, and B is identified as an outer build, the graph node for project A will reference both the outer build of B, and all the inner builds of B. The edges to the inner builds are speculative, as at build time only one inner build gets referenced. However, the graph cannot know at evaluation time which inner build will get chosen. - - When multitargeting project B is a root, then the outer build node for B will reference the inner builds of B. - - For multitargeting projects, the `ProjectReference` item gets applied only to inner builds. An outer build cannot have its own distinct `ProjectReference`s, it is the inner builds that reference other project files, not the outer build. This constraint might get relaxed in the future via additional configuration, to allow outer build specific references. + - When project A references crosstargeting project B, and B is identified as an outer build, the graph node for project A will reference both the outer build of B, and all the inner builds of B. The edges to the inner builds are speculative, as at build time only one inner build gets referenced. However, the graph cannot know at evaluation time which inner build will get chosen. + - When crosstargeting project B is a root, then the outer build node for B will reference the inner builds of B. + - For crosstargeting projects, the `ProjectReference` item gets applied only to inner builds. An outer build cannot have its own distinct `ProjectReference`s, it is the inner builds that reference other project files, not the outer build. This constraint might get relaxed in the future via additional configuration, to allow outer build specific references. -These specific rules represent the minimal rules required to represent multitargeting in `Microsoft.Net.Sdk`. As we adopt SDKs whose multitargeting complexity that cannot be expressed with the above rules, we'll extend the rules. -For example, `InnerBuildProperty` could become `InnerBuildProperties` for SDKs where there's multiple multitargeting global properties. +These specific rules represent the minimal rules required to represent crosstargeting in `Microsoft.Net.Sdk`. As we adopt SDKs whose crosstargeting complexity that cannot be expressed with the above rules, we'll extend the rules. +For example, `InnerBuildProperty` could become `InnerBuildProperties` for SDKs where there's multiple crosstargeting global properties. -For example, here is a trimmed down `Microsoft.Net.Sdk` multitargeting project: +For example, here is a trimmed down `Microsoft.Net.Sdk` crosstargeting project: ```xml @@ -98,7 +98,7 @@ For example, here is a trimmed down `Microsoft.Net.Sdk` multitargeting project: ``` To summarize, there are two main patterns for build dimensions which are handled: -1. The project multitargets, in which case the SDK needs to specify the multitargeting build dimensions. +1. The project crosstargets, in which case the SDK needs to specify the crosstargeting build dimensions. 2. A different set of global properties are used to choose the dimension like with Configuration or Platform. The project graph supports this via multiple entry points. ### Building a project graph @@ -183,23 +183,23 @@ We'll represent the project reference protocols as `ProjectReferenceTargets` ite ``` -#### Multitargeting details +#### Crosstargeting details -A multitargeting project can get called with different targets for the outer build and the inner builds. In this case, the `ProjectReferenceTargets` items containing targets for the outer build are marked with the `OuterBuild=true` metadata. Here are the rules for how targets from `ProjectReferenceTargets` get assigned to different project types: +A crosstargeting project can get called with different targets for the outer build and the inner builds. In this case, the `ProjectReferenceTargets` items containing targets for the outer build are marked with the `OuterBuild=true` metadata. Here are the rules for how targets from `ProjectReferenceTargets` get assigned to different project types: - *Outer build*: targets with `OuterBuild=true` metadata - *Dependent inner build*: targets without `OuterBuild=true` metadata - - *Standalone inner build*: the same as non multitargeting builds. - - *Non multitargeting build*: concatenation of targets with `OuterBuild=true` metadata and targets without `OuterBuild=true` metadata + - *Standalone inner build*: the same as non crosstargeting builds. + - *Non crosstargeting build*: concatenation of targets with `OuterBuild=true` metadata and targets without `OuterBuild=true` metadata **OPEN ISSUE:** Current implementation does not disambiguate between the two types of inner builds, leading to overbuilding certain targets by conservatively treating both inner build types as standalone inner builds. -For example, consider the graph of `A (non multitargeting) -> B (multitargeting with 2 innerbuilds) -> C (standalone inner build)`, with the following target propagation rules: +For example, consider the graph of `A (non crosstargeting) -> B (crosstargeting with 2 innerbuilds) -> C (standalone inner build)`, with the following target propagation rules: ``` A -> Ao when OuterBuild=true A -> Ai, A ``` -According to the graph construction rules defined in the [multitargeting section](#multitargeting), we get the following graph, annotated with the target propagation for target `A`. +According to the graph construction rules defined in the [crosstargeting section](#crosstargeting), we get the following graph, annotated with the target propagation for target `A`. ``` A+-->ProjA @@ -290,7 +290,7 @@ namespace Microsoft.Build.Experimental.Graph ## Isolated builds Building a project in isolation means that any build results for project references must be pre-computed and provided as input. -If a project uses the MSBuild task, the build result must be in MSBuild's build result cache instead of just-in-time executing targets on that referenced project. If it is not in the build result cache, an error will be logged and the build will fail. If the project is calling into itself either via `CallTarget` or the MSBuild task with a different set of global properties, this will be allowed to support multitargeting and other build dimensions implemented in a similar way. +If a project uses the MSBuild task, the build result must be in MSBuild's build result cache instead of just-in-time executing targets on that referenced project. If it is not in the build result cache, an error will be logged and the build will fail. If the project is calling into itself either via `CallTarget` or the MSBuild task with a different set of global properties, this will be allowed to support crosstargeting and other build dimensions implemented in a similar way. Because referenced projects and their entry targets are guaranteed to be in the cache, they will not build again. Therefore we do not need to set `/p:BuildProjectReferences=false` or any other gesture that tells SDKs to not do recursive operations. From 9487df8df8592b9faaa8f7a0993a4780d8ba7468 Mon Sep 17 00:00:00 2001 From: Mihai Codoban Date: Wed, 15 Apr 2020 18:44:30 -0700 Subject: [PATCH 03/14] detail project graph construction and target execution --- documentation/specs/static-graph.md | 22 ++++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-) diff --git a/documentation/specs/static-graph.md b/documentation/specs/static-graph.md index 34d45ba7e8a..a73488a3fb7 100644 --- a/documentation/specs/static-graph.md +++ b/documentation/specs/static-graph.md @@ -2,9 +2,12 @@ - [Overview](#overview) - [Motivations](#motivations) - [Project Graph](#project-graph) + - [Constructing the project graph](#constructing-the-project-graph) - [Build dimensions](#build-dimensions) - [Crosstargeting](#crosstargeting) - - [Building a project graph](#building-a-project-graph) + - [Executing targets on a graph](#executing-targets-on-a-graph) + - [Command line](#command-line) + - [APIs](#apis) - [Inferring which targets to run for a project within the graph](#inferring-which-targets-to-run-for-a-project-within-the-graph) - [Crosstargeting details](#crosstargeting-details) - [Underspecified graphs](#underspecified-graphs) @@ -12,8 +15,8 @@ - [Isolated builds](#isolated-builds) - [Isolated graph builds](#isolated-graph-builds) - [Single project isolated builds](#single-project-isolated-builds) - - [APIs](#apis) - - [Command line](#command-line) + - [APIs](#apis-1) + - [Command line](#command-line-1) - [Exempting references from isolation constraints](#exempting-references-from-isolation-constraints) - [I/O Tracking](#io-tracking) - [Detours](#detours) @@ -34,11 +37,15 @@ ## Project Graph +### Constructing the project graph Calculating the project graph will be very similar to the MS internal build engine's existing Traversal logic. For a given evaluated project, all project references will be identified and recursively evaluated (with deduping). Project references are identified via the `ProjectReference` item. A node in the graph is a tuple of the project file and global properties. Each (project, global properties) combo can be evaluated in parallel. +Transitive project references are opt-in per project. Once a project opts-in, transitivity is applied for all ProjectReference items. +A project opt-ins by setting the property `AddTransitiveProjectReferencesInStaticGraph` to true. + ### Build dimensions Build dimensions can be thought of as different ways to build a particular project. For example, a project can be built Debug or Retail, x86 or x64, for .NET Framework 4.7.1 or .NET Core 2.0. @@ -101,7 +108,7 @@ To summarize, there are two main patterns for build dimensions which are handled 1. The project crosstargets, in which case the SDK needs to specify the crosstargeting build dimensions. 2. A different set of global properties are used to choose the dimension like with Configuration or Platform. The project graph supports this via multiple entry points. -### Building a project graph +### Executing targets on a graph When building a graph, project references should be built before the projects that reference them, as opposed to the existing msbuild scheduler which builds projects just in time. For example if project A depends on project B, then project B should build first, then project A. Existing msbuild scheduling would start building project A, reach an MSBuild task for project B, yield project A, build project B, then resume project A once unblocked. @@ -110,6 +117,13 @@ Building in this way should make better use of parallelism as all CPU cores can Note that graph cycles are disallowed, even if they're using disconnected targets. This is a breaking change, as today you can have two projects where each project depends on a target from the other project, but that target doesn't depend on the default target or anything in its target graph. +#### Command line +`msbuild /graph` - msbuild will create a static graph from the entry point project and build it in topological order with the specified targets. Targets to call on each node are inferred via the rules in [this section](#inferring-which-targets-to-run-for-a-project-within-the-graph). + +#### APIs + +[BuildManager.PendBuildRequest(GraphBuildRequestData requestData)](https://github.com/microsoft/msbuild/blob/37c5a9fec416b403212a63f95f15b03dbd5e8b5d/src/Build/BackEnd/BuildManager/BuildManager.cs#L676) + ### Inferring which targets to run for a project within the graph In the classic traversal, the referencing project chooses which targets to call on the referenced projects and may call into a project multiple times with different target lists and global properties (examples in [project reference protocol](../ProjectReference-Protocol.md)). When building a graph, where projects are built before the projects that reference them, we have to determine the target list to execute on each project statically. From 76f0867ed601bdf997736feb7f4f3e56214fd232 Mon Sep 17 00:00:00 2001 From: Mihai Codoban Date: Wed, 29 Apr 2020 19:45:11 -0700 Subject: [PATCH 04/14] crosstarget -> multitarget --- documentation/specs/static-graph.md | 50 ++++++++++++++--------------- 1 file changed, 25 insertions(+), 25 deletions(-) diff --git a/documentation/specs/static-graph.md b/documentation/specs/static-graph.md index a73488a3fb7..38c9802112a 100644 --- a/documentation/specs/static-graph.md +++ b/documentation/specs/static-graph.md @@ -4,12 +4,12 @@ - [Project Graph](#project-graph) - [Constructing the project graph](#constructing-the-project-graph) - [Build dimensions](#build-dimensions) - - [Crosstargeting](#crosstargeting) + - [Multitargeting](#multitargeting) - [Executing targets on a graph](#executing-targets-on-a-graph) - [Command line](#command-line) - [APIs](#apis) - [Inferring which targets to run for a project within the graph](#inferring-which-targets-to-run-for-a-project-within-the-graph) - - [Crosstargeting details](#crosstargeting-details) + - [Multitargeting details](#multitargeting-details) - [Underspecified graphs](#underspecified-graphs) - [Public API](#public-api) - [Isolated builds](#isolated-builds) @@ -58,37 +58,37 @@ The graph also supports multiple entry points, so this enables scenarios where p For example, if project A had a project reference to project B with `GlobalPropertiesToRemove=Platform`, and we wanted to build project A for x86 and x64 so used both as entry points, the graph would consist of 3 nodes: project A with `Platform=x86`, project A with `Platform=x64`, and project B with no global properties set. -#### Crosstargeting +#### Multitargeting -Crosstargeting refers to projects that specify multiple build dimensions applicable to themselves. For example, `Microsoft.Net.Sdk` based projects can target multiple target frameworks (e.g. `net472;netcoreapp2.2`). As discussed, build dimensions are expressed as global properties. Let's call the global properties that define the crosstargeting set as the crosstargeting global properties. +Multitargeting refers to projects that specify multiple build dimensions applicable to themselves. For example, `Microsoft.Net.Sdk` based projects can target multiple target frameworks (e.g. `net472;netcoreapp2.2`). As discussed, build dimensions are expressed as global properties. Let's call the global properties that define the multitargeting set as the multitargeting global properties. -Crosstargeting is implemented by having a project reference itself multiple times, once for each combination of crosstargeting global properties. This leads to multiple evaluations of the same project, with different global properties. These evaluations can be classified in two groups -1. Multiple inner builds. Each inner build is evaluated with one set of crosstargeting global properties (e.g. the `TargetFramework=net472` inner build, or the `TargetFramework=netcoreapp2.2` inner build). -2. One outer build. This evaluation does not have any crosstargeting global properties set. It can be viewed as a proxy for the inner builds. Other projects query the outer build in order to learn the set of valid crosstargeting global properties (the set of valid inner builds). When the outer build is also the root of the project to project graph, the outer build multicasts the entry target (i.e. `Build`, `Clean`, etc) to all inner builds. +Multitargeting is implemented by having a project reference itself multiple times, once for each combination of multitargeting global properties. This leads to multiple evaluations of the same project, with different global properties. These evaluations can be classified in two groups +1. Multiple inner builds. Each inner build is evaluated with one set of multitargeting global properties (e.g. the `TargetFramework=net472` inner build, or the `TargetFramework=netcoreapp2.2` inner build). +2. One outer build. This evaluation does not have any multitargeting global properties set. It can be viewed as a proxy for the inner builds. Other projects query the outer build in order to learn the set of valid multitargeting global properties (the set of valid inner builds). When the outer build is also the root of the project to project graph, the outer build multicasts the entry target (i.e. `Build`, `Clean`, etc) to all inner builds. -In order for the graph to represent inner and outer builds as nodes, it imposes a contract on what crosstargeting means, and requires the crosstargeting supporting SDKs to implement this contract. +In order for the graph to represent inner and outer builds as nodes, it imposes a contract on what multitargeting means, and requires the multitargeting supporting SDKs to implement this contract. -Crosstargeting supporting SDKs MUST implement the following properties and semantics: -- `InnerBuildProperty`. It contains the property name that defines the crosstargeting build dimension. +Multitargeting supporting SDKs MUST implement the following properties and semantics: +- `InnerBuildProperty`. It contains the property name that defines the multitargeting build dimension. - `InnerBuildPropertyValues`. It contains the property name that holds the possible values for the `InnerBuildProperty`. - Project classification: - *Outer build*, when `$($(InnerBuildProperty))` is empty AND `$($(InnerBuildPropertyValues))` is not empty. - *Dependent inner build*, when both `$($(InnerBuildProperty))` and `$($(InnerBuildPropertyValues))` are non empty. These are inner builds that were generated from an outer build. - *Standalone inner build*, when `$($(InnerBuildProperty))` is not empty and `$($(InnerBuildPropertyValues))` is empty. These are inner builds that were not generated from an outer build. - - *Non crosstargeting build*, when both `$($(InnerBuildProperty))` and `$($(InnerBuildPropertyValues))` are empty. + - *Non multitargeting build*, when both `$($(InnerBuildProperty))` and `$($(InnerBuildPropertyValues))` are empty. - Node edges - - When project A references crosstargeting project B, and B is identified as an outer build, the graph node for project A will reference both the outer build of B, and all the inner builds of B. The edges to the inner builds are speculative, as at build time only one inner build gets referenced. However, the graph cannot know at evaluation time which inner build will get chosen. - - When crosstargeting project B is a root, then the outer build node for B will reference the inner builds of B. - - For crosstargeting projects, the `ProjectReference` item gets applied only to inner builds. An outer build cannot have its own distinct `ProjectReference`s, it is the inner builds that reference other project files, not the outer build. This constraint might get relaxed in the future via additional configuration, to allow outer build specific references. + - When project A references multitargeting project B, and B is identified as an outer build, the graph node for project A will reference both the outer build of B, and all the inner builds of B. The edges to the inner builds are speculative, as at build time only one inner build gets referenced. However, the graph cannot know at evaluation time which inner build will get chosen. + - When multitargeting project B is a root, then the outer build node for B will reference the inner builds of B. + - For multitargeting projects, the `ProjectReference` item gets applied only to inner builds. An outer build cannot have its own distinct `ProjectReference`s, it is the inner builds that reference other project files, not the outer build. This constraint might get relaxed in the future via additional configuration, to allow outer build specific references. -These specific rules represent the minimal rules required to represent crosstargeting in `Microsoft.Net.Sdk`. As we adopt SDKs whose crosstargeting complexity that cannot be expressed with the above rules, we'll extend the rules. -For example, `InnerBuildProperty` could become `InnerBuildProperties` for SDKs where there's multiple crosstargeting global properties. +These specific rules represent the minimal rules required to represent multitargeting in `Microsoft.Net.Sdk`. As we adopt SDKs whose multitargeting complexity that cannot be expressed with the above rules, we'll extend the rules. +For example, `InnerBuildProperty` could become `InnerBuildProperties` for SDKs where there's multiple multitargeting global properties. -For example, here is a trimmed down `Microsoft.Net.Sdk` crosstargeting project: +For example, here is a trimmed down `Microsoft.Net.Sdk` multitargeting project: ```xml @@ -105,7 +105,7 @@ For example, here is a trimmed down `Microsoft.Net.Sdk` crosstargeting project: ``` To summarize, there are two main patterns for build dimensions which are handled: -1. The project crosstargets, in which case the SDK needs to specify the crosstargeting build dimensions. +1. The project multitargets, in which case the SDK needs to specify the multitargeting build dimensions. 2. A different set of global properties are used to choose the dimension like with Configuration or Platform. The project graph supports this via multiple entry points. ### Executing targets on a graph @@ -197,23 +197,23 @@ We'll represent the project reference protocols as `ProjectReferenceTargets` ite ``` -#### Crosstargeting details +#### Multitargeting details -A crosstargeting project can get called with different targets for the outer build and the inner builds. In this case, the `ProjectReferenceTargets` items containing targets for the outer build are marked with the `OuterBuild=true` metadata. Here are the rules for how targets from `ProjectReferenceTargets` get assigned to different project types: +A multitargeting project can get called with different targets for the outer build and the inner builds. In this case, the `ProjectReferenceTargets` items containing targets for the outer build are marked with the `OuterBuild=true` metadata. Here are the rules for how targets from `ProjectReferenceTargets` get assigned to different project types: - *Outer build*: targets with `OuterBuild=true` metadata - *Dependent inner build*: targets without `OuterBuild=true` metadata - - *Standalone inner build*: the same as non crosstargeting builds. - - *Non crosstargeting build*: concatenation of targets with `OuterBuild=true` metadata and targets without `OuterBuild=true` metadata + - *Standalone inner build*: the same as non multitargeting builds. + - *Non multitargeting build*: concatenation of targets with `OuterBuild=true` metadata and targets without `OuterBuild=true` metadata **OPEN ISSUE:** Current implementation does not disambiguate between the two types of inner builds, leading to overbuilding certain targets by conservatively treating both inner build types as standalone inner builds. -For example, consider the graph of `A (non crosstargeting) -> B (crosstargeting with 2 innerbuilds) -> C (standalone inner build)`, with the following target propagation rules: +For example, consider the graph of `A (non multitargeting) -> B (multitargeting with 2 innerbuilds) -> C (standalone inner build)`, with the following target propagation rules: ``` A -> Ao when OuterBuild=true A -> Ai, A ``` -According to the graph construction rules defined in the [crosstargeting section](#crosstargeting), we get the following graph, annotated with the target propagation for target `A`. +According to the graph construction rules defined in the [multitargeting section](#multitargeting), we get the following graph, annotated with the target propagation for target `A`. ``` A+-->ProjA @@ -304,7 +304,7 @@ namespace Microsoft.Build.Experimental.Graph ## Isolated builds Building a project in isolation means that any build results for project references must be pre-computed and provided as input. -If a project uses the MSBuild task, the build result must be in MSBuild's build result cache instead of just-in-time executing targets on that referenced project. If it is not in the build result cache, an error will be logged and the build will fail. If the project is calling into itself either via `CallTarget` or the MSBuild task with a different set of global properties, this will be allowed to support crosstargeting and other build dimensions implemented in a similar way. +If a project uses the MSBuild task, the build result must be in MSBuild's build result cache instead of just-in-time executing targets on that referenced project. If it is not in the build result cache, an error will be logged and the build will fail. If the project is calling into itself either via `CallTarget` or the MSBuild task with a different set of global properties, this will be allowed to support multitargeting and other build dimensions implemented in a similar way. Because referenced projects and their entry targets are guaranteed to be in the cache, they will not build again. Therefore we do not need to set `/p:BuildProjectReferences=false` or any other gesture that tells SDKs to not do recursive operations. From 30445fc8af1813ab3e46272e1d54638abad9c098 Mon Sep 17 00:00:00 2001 From: Mihai Codoban Date: Mon, 4 May 2020 14:25:09 -0700 Subject: [PATCH 05/14] Update documentation/specs/static-graph-implementation-details.md Co-authored-by: Forgind --- documentation/specs/static-graph-implementation-details.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/documentation/specs/static-graph-implementation-details.md b/documentation/specs/static-graph-implementation-details.md index eae8d97682b..1cf286b285f 100644 --- a/documentation/specs/static-graph-implementation-details.md +++ b/documentation/specs/static-graph-implementation-details.md @@ -1,7 +1,7 @@ - [Single project isolated builds: implementation details](#single-project-isolated-builds-implementation-details) - [Input / Output cache implementation](#input--output-cache-implementation) - [Isolation implementation](#isolation-implementation) - - [How isolation exemption complicates everything :(](#how-isolation-exemption-complicates-everything) + - [How isolation exemption complicates everything](#how-isolation-exemption-complicates-everything) # Single project isolated builds: implementation details @@ -60,4 +60,4 @@ The build results for the exempt project are also included in the current cache Cases `2.2.` and `3.2.` complicate the requirement that the output cache should only contain newly built targets, and complicate the desirable goal that the override cache should never be mutated. In these cases initial entries (`BuildRequestConfiguration` for the `ConfigCache` and `BuildResult` / `TargetResult` for the `ResultsCache`) are already loaded in the override cache from previous builds, but then additional new builds on the isolation exempt entries need to be migrated / promoted to the current cache. This promotion is achieved differently for configs and build results: - `ConfigCache` entries [are moved](https://github.com/cdmihai/msbuild/blob/37c5a9fec416b403212a63f95f15b03dbd5e8b5d/src/Build/BackEnd/Components/Caching/ConfigCacheWithOverride.cs#L178) from the override cache into the current cache whenever a corresponding `BuildResult` is written into `BuildResultsWithOverride` (`BuildResult.ConfigurationId` == `BuildRequestConfiguration.ConfigurationId`). -- `BuildResult` / `TargetResult` entries are trickier. Sadly, the engine has a deep dependency on mutating existing results entries, so it's not possible to migrate result entries like config entries. Once the engine has obtained a reference to a result entry from the override cache, it will mutate it. In this particular case, [the BuildResultsWithOverride cache waives the requirement of non overlapping caches](https://github.com/cdmihai/msbuild/blob/37c5a9fec416b403212a63f95f15b03dbd5e8b5d/src/Build/BackEnd/Components/Caching/ResultsCacheWithOverride.cs#L139), and the new results are written both in the override caches (alongside the result entries deserialized from input caches), and also in the current cache. Thus, the override results cache will contain all build results, both old and new, while the current cache will contain only the new build results executed by the current build session. \ No newline at end of file +- `BuildResult` / `TargetResult` entries are trickier. Sadly, the engine has a deep dependency on mutating existing results entries, so it's not possible to migrate result entries like config entries. Once the engine has obtained a reference to a result entry from the override cache, it will mutate it. In this particular case, [the BuildResultsWithOverride cache waives the requirement of non overlapping caches](https://github.com/cdmihai/msbuild/blob/37c5a9fec416b403212a63f95f15b03dbd5e8b5d/src/Build/BackEnd/Components/Caching/ResultsCacheWithOverride.cs#L139), and the new results are written both in the override caches (alongside the result entries deserialized from input caches), and also in the current cache. Thus, the override results cache will contain all build results, both old and new, while the current cache will contain only the new build results executed by the current build session. From 793d2c0e288e98d6d43035808ddce157be1d2c0a Mon Sep 17 00:00:00 2001 From: Mihai Codoban Date: Mon, 4 May 2020 14:26:49 -0700 Subject: [PATCH 06/14] Update documentation/specs/static-graph-implementation-details.md Co-authored-by: Forgind --- documentation/specs/static-graph-implementation-details.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/specs/static-graph-implementation-details.md b/documentation/specs/static-graph-implementation-details.md index 1cf286b285f..69d34b2f2fd 100644 --- a/documentation/specs/static-graph-implementation-details.md +++ b/documentation/specs/static-graph-implementation-details.md @@ -8,7 +8,7 @@ Single project isolated builds can be achieved by providing MSBuild with input and output cache files. -The input cache files should contain the cached results all the targets that a project calls on its references. When a project executes, it will naturally build its references via [MSBuild task](https://docs.microsoft.com/en-us/visualstudio/msbuild/msbuild-task) calls. In isolated builds, the engine, instead of executing these tasks, will serve them from the provided input caches. In an isolated project build, only the current project should be building targets. Any other referenced projects should be provided form the input caches. +The input cache files contain the cached results of all the targets that a project calls on its references. When a project executes, it builds its references via [MSBuild task](aka.ms/msbuild_tasks) calls. In isolated builds, the engine, instead of executing these tasks, serves them from the provided input caches. In an isolated project build, only the current project should build targets. Any other referenced projects should be provided from the input caches. The output cache file tells MSBuild where it should serialize the results of the current project. This output cache would become an input cache for all other projects that depend on the current project. The output cache file can be omitted in which case the build would just reuse prior results but not write out any new results. This could be useful when one wants to replay a build from previous caches. From 1c9d1435f286629e7f441da71461df689a31f0b2 Mon Sep 17 00:00:00 2001 From: Mihai Codoban Date: Mon, 4 May 2020 14:27:46 -0700 Subject: [PATCH 07/14] Update documentation/specs/static-graph-implementation-details.md Co-authored-by: Forgind --- documentation/specs/static-graph-implementation-details.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/specs/static-graph-implementation-details.md b/documentation/specs/static-graph-implementation-details.md index 69d34b2f2fd..12a442cafe7 100644 --- a/documentation/specs/static-graph-implementation-details.md +++ b/documentation/specs/static-graph-implementation-details.md @@ -10,7 +10,7 @@ Single project isolated builds can be achieved by providing MSBuild with input a The input cache files contain the cached results of all the targets that a project calls on its references. When a project executes, it builds its references via [MSBuild task](aka.ms/msbuild_tasks) calls. In isolated builds, the engine, instead of executing these tasks, serves them from the provided input caches. In an isolated project build, only the current project should build targets. Any other referenced projects should be provided from the input caches. -The output cache file tells MSBuild where it should serialize the results of the current project. This output cache would become an input cache for all other projects that depend on the current project. +The output cache file tells MSBuild where to serialize the results of building the current project. This output cache becomes an input cache for all other projects that depend on the current project. The output cache file can be omitted in which case the build would just reuse prior results but not write out any new results. This could be useful when one wants to replay a build from previous caches. The presence of either input or output caches turns on [isolated build constraints](static-graph.md##single-project-isolated-builds). From 59fc63beb51784d326d0832864ce31577c510100 Mon Sep 17 00:00:00 2001 From: Mihai Codoban Date: Mon, 4 May 2020 14:34:48 -0700 Subject: [PATCH 08/14] Update documentation/specs/static-graph-implementation-details.md --- documentation/specs/static-graph-implementation-details.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/specs/static-graph-implementation-details.md b/documentation/specs/static-graph-implementation-details.md index 12a442cafe7..279a86d511e 100644 --- a/documentation/specs/static-graph-implementation-details.md +++ b/documentation/specs/static-graph-implementation-details.md @@ -11,7 +11,7 @@ Single project isolated builds can be achieved by providing MSBuild with input a The input cache files contain the cached results of all the targets that a project calls on its references. When a project executes, it builds its references via [MSBuild task](aka.ms/msbuild_tasks) calls. In isolated builds, the engine, instead of executing these tasks, serves them from the provided input caches. In an isolated project build, only the current project should build targets. Any other referenced projects should be provided from the input caches. The output cache file tells MSBuild where to serialize the results of building the current project. This output cache becomes an input cache for all other projects that depend on the current project. -The output cache file can be omitted in which case the build would just reuse prior results but not write out any new results. This could be useful when one wants to replay a build from previous caches. +The output cache file can be omitted in which case the build reuses prior results but does not write out any new results. This is useful when one wants to re-execute the build for a project without building its references. The presence of either input or output caches turns on [isolated build constraints](static-graph.md##single-project-isolated-builds). From ad4acb40390c8c589b2a370f56a06eb9e4ec0e43 Mon Sep 17 00:00:00 2001 From: Mihai Codoban Date: Mon, 4 May 2020 14:48:29 -0700 Subject: [PATCH 09/14] Update documentation/specs/static-graph-implementation-details.md --- documentation/specs/static-graph-implementation-details.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/specs/static-graph-implementation-details.md b/documentation/specs/static-graph-implementation-details.md index 279a86d511e..8361ff6b2a1 100644 --- a/documentation/specs/static-graph-implementation-details.md +++ b/documentation/specs/static-graph-implementation-details.md @@ -17,7 +17,7 @@ The presence of either input or output caches turns on [isolated build constrain ## Input / Output cache implementation -The cache files contain the serialized state of MSBuild's [ConfigCache](https://github.com/Microsoft/msbuild/blob/master/src/Build/BackEnd/Components/Caching/ConfigCache.cs) and [ResultsCache](https://github.com/Microsoft/msbuild/blob/master/src/Build/BackEnd/Components/Caching/ResultsCache.cs). These two caches have been traditionally used by the engine to cache build results. For example, it is these caches which ensure that a target is only built once per build submission. The `ConfigCache` entries are instances of [BuildRequestConfiguration](https://github.com/microsoft/msbuild/blob/37c5a9fec416b403212a63f95f15b03dbd5e8b5d/src/Build/BackEnd/Shared/BuildRequestConfiguration.cs#L25). The `ResultsCache` entries are instances of [BuildResult](https://github.com/microsoft/msbuild/blob/37c5a9fec416b403212a63f95f15b03dbd5e8b5d/src/Build/BackEnd/Shared/BuildResult.cs#L34), which compose instances of [TargetResult](https://github.com/microsoft/msbuild/blob/37c5a9fec416b403212a63f95f15b03dbd5e8b5d/src/Build/BackEnd/Shared/TargetResult.cs#L22). +The cache files contain the serialized state of MSBuild's [ConfigCache](https://github.com/Microsoft/msbuild/blob/master/src/Build/BackEnd/Components/Caching/ConfigCache.cs) and [ResultsCache](https://github.com/Microsoft/msbuild/blob/master/src/Build/BackEnd/Components/Caching/ResultsCache.cs). These two caches have been traditionally used by the engine to cache build results. For example, it is these caches which ensure that a target is only built once per build submission. The `ConfigCache` entries are instances of [BuildRequestConfiguration](https://github.com/microsoft/msbuild/blob/37c5a9fec416b403212a63f95f15b03dbd5e8b5d/src/Build/BackEnd/Shared/BuildRequestConfiguration.cs#L25). The `ResultsCache` entries are instances of [BuildResult](https://github.com/microsoft/msbuild/blob/37c5a9fec416b403212a63f95f15b03dbd5e8b5d/src/Build/BackEnd/Shared/BuildResult.cs#L34), which contain or more instances of [TargetResult](https://github.com/microsoft/msbuild/blob/37c5a9fec416b403212a63f95f15b03dbd5e8b5d/src/Build/BackEnd/Shared/TargetResult.cs#L22). One can view the two caches as the following mapping: `(project path, global properties) -> results`. `(project path, global properties)` is represented by a `BuildRequestConfiguration`, and the results are represented by `BuildResult` and `TargetResult`. From 4841fde87505a7f19d6d9e9defec6191deafb476 Mon Sep 17 00:00:00 2001 From: Mihai Codoban Date: Mon, 4 May 2020 14:49:24 -0700 Subject: [PATCH 10/14] Update documentation/specs/static-graph-implementation-details.md Co-authored-by: Forgind --- documentation/specs/static-graph-implementation-details.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/specs/static-graph-implementation-details.md b/documentation/specs/static-graph-implementation-details.md index 8361ff6b2a1..76cca9069a9 100644 --- a/documentation/specs/static-graph-implementation-details.md +++ b/documentation/specs/static-graph-implementation-details.md @@ -39,7 +39,7 @@ Entries that make it into the output cache file are separated from entries seria ## Isolation implementation -[Isolation constraints](static-graph.md##single-project-isolated-builds) are implemented in the Scheduler and the TaskBuilder. [TaskBuilder.ExecuteInstantiatedTask](https://github.com/microsoft/msbuild/blob/37c5a9fec416b403212a63f95f15b03dbd5e8b5d/src/Build/BackEnd/Components/RequestBuilder/TaskBuilder.cs#L743) ensures all `MSBuild` tasks are made on projects declared in `ProjectReference`. [Scheduler.CheckIfCacheMissOnReferencedProjectIsAllowedAndErrorIfNot](https://github.com/microsoft/msbuild/blob/37c5a9fec416b403212a63f95f15b03dbd5e8b5d/src/Build/BackEnd/Components/Scheduler/Scheduler.cs#L1818) ensures all `MSBuild` tasks are cache hits. +[Isolation constraints](static-graph.md##single-project-isolated-builds) are implemented in the Scheduler and the TaskBuilder. [TaskBuilder.ExecuteInstantiatedTask](https://github.com/microsoft/msbuild/blob/37c5a9fec416b403212a63f95f15b03dbd5e8b5d/src/Build/BackEnd/Components/RequestBuilder/TaskBuilder.cs#L743) ensures that the `MSBuild` task is only called on projects declared in `ProjectReference`. [Scheduler.CheckIfCacheMissOnReferencedProjectIsAllowedAndErrorIfNot](https://github.com/microsoft/msbuild/blob/37c5a9fec416b403212a63f95f15b03dbd5e8b5d/src/Build/BackEnd/Components/Scheduler/Scheduler.cs#L1818) ensures that all `MSBuild` tasks are cache hits. ### How isolation exemption complicates everything :( From 78a55246b9d53640ffa5a31cbfd150f3296629b0 Mon Sep 17 00:00:00 2001 From: Mihai Codoban Date: Mon, 4 May 2020 14:50:00 -0700 Subject: [PATCH 11/14] Update documentation/specs/static-graph-implementation-details.md Co-authored-by: Forgind --- documentation/specs/static-graph-implementation-details.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/specs/static-graph-implementation-details.md b/documentation/specs/static-graph-implementation-details.md index 76cca9069a9..a809864259c 100644 --- a/documentation/specs/static-graph-implementation-details.md +++ b/documentation/specs/static-graph-implementation-details.md @@ -47,7 +47,7 @@ Project references [can be exempt](static-graph.md#exempting-references-from-iso The `Scheduler` knows to skip isolation constraints on an exempt `BuildRequest` because the [ProjectBuilder compares](https://github.com/microsoft/msbuild/blob/37c5a9fec416b403212a63f95f15b03dbd5e8b5d/src/Build/BackEnd/Components/RequestBuilder/RequestBuilder.cs#L349) each new `BuildRequest` against the `GraphIsolationExemptReference` items defined in the calling project, and if exempt, sets `BuildRequest.SkipStaticGraphIsolationConstraints`. When a `BuildRequest` is marked as exempt, the `Scheduler` also marks its corresponding `BuildRequestConfiguration` as exempt as well, which aids in further identification of exempt projects outside the `Scheduler`. -The build results for the exempt project are also included in the current cache (according to the above mentioned rule that newly built results are serialized in the output cache file). This complicates the way caches interact, due to multiple reasons: +The build results for the exempt project are also included in the current cache (according to the above mentioned rule that newly built results are serialized in the output cache file). This complicates the way caches interact in several scenarios: 1. the same project can be exempt by several references, thus potentially colliding when multiple output cache files containing the same exempt project get aggregated. For example, given the graph `{A->B, A->C}`, where both `B` and `C` build the exempt project `D`, `D` will appear in the output caches of both `B` and `C`. When `A` aggregates these two caches, it will encounter duplicate entries for `D`, and will need to merge the results. 2. a project can be both exempt and present in the graph at the same time. For example, given the graph `{A->B}`, both `A` and `B` are in the graph, but `A` can also mark `B` as exempt (meaning that `A` contains both a `ProjectReference` item to `B`, and a `GraphIsolationExemptReference` item to `B`). The fact that `B` is in the graph means that `A` will receive an input cache containing B's build results. There are two subcases here: 1. `A` builds targets from `B` that already exist in the input cache file from `B`. In this case, all the builds of `B` will be cache hits, and no target results from `B` will make it into `A`'s output cache, since nothing new was built. From e361607c357e52524ba591533da477253e2fd21f Mon Sep 17 00:00:00 2001 From: Mihai Codoban Date: Mon, 4 May 2020 14:50:13 -0700 Subject: [PATCH 12/14] Update documentation/specs/static-graph-implementation-details.md Co-authored-by: Forgind --- documentation/specs/static-graph-implementation-details.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/specs/static-graph-implementation-details.md b/documentation/specs/static-graph-implementation-details.md index a809864259c..20b2ab329f6 100644 --- a/documentation/specs/static-graph-implementation-details.md +++ b/documentation/specs/static-graph-implementation-details.md @@ -48,7 +48,7 @@ Project references [can be exempt](static-graph.md#exempting-references-from-iso The `Scheduler` knows to skip isolation constraints on an exempt `BuildRequest` because the [ProjectBuilder compares](https://github.com/microsoft/msbuild/blob/37c5a9fec416b403212a63f95f15b03dbd5e8b5d/src/Build/BackEnd/Components/RequestBuilder/RequestBuilder.cs#L349) each new `BuildRequest` against the `GraphIsolationExemptReference` items defined in the calling project, and if exempt, sets `BuildRequest.SkipStaticGraphIsolationConstraints`. When a `BuildRequest` is marked as exempt, the `Scheduler` also marks its corresponding `BuildRequestConfiguration` as exempt as well, which aids in further identification of exempt projects outside the `Scheduler`. The build results for the exempt project are also included in the current cache (according to the above mentioned rule that newly built results are serialized in the output cache file). This complicates the way caches interact in several scenarios: -1. the same project can be exempt by several references, thus potentially colliding when multiple output cache files containing the same exempt project get aggregated. For example, given the graph `{A->B, A->C}`, where both `B` and `C` build the exempt project `D`, `D` will appear in the output caches of both `B` and `C`. When `A` aggregates these two caches, it will encounter duplicate entries for `D`, and will need to merge the results. +1. the same project can be exempt via multiple references, thus potentially colliding when multiple output cache files containing the same exempt project get aggregated. For example, given the graph `{A->B, A->C}`, where both `B` and `C` build the exempt project `D`, `D` will appear in the output caches of both `B` and `C`. When `A` aggregates these two caches, it will encounter duplicate entries for `D`, and will need to merge the results. 2. a project can be both exempt and present in the graph at the same time. For example, given the graph `{A->B}`, both `A` and `B` are in the graph, but `A` can also mark `B` as exempt (meaning that `A` contains both a `ProjectReference` item to `B`, and a `GraphIsolationExemptReference` item to `B`). The fact that `B` is in the graph means that `A` will receive an input cache containing B's build results. There are two subcases here: 1. `A` builds targets from `B` that already exist in the input cache file from `B`. In this case, all the builds of `B` will be cache hits, and no target results from `B` will make it into `A`'s output cache, since nothing new was built. 2. `A` builds targets from `B` that do not exist in the input cache file from `B`. If `B` weren't exempt from isolation constraints, this scenario would lead to a build break, as cache misses are illegal under isolation. With `B` being exempt, the new builds of `B` will get included in `A`'s output cache. The results from `B`'s cache file won't get included in `A`'s output cache file, as they weren't built by `A`. From 2c6ff616f5e526468b5bd5eea1a5333ed2c7de2c Mon Sep 17 00:00:00 2001 From: Mihai Codoban Date: Mon, 4 May 2020 14:51:32 -0700 Subject: [PATCH 13/14] Update documentation/specs/static-graph.md Co-authored-by: Forgind --- documentation/specs/static-graph.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/specs/static-graph.md b/documentation/specs/static-graph.md index 38c9802112a..71db33abeb9 100644 --- a/documentation/specs/static-graph.md +++ b/documentation/specs/static-graph.md @@ -352,7 +352,7 @@ A project is exempt from isolation constraints by adding its full path to the `G A reference is exempt only in projects that add the reference in `GraphIsolationExemptReference`. If multiple projects need to exempt the same reference, all of them need to add the reference to `GraphIsolationExemptReference`. -For now, self builds (a project building itself with different global properties) are also exempt from isolation constraints, but this behaviour is of dubious value and might be changed in the future. +For now, self-builds (a project building itself with different global properties) are also exempt from isolation constraints, but this behaviour is of dubious value and might be changed in the future. Details on how isolation and cache files are implemented in MSBuild can be found [here](./static-graph-implementation-details.md). From a009ee4950f6e56c4b16382f6e7db6b80679273b Mon Sep 17 00:00:00 2001 From: Mihai Codoban Date: Mon, 4 May 2020 15:09:46 -0700 Subject: [PATCH 14/14] updates --- documentation/specs/static-graph-implementation-details.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/specs/static-graph-implementation-details.md b/documentation/specs/static-graph-implementation-details.md index 20b2ab329f6..03c4bbaeff0 100644 --- a/documentation/specs/static-graph-implementation-details.md +++ b/documentation/specs/static-graph-implementation-details.md @@ -41,7 +41,7 @@ Entries that make it into the output cache file are separated from entries seria [Isolation constraints](static-graph.md##single-project-isolated-builds) are implemented in the Scheduler and the TaskBuilder. [TaskBuilder.ExecuteInstantiatedTask](https://github.com/microsoft/msbuild/blob/37c5a9fec416b403212a63f95f15b03dbd5e8b5d/src/Build/BackEnd/Components/RequestBuilder/TaskBuilder.cs#L743) ensures that the `MSBuild` task is only called on projects declared in `ProjectReference`. [Scheduler.CheckIfCacheMissOnReferencedProjectIsAllowedAndErrorIfNot](https://github.com/microsoft/msbuild/blob/37c5a9fec416b403212a63f95f15b03dbd5e8b5d/src/Build/BackEnd/Components/Scheduler/Scheduler.cs#L1818) ensures that all `MSBuild` tasks are cache hits. -### How isolation exemption complicates everything :( +### How isolation exemption complicates everything Project references [can be exempt](static-graph.md#exempting-references-from-isolation-constraints) from isolation constraints via the `GraphIsolationExemptReference` item.