-
Notifications
You must be signed in to change notification settings - Fork 435
Add a VMR initialization (bootstrap) process #14343
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
33 commits
Select commit
Hold shift + click to select a range
fb5beeb
Add source-mappings.json with a defaults section
premun 3e040ce
Add empty InitializeVMR targets, VMR tasks
premun f1bd6fb
Copy initial VMR content and commit it
premun bda9a9d
Add steps to generate installer.props
premun 6da9799
Add a target for recursive repo initialization
premun 61bae2d
Build SourceBuild tasks too
premun 6be1810
Implement the "initialize VMR" task
premun 356c022
Add console logging
premun f4e99fd
Use newest DarcLib
premun 2ac3f67
Make the task cancellable
premun fad42c7
Set InitializeVMR to true always, improve dependency print, remove pa…
premun e72fa32
Improve logging in SB task
premun 122434a
Do not log dependencies when there are none
premun 7bfbbea
Merge remote-tracking branch 'origin/main' into vmr
premun d730b59
Add rest of the missing tarball steps
premun 61da10a
Add a TODO issue link
premun f2109ad
Use WorkingDirectory instead of -C
premun f37f3ef
Make VmrDir a required parameter
premun 51e53d0
Do not use Tarball properties
premun 71a2283
Move tasks under Tasks/
premun 6ff48ea
Use .NET 7.0 TFM
premun 0d78123
Remove obsolete exclusions
premun e73d666
Move RemoteFactory to Tasks/
premun 3557371
Extract Microsoft.DotNet.DarcLib version into Version.props
premun 623eacf
Bump SB tasks to net7.0
premun 4c3c2c1
Fix XML tag
premun 566403a
Update net7.0 task references
premun c100611
Simplify .csproj
premun f555b3c
Address PR feedback
premun 585cdd4
Add the DarcLib dependency into Version.Details
premun 557cb47
Merge remote-tracking branch 'origin/main' into vmr
premun ef1063a
Run `darc update-dependencies`
premun 7054873
Add subscription for `Microsoft.Extensions.Logging.Console`
premun File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,12 +1,30 @@ | ||
| <Project> | ||
| <ItemGroup Condition=" '$(ArcadeBuildTarball)' != 'true' "> | ||
| <ProjectToBuild Include="$(RepoRoot)Microsoft.DotNet.Cli.sln" /> | ||
| <ProjectToBuild Condition="'$(OS)' == 'Windows_NT' And ('$(Architecture)' == 'x86' Or '$(Architecture)' == 'x64' Or '$(Architecture)' == 'arm64')" | ||
| Include="$(RepoRoot)eng\version.csproj; | ||
| $(RepoRoot)eng\native.proj" /> | ||
| </ItemGroup> | ||
| <ItemGroup Condition=" '$(ArcadeBuildTarball)' == 'true' " > | ||
| <ProjectToBuild Include="$(RepoRoot)src/SourceBuild/Arcade/src/SourceBuild.Tasks.csproj" BuildInParallel="false" /> | ||
| <ProjectToBuild Include="$(RepoRoot)src/SourceBuild/tarball/BuildSourceBuildTarball.proj" BuildInParallel="false" /> | ||
| </ItemGroup> | ||
| <Choose> | ||
| <When Condition=" '$(ArcadeBuildTarball)' == 'true' "> | ||
| <!-- Source build / tarball generation --> | ||
| <ItemGroup> | ||
| <ProjectToBuild Include="$(RepoRoot)src/SourceBuild/Arcade/src/SourceBuild.Tasks.csproj" BuildInParallel="false" /> | ||
| <ProjectToBuild Include="$(RepoRoot)src/SourceBuild/tarball/BuildSourceBuildTarball.proj" BuildInParallel="false" /> | ||
| </ItemGroup> | ||
| </When> | ||
|
|
||
| <When Condition=" '$(InitializeVMR)' == 'true' "> | ||
| <!-- VMR bootstrap --> | ||
| <ItemGroup> | ||
| <ProjectToBuild Include="$(RepoRoot)src/SourceBuild/Arcade/src/SourceBuild.Tasks.csproj" BuildInParallel="false" /> | ||
| <ProjectToBuild Include="$(RepoRoot)src/VirtualMonoRepo/Tasks/VirtualMonoRepo.Tasks.csproj" BuildInParallel="false" /> | ||
| <ProjectToBuild Include="$(RepoRoot)src/VirtualMonoRepo/InitializeVMR.proj" BuildInParallel="false" /> | ||
| </ItemGroup> | ||
| </When> | ||
|
|
||
| <Otherwise> | ||
| <!-- Regular build --> | ||
| <ItemGroup> | ||
| <ProjectToBuild Include="$(RepoRoot)Microsoft.DotNet.Cli.sln" /> | ||
| <ProjectToBuild Condition="'$(OS)' == 'Windows_NT' And ('$(Architecture)' == 'x86' Or '$(Architecture)' == 'x64' Or '$(Architecture)' == 'arm64')" | ||
| Include="$(RepoRoot)eng\version.csproj; | ||
| $(RepoRoot)eng\native.proj" /> | ||
| </ItemGroup> | ||
| </Otherwise> | ||
| </Choose> | ||
| </Project> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
140 changes: 140 additions & 0 deletions
140
src/SourceBuild/tarball/content/src/source-mappings.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,140 @@ | ||
| { | ||
| "defaults": { | ||
| "defaultRef": "main", | ||
| "exclude": [ | ||
| "**/*.dll", | ||
| "**/*.Dll", | ||
| "**/*.exe", | ||
| "**/*.pdb", | ||
| "**/*.mdb", | ||
| "**/*.zip", | ||
| "**/*.nupkg" | ||
| ] | ||
| }, | ||
| "mappings": [ | ||
| { | ||
| "name": "arcade", | ||
|
premun marked this conversation as resolved.
|
||
| "defaultRemote": "https://github.com/dotnet/arcade", | ||
| "include": [ | ||
| "*.*", | ||
| ".*", | ||
| "eng/**/*", | ||
| "src/**/*", | ||
| "scripts/**/*" | ||
| ], | ||
| "exclude": [ | ||
| "**/tests/**/*", | ||
| "**/*.Tests/**/*" | ||
| ] | ||
| }, | ||
| { | ||
| "name": "aspnetcore", | ||
| "defaultRemote": "https://github.com/dotnet/aspnetcore" | ||
|
premun marked this conversation as resolved.
|
||
| }, | ||
| { | ||
| "name": "clicommandlineparser", | ||
| "defaultRemote": "https://github.com/dotnet/clicommandlineparser" | ||
| }, | ||
| { | ||
| "name": "command-line-api", | ||
| "defaultRemote": "https://github.com/dotnet/command-line-api" | ||
| }, | ||
| { | ||
| "name": "deployment-tools", | ||
| "defaultRemote": "https://github.com/dotnet/deployment-tools" | ||
| }, | ||
| { | ||
| "name": "diagnostics", | ||
| "defaultRemote": "https://github.com/dotnet/diagnostics" | ||
| }, | ||
| { | ||
| "name": "format", | ||
| "defaultRemote": "https://github.com/dotnet/format" | ||
| }, | ||
| { | ||
| "name": "fsharp", | ||
| "defaultRemote": "https://github.com/dotnet/fsharp" | ||
| }, | ||
| { | ||
| "name": "installer", | ||
| "defaultRemote": "https://github.com/dotnet/installer", | ||
| "defaultRef": "vmr" | ||
| }, | ||
| { | ||
| "name": "linker", | ||
| "defaultRemote": "https://github.com/dotnet/linker" | ||
| }, | ||
| { | ||
| "name": "msbuild", | ||
| "defaultRemote": "https://github.com/dotnet/msbuild" | ||
| }, | ||
| { | ||
| "name": "nuget-client", | ||
| "defaultRemote": "https://github.com/NuGet/NuGet.Client", | ||
| "defaultRef": "dev" | ||
| }, | ||
| { | ||
| "name": "razor-compiler", | ||
| "defaultRemote": "https://github.com/dotnet/razor-compiler" | ||
| }, | ||
| { | ||
| "name": "roslyn", | ||
| "defaultRemote": "https://github.com/dotnet/roslyn" | ||
| }, | ||
| { | ||
| "name": "roslyn-analyzers", | ||
| "defaultRemote": "https://github.com/dotnet/roslyn-analyzers" | ||
| }, | ||
| { | ||
| "name": "runtime", | ||
| "defaultRemote": "https://github.com/dotnet/runtime" | ||
| }, | ||
| { | ||
| "name": "sdk", | ||
| "defaultRemote": "https://github.com/dotnet/sdk" | ||
| }, | ||
| { | ||
| "name": "source-build", | ||
| "defaultRemote": "https://github.com/dotnet/source-build" | ||
| }, | ||
| { | ||
| "name": "source-build-externals", | ||
| "defaultRemote": "https://github.com/dotnet/source-build-externals", | ||
| "exclude": [ | ||
| "**/humanizer/samples/**/*.js" | ||
| ] | ||
| }, | ||
| { | ||
| "name": "source-build-reference-packages", | ||
| "defaultRemote": "https://github.com/dotnet/source-build-reference-packages" | ||
| }, | ||
| { | ||
| "name": "sourcelink", | ||
| "defaultRemote": "https://github.com/dotnet/sourcelink" | ||
| }, | ||
| { | ||
| "name": "symreader", | ||
| "defaultRemote": "https://github.com/dotnet/symreader" | ||
| }, | ||
| { | ||
| "name": "templating", | ||
| "defaultRemote": "https://github.com/dotnet/templating" | ||
| }, | ||
| { | ||
| "name": "test-templates", | ||
| "defaultRemote": "https://github.com/dotnet/test-templates" | ||
| }, | ||
| { | ||
| "name": "vstest", | ||
| "defaultRemote": "https://github.com/microsoft/vstest" | ||
| }, | ||
| { | ||
| "name": "xdt", | ||
| "defaultRemote": "https://github.com/dotnet/xdt" | ||
| }, | ||
| { | ||
| "name": "xliff-tasks", | ||
| "defaultRemote": "https://github.com/dotnet/xliff-tasks" | ||
| } | ||
| ] | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,121 @@ | ||
| <!-- Licensed to the .NET Foundation under one or more agreements. The .NET Foundation licenses this file to you under the MIT license. --> | ||
| <Project> | ||
| <PropertyGroup> | ||
| <ArtifactsDir>$(RepoRoot)artifacts/</ArtifactsDir> | ||
| <VirtualMonoRepoTasksAssembly>$(ArtifactsDir)bin/VirtualMonoRepo.Tasks/$(Configuration)/net7.0/VirtualMonoRepo.Tasks.dll</VirtualMonoRepoTasksAssembly> | ||
|
|
||
| <VmrDir Condition=" '$(VmrDir)' != '' ">$([MSBuild]::EnsureTrailingSlash('$(VmrDir)'))</VmrDir> | ||
| <TmpDir Condition=" '$(TmpDir)' == '' ">$(ArtifactsDir)tmp/</TmpDir> | ||
| <VmrDefaultBranch Condition=" '$(VmrDefaultBranch)' == '' ">main</VmrDefaultBranch> | ||
|
mmitche marked this conversation as resolved.
|
||
|
|
||
| <!-- We set this for the source build targets to work well --> | ||
| <TarballDir>$(VmrDir)</TarballDir> | ||
| <InitializeVMR>true</InitializeVMR> | ||
| </PropertyGroup> | ||
|
|
||
| <Import Project="../SourceBuild/Arcade/tools/SourceBuildArcadeTarball.targets" /> | ||
|
|
||
| <!-- Pull back in some properties from the tarball defaults --> | ||
| <PropertyGroup> | ||
| <VmrGitInfoDir>$(TarballGitInfoDir)</VmrGitInfoDir> | ||
| <VmrSourceDir>$(TarballSourceDir)</VmrSourceDir> | ||
| </PropertyGroup> | ||
|
|
||
| <UsingTask TaskName="Microsoft.DotNet.VirtualMonoRepo.Tasks.VirtualMonoRepo_Initialize" AssemblyFile="$(VirtualMonoRepoTasksAssembly)" /> | ||
|
|
||
| <Target Name="Build" DependsOnTargets="InitializeVMR" /> | ||
|
|
||
| <!-- | ||
| This target initializes a fresh VMR from scratch by assembling it similarly to the tarball. | ||
| Instead of cloning the repositories, we pull the sources via the `darc vmr initialize` command. | ||
| Most of the targets are reused from the tarball generation process (SourceBuildArcadeTarball.targets). | ||
| --> | ||
| <Target Name="InitializeVMR" DependsOnTargets="GenerateFullNuGetVersion; | ||
| GenerateVersionFile; | ||
| InitializeCleanVmr; | ||
| CopyTarballContent; | ||
| CommitInitialContent; | ||
| SetupSelfGithubInfo; | ||
| InitializeRepoAndDependentsRecursive; | ||
| CommitGitInfoFiles; | ||
| CopyTextOnlyPackages; | ||
| CommitTextOnlyPackages"> | ||
| <Message Text="VMR was successfully initialized in '$(VmrDir)'" Importance="High" /> | ||
| </Target> | ||
|
|
||
| <Target Name="InitializeCleanVmr"> | ||
| <RemoveDir Directories='$(VmrDir)' Condition=" EXISTS('$(VmrDir)') " /> | ||
| <MakeDir Directories="$(VmrDir)" /> | ||
| <MakeDir Directories="$(VmrGitInfoDir)" Condition=" '$(VmrGitInfoDir)' != '' " /> | ||
| <MakeDir Directories="$(TmpDir)" Condition=" !EXISTS('$(TmpDir)') " /> | ||
|
|
||
| <Exec Command="git init -b $(VmrDefaultBranch)" WorkingDirectory="$(VmrDir)" /> | ||
| </Target> | ||
|
|
||
| <Target Name="CommitInitialContent"> | ||
| <Exec Command="git add -A" WorkingDirectory="$(VmrDir)" /> | ||
| <Exec Command="git commit -m 'VMR initialized'" WorkingDirectory="$(VmrDir)" /> | ||
| </Target> | ||
|
|
||
| <Target Name="InitializeRepoAndDependentsRecursive" | ||
| DependsOnTargets="GetSourceBuildIntermediateNupkgNameConvention"> | ||
|
|
||
| <Message Text="--> Initializing repo $(SourceBuildRepoName)" Importance="High" /> | ||
| <PropertyGroup> | ||
| <SourceDir>$(SourceBuildRepoName)/</SourceDir> | ||
| <IndividualRepoSourceDir>$(VmrSourceDir)$(SourceDir)</IndividualRepoSourceDir> | ||
| <IndividualRepoSourceEngDir>$(IndividualRepoSourceDir)eng/</IndividualRepoSourceEngDir> | ||
| <IndividualRepoVersionDetailsFile>$(IndividualRepoSourceEngDir)Version.Details.xml</IndividualRepoVersionDetailsFile> | ||
| </PropertyGroup> | ||
|
|
||
| <VirtualMonoRepo_Initialize | ||
| Repository="$(SourceBuildRepoName)" | ||
| Revision="$(RepoSha)" | ||
| VmrPath="$(VmrDir)" | ||
| TmpPath="$(TmpDir)" /> | ||
|
|
||
| <Message Text=" -> Done initializing repo $(SourceBuildRepoName)" Importance="High" /> | ||
|
|
||
| <Tarball_ReadSourceBuildIntermediateNupkgDependencies | ||
| VersionDetailsXmlFile="$([MSBuild]::NormalizePath($(IndividualRepoVersionDetailsFile)))" | ||
| SourceBuildIntermediateNupkgPrefix="$(SourceBuildIntermediateNupkgPrefix)" | ||
| SourceBuildIntermediateNupkgRid="$(SourceBuildIntermediateNupkgRid)" | ||
| ConvertInternalRepos="$(ConvertInternalRepos)"> | ||
| <Output TaskParameter="Dependencies" ItemName="SourceBuildRepos" /> | ||
| </Tarball_ReadSourceBuildIntermediateNupkgDependencies> | ||
|
|
||
| <!-- Remove repo if it has already be cloned at any sha. This results in | ||
| The commit sha for the cloned repo in the tarball being the one that | ||
| was first encountered. --> | ||
| <ItemGroup> | ||
| <SourceBuildRepos Remove="@(SourceBuildRepos)" Condition=" EXISTS('$(VmrGitInfoDir)%(SourceBuildRepoName).props')" /> | ||
| </ItemGroup> | ||
|
|
||
| <Tarball_WriteSourceRepoProperties | ||
| SourceBuildMetadataDir="$(VmrGitInfoDir)" | ||
| Dependencies="@(SourceBuildRepos)" /> | ||
|
|
||
| <Message Text="--> Dependencies for $(IndividualRepoVersionDetailsFile):" Importance="High" Condition=" '@(SourceBuildRepos)' != '' " /> | ||
| <Message Text="--> %(SourceBuildRepos.SourceBuildRepoName) / %(SourceBuildRepos.Sha)" Importance="High" Condition=" '@(SourceBuildRepos)' != '' " /> | ||
| <MSBuild Projects="$(MSBuildProjectFile)" | ||
| Condition=" '@(SourceBuildRepos)' != '' " | ||
| Targets="InitializeRepoAndDependentsRecursive" | ||
| Properties="SourceBuildRepoName=%(SourceBuildRepos.SourceBuildRepoName);RepoSha=%(SourceBuildRepos.Sha)" /> | ||
|
|
||
| </Target> | ||
|
|
||
| <Target Name="CommitGitInfoFiles"> | ||
| <Copy | ||
| SourceFiles="$(VmrGitInfoDir)runtime.props" | ||
| DestinationFiles="$(VmrGitInfoDir)runtime-portable.props" /> | ||
|
|
||
| <Exec Command="git add git-info" WorkingDirectory="$(VmrDir)" /> | ||
| <Exec Command="git commit -m 'Initialized git-info files'" WorkingDirectory="$(VmrDir)" /> | ||
| </Target> | ||
|
|
||
| <Target Name="CommitTextOnlyPackages"> | ||
| <Exec Command="git add packages/text-only" WorkingDirectory="$(VmrDir)" /> | ||
| <Exec Command="git commit -m 'Initialized text-only packages'" WorkingDirectory="$(VmrDir)" /> | ||
| </Target> | ||
|
|
||
| </Project> | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.