Create a common build script auto-updater

[dagood]

Core repos all have a few scripts (init-tools.cmd/sh, run.cmd/sh, etc.) that are duplicated and drift apart over time. These should be automatically updated from a central source. A VersionTools updater can do this.

The complicated part of this is figuring out the "path" to the central source. I suggest we put the "master copy" of the scripts into BuildTools. Each Core repo then updates to the script version associated with the BuildTools package it's depending on. The updater would:

• Download the currently-used BuildTools package
• Open it and read version.txt for the BuildTools commit that built it
• Download the scripts from BuildTools at that commit hash.

(When we publish built commit hashes to dotnet/versions, the updater doesn't need to do the package download of ~5 MB.)

We planned to do this for bootstrap.sh/ps1 initially (https://github.com/dotnet/core-eng/issues/393), but auto-updating scripts is useful independent of that.

Came up in dotnet/core-setup#3252 (comment). /cc @weshaggard @dleeapho @eerhardt

---

Edit: As a historical note, the WIP I had for this is here: master...dagood:script-updater/master

[eerhardt]

I remember hearing some work that @jaredpar and @tmat were doing in this area as well. We should have a coordinated effort across all the .NET Core repos for auto-updating build scripts.

[jaredpar]

CC @tannergooding

[weshaggard]

@tannergooding is this something you are looking at implementing as well?

[dagood]

I'm taking a look soon at making an implementation of this as part of VersionTools. This will slot in nicely with existing auto-updates and should be a fairly easy task. However, it won't be a unified way to do the script updates unless other repos start adopting VersionTools.

[dagood]

@weshaggard, I want to run the configuration and functionality I have so far by you to make sure it makes sense:

<PropertyGroup>
  <BuildScriptsCurrentRef>145def38c43191e9b11666a6e8a14990b3b9dfb7</BuildScriptsCurrentRef>
</PropertyGroup>

<ItemGroup>
  <DependencyInfo Include="BuildScripts">
    <DependencyType>Repository</DependencyType>
    <Repository>https://github.com/dotnet/source-build</Repository>
    <Ref>dev/release/2.0</Ref>
    <CurrentRef>$(BuildScriptsCurrentRef)</CurrentRef>
  </DependencyInfo>

  <UpdatedBuildScriptPath Include="init-tools.cmd;init-tools.sh" />

  <UpdateStep Include="Build script auto-update">
    <UpdaterType>ExternalRepoFileUpdater</UpdaterType>
    <RepositoryIdentity>BuildScripts</RepositoryIdentity>
    <LocalRootDir>$(MSBuildThisFileDirectory)</LocalRootDir>
    <RelativePaths>@(UpdatedBuildScriptPath)</RelativePaths>
  </UpdateStep>
</ItemGroup>

The updater requires that the file already exists on disk (it won't "seed" the files it's set to auto-update) for two reasons:

1. CRLF vs. LF. If it detects \r\n anywhere in the file, it will use CRLF for the update, otherwise LF. I'm not basing it on platform, because someone on Windows could have autocrlf off for some reason. Any sane repo stores its files with LF, so it's easy to convert from what the updater grabs from GitHub to the detected format.
2. File modes. Specifically, executable status for .shes. On Windows, you can only set the file mode once the file is in the index, which currently VersionTools doesn't deal with at all. The updater won't handle auto-updating file mode, if that ever changes--the files must be seeded in each repo with the correct mode.

Does that sound reasonable?

[weshaggard]

The configuration looks reasonable. However I'm curious about the seeding/updating part. My expectation is you would simply take the file from the source and just copy straight down and not actually try to update, essentially just overwrite the file if it is there or write it if isn't there yet.

[dagood]

Yeah, 1 and 2 are describing reasons that make that difficult, but it's possible. Thoughts on the quickest way around them:

1. We could always place down new files as LF. Git complains about this if CRLF is expected for that file, but I believe the end result is what we want, where it's LF in the repo and new autocrlf-enabled checkouts get the CRLFs on disk.
2. Run a Git command during the auto-updater that does git add then git update-index --chmod=(+/-)x just after it sets the file contents.

I'll give these a shot and see if it works out ok.

[ViktorHofer]

// Auto-generated message

Going forward, the .NET team is using https://github.com/dotnet/arcade to develop the code and issues formerly in this repository. Feel free to reopen/move this issue if it still applies to servicing branches in this repository, or source code which now resides in the Arcade repository.