Make Visual Studio Toolset updates declarative

[jaredpar]

This is in response to dotnet/runtime#32876 as well as similar issues in roslyn where the version of Visual Studio was silently rolled forward for all consuming repositories. These centralized roll forwards end up breaking consuming repositories due to incompatibilities in the new Visual Studio toolset. This is most likely to affect repositories with heavy native toolset dpendencies, as they are not as compatible version to version, but can also impact repositories that do integration testing on Visual Studio.

After discussion it seems like core-eng and the consuming repositories have somewhat conflicting incentives here.

1. Repositories like dotnet/runtime want to be extremely deliberate and declarative about updates to native toolsets. The ideal situation is that it should be done in a PR so all of the PR infrastructure can be used to validate the new toolset. This is due to the fact that native tools are simply not as copmatible release to release and this makes automatic updates risky.
2. Core-Eng wants to:
   1. Be constantly pushing forward the version of the Visual Studio Toolset used to build the product. This has all sorts of positive benefits: dogfooding SDKs, getting bug fixes, etc ...
   2. Be maintaining the fewest number of possible images (and the associated Helix queues).

I think there is a straight forward way that we can satisfy the concerns of both parties here:

1. Arcade should define a YML template file that defines a variable named windows-vs-16. The value of this variable will be the name of the queue which has the latest Visual Studio 16 installation. The value of the variable would be in the form BuildPool.Windows.VisualStudio.16.4
2. Repositories which use Visual Studio Images for build or test would switch to consuming the template and using the variable windows-vs-16 to define the pool queue name.
3. When Core-Eng wants to move to a new Visual Studio Toolset, say 16.5, they'd take the existing image, update it to 16.5 and then save it with a new name: BuildPool.Windows.VisualStudio.16.5. They would then send a PR to arcade to update the value of windows-vs-16 to the new queue name.
4. Darc then pushes the change through all of the repos like it does any other arcade change. This means dotnet/runtime gets a PR that pushes the Visual Studio Toolset forward. Any errors that result from this will be caught through normal PR validation.
   1. Fix the errors and merge the PR
   2. Temporarily switch from using windows-vs-16 to the previous real moniker (in this case BUildPool.Windows.VisualStudio.16.4). They can then prioritize fixing the issues and moving back to the variable.

This system does mean that Core-Eng will have to temporarily maintain two Visual Studio images during the darc push. That should be temporary though as the expectation is that repositories should quickly ingest the new toolset. In the case that the image can't be immedaitely ingested that does mean that Core-Eng will need to maintain both images longer. But it also means that the repostiory that can't ingest the change stays productive while in the current system it would simply be down for those few days.

Recently Triaged Issues

All issues in this section should be triaged by the v-team into one of their business objectives or features.

[jaredpar]

@jkotas @ViktorHofer @jasonmalinowski

@markwilkie @Chrisboh

[markwilkie]

Turning this into an epic as there are quite a few pieces to get right before proceeding. cc/ @tkapin who's probably interested.

[tkapin]

This is still a problem for Roslyn (and perhaps other teams) not sufficiently solved by introduction of scouting queues as pointed out by recent comment by @sharwell on CI Council:

The scouting queue caught a problem that required a change to our code to correct. This was good if we only needed CI for main. The problem is we have a bunch of different people maintaining a bunch of different branches at different points in time, and as soon as we hit this scenario we have to unravel the situation for every single feature branch so even though I updated the main branch in a timely manner, I'm getting all the same questions I was getting about broken builds before we had a scouting queue