MSBuild 15 Sdk Design

[AndyGerlicher]

MSBuild will allow third-party-defined and dynamically-delivered extensions of the build process via the new “Sdk” concept. This will extend the experience delivered with RC.2 to include an acquisition process.

Changes to project files

An Sdk can be consumed in two ways: with implicit top and bottom imports at through the Sdk attribute on the Project element:

<Project Sdk="Microsoft.NET.Sdk">
...
</Project>

and through a modified <Import> element with the Sdk attribute:

<Project>
  <Import Project="Sdk.props" Sdk="Microsoft.NET.Sdk/1.0.0" />
...
  <Import Project="Sdk.targets" Sdk="Microsoft.NET.Sdk/1.0.0" />
</Project>

The former mechanism is syntactic sugar for the latter, with an Sdk.props import at the top of the project and an Sdk.targets import at its bottom.

User experience

When a user opens an Sdk-using project in an IDE, the IDE will try to evaluate the project. MSBuild will coordinate with the IDE to fetch and integrate all required Sdks. The IDE will probably wish to display a progress bar for any required downloads.

On the command line, a dotnet restore or msbuild.exe /t:Restore invocation will fetch required Sdks before proceeding with the restore process, ensuring that only one user gesture is required to bring the project to a buildable state.

Sdk acquisition

How does MSBuild know how to acquire an Sdk when it needs to? Through a provider model. A caller of the MSBuild APIs can provide an instance of a new interface ISdkResolver which will be used to map the XML from above to files on the local filesystem.

Note: Although the acquisition could differ between builds the resolved SDK would be identical.

struct SdkReference
{
    public string Name;
    public Version? Version;
}

interface ISdkResolver
{
    public Dictionary<SdkReference, Path> Resolve(IEnumerable<SdkReference>);
}

We expect most ISdkResolver implementations to additionally have affordances for reporting progress back to a UI and to log using the MSBuild log interface.

🚧 Exposing logging may require an interface change here.

🚧 Do we need to pass path-to-project, so that something NuGet-like can walk up to find its feed configuration?

MSBuild evaluation changes

MSBuild will collect Sdks needed for a given project early in its evaluation (before pass 1 expands imports). It will then unify Sdks specified within the project with those specified in the (optional) lineup file. This produces a list of Sdks that is then passed to ISdkResolver.Resolve(), producing a lookup table for Sdk imports.

Evaluator.GetCurrentDirectoryForConditionEvaluation should return the Sdk directory if there is an Sdk attribute on an import, so that you can have something like Condition="Exists('Sdk\Sdk.props')" on an Sdk import.

Evaluation pass 1 (properties and imports) then continues as usual. When considering an Import element with an Sdk attribute, the specified Project attribute is treated as a relative path from the base Sdk path looked up in the table generated in the new pre-pass-1 step.

Lineups

After growing past being “small” projects, most repos/solutions will want a simple way to manage the set of Sdks they use--for instance, to unify versions of Sdks available for projects within the repo. This will be accomplished by a lineup file, format TBD, that MSBuild will consider when putting together the list of available Sdks.

🚧 What does this look like on disk?

We expect a few common patterns of use:

• Small-scale “demo projects”
  • Consist of ~1 project.
  • Do not have a lineup file.
  • Specify Sdk version for any “non-core” Sdk referenced.
• Larger-scale projects
  • Have many projects in a solution.
  • Specify a lineup
  • Do not specify versions in the project files themselves, only in the lineup.

Project Load

We'll need to add to ProjectLoadSettings so a user can opt out of SDK resolution. They would need to specify both IgnoreMissingImports and a new DoNotResolveSdks if they wanted to open a project without getting errors. But this would allow minimal property evaluation without resolving anything.

Concrete work items

• Get feedback from VS folks
  • Especially around UI needs for status updating.
• Expose ISdkResolver definition for prototyping.
• Implement Sdk-aware imports (Import from SDKs #1400)
• Implement Sdk-gathering (pass 0.5)
• Augment build-starting methods to accept an ISdkResolver
  • In BuildParameters? BuildRequestData?
• Minimal FilesystemSdkResolver for prototyping and testing
  • Maybe looks in NuGet cache folder? That'd get us a long way toward seeing what the once-everything-is-downloaded scenario looks like.
• Implement NuGetSdkResolver: Proposal: implement a SdkResolver for NuGet packages NuGet/Home#5220
  • Straw man implementation idea: write sdk names + package versions to a minimal project file, restore it out of proc, return paths 🚧 magically.
• Use resolvers in various scenarios
  • Change MSBuild.exe to use a resolver
  • Change dotnet CLI to invoke MSBuild with a resolver
  • Change VS to invoke MSBuild with a resolver
  • Change VS Code to invoke MSBuild with a resolver
  • Change VS Mac to invoke MSBuild with a resolver
  • 🚧 Some of those could be helped by changing the default, if blocking evaluation without progress reporting is ok. Probably is for OmniSharp (out of proc) and CLI (synchronous command line build). UIs probably want advanced progress reporting + cancellability.

[maartenba]

How does this play with VS Code, JetBrains Rider, Xamarin, MonoDevelop, ...?

[jeffkl]

@jviau We're designing a way for a UI to provide to MSBuild some logic that resolves an SDK. The implication for VS is that it would want to present the user with a progress dialog while it restores packages during project load. Our design here means that VS would specify a class to MSBuild that we would create an instance of and call Resolve. If a user loaded a solution with multiple projects, then Resolve would be called multiple times. Would this work okay with VS? Can you review the design here and give any feedback?

[jeffkl]

@maartenba

How does this play with VS Code, JetBrains Rider, Xamarin, MonoDevelop, ...?

We have items near the bottom to involve VS Code, VS Mac and dotnet CLI to ensure they can pass a resolver to give the user a good experience.

JetBrains Rider and MonoDevelop will need to implement the interface for displaying a progress dialog to the user. I'm fairly certain they already have a custom MSBuild logger to handle log output like Visual Studio so we're hoping it will be easy for IDEs to make this happen for the SDK resolution. We can probably pave the way with the ones we ship in VS that they can re-use or take as an example.

[jeffkl]

@AndyGerlicher What do you think about making this more generic and extensible? Right now it's specifically for resolving SDKs but do you think we'll ever want to resolve other project assets? My thought was to have the logic be more centered around resolving assets, one of which would be an SDK. Something like:

public interface AssetResolver
{
    IEnumerable<IResolvedAsset> Resolve(IEnumerable<IAsset> assets);
}

public enum AssetType
{
    None,
    SDK,
}

public interface IAsset
{
    AssetType AssetType { get; }
    string Name { get; }
    Version Version { get; }
}

public interface IResolvedAsset : IAsset
{
    /// <summary>
    /// The path to the resolved asset.
    /// </summary>
    string Path { get; }
}

Thoughts?

[jeffkl]

I also looked at logging and it's going to be tricky. The only thing available during project evaluation is the internal LoggingService so we'd need a new interface to expose to the caller or allow them to throw exceptions. If we want them to be able to log messages and warnings we'll need to come up with something.

[Mike-E-angelo]

This sounds like a very intriguing and valuable feature!

a lineup file, format TBD

Please consider making this an XML file, to keep it consistent with the current ecosystem. It would be confusing to introduce differing and inconsistent data formats, especially after moving from XML to JSON and then back to XML again.

The spirit of the request here isn't to favor a particular format, but rather to favor consistency. I might be alone on this, however. But to me, exploring a given a solution (VS or otherwise) and seeing two different data files in two different formats when they both simply describe data is not only maddeningly inconsistent, but inefficient as well. That means you ultimately have two libraries/concerns in your solution that parse/save data, when it could be done with just one.

Of course, if you provide a strictly POCO-based model whereby the end user could describe their data in the format they prefer, then this is a non-issue. 😄 What strikes me about this feature is that the same sort of magic could eventually be used to download/install data-format preferences/serialization components as well -- something that has been discussed in #613.

[maartenba]

@jeffki Is he resolver ever an IDE-specific extension or is the IDE's only role to call into msbuild and show progress?

[jviau]

@jeffkl I am a bit confused by the VS UI interaction. Is VS supposed to implement its own ISdkResolver that performs both progress UI and the sdk resolution? Or are you going to have a VS-agnostic implementation that performs only sdk resolution (then it can be re-used for the command line) and then CPS hooks up to events or something and is responsible for displaying the progress UI?

[jeffkl]

@jviau Yes Visual Studio's implementation of ISdkResolver would do the resolution and the UI. We didn't think much of the resolvers could be re-used because Visual Studio might use the NuGet object model since it has NuGet assemblies while dotnet CLI might just run NuGet.exe and pipe it's output. Does this sound doable for VS?