Add build manifest#5210
Conversation
| Pack "true" to build NuGet packages and VS insertion manifests | ||
| Sign "true" to sign built binaries | ||
| Publish "true" to publish artifacts (e.g. symbols) | ||
| GenerateBuildManifest "true" to generate a build manifest during publish step. Requires "Sign" and "Publish" are also true so that signing metadata can be included in the build manifest. |
There was a problem hiding this comment.
GenerateBuildManifest [](start = 4, length = 21)
I think we should generate the manifest when ContinuousIntegrationBuild == true
|
Latest changes validated internally with https://dev.azure.com/dnceng/internal/_build/results?buildId=592451&view=logs&j=0a6c679f-72be-5867-422e-acb741a068a3&t=d8e44ddf-8ea2-5518-25e7-4300471a271a |
|
Long term, that's a goal. They're serving two slightly different purposes. Build manifest - defines the artifacts produced by the build and metadata for how the build believes they should be used (signing info at the moment). Asset manifest - defines the artifacts that were published externally and where they can be found. There's some overlap here in the sense that the I agree, it would be nice if there was one manifest (or they were named better), but I don't yet have an idea for how we could combine these. |
|
Perhaps, if the build manifest contained the metadata and rather than using the asset manifest as the "here's what I actually did" document, we used the build manifest as the "here's what you should do" document and then later we trusted that it actually happened, then we could eliminate the asset manifest. |
|
The asset manifest actually isn't quite that unfortunately. It doesn't define entirely what was done. It's a combo of "what should I do" and "what I did". The asset manifest is produced prior to publishing and contains a list of artifacts. But it also contains information for publishing that identifies where those artifacts should go if the build happens to be published. The big one there is for blobs. The azdo build artifacts have the files that got produced, but the asset manifest tells us what subpaths those artifacts should go live on if they happen to be finally published. Publishing defines the base endpoints (e.g. dotnetcli.blob.core.windows.net/dotnet) and then the relative paths are defined by the asset manifests @JohnTortugo and I were talking about the possibility of getting rid of the asset manifest the other day. Much of what the asset manifest contains is duplicated in the BAR build info. But the asset manifest also contains info that is not duplicated, and potentially could change over time as we change publishing. In order to get rid of the manifest, you need to encapsulate everything in there today in BAR, and make it extensible enough such that changes to that info do not involve a database change every time. The xml based manifest is pretty perfect for that.
This actually could work really well. What if we called the build manifest in this case a "signing manifest" (since that's what it's primarily targeted at) and then that info gets added to the asset manifest like you say. We also add to the asset manifest, a version number. Let's call it version 2, where version one of the asset manifest is the current 3.x and 5.x builds. This version of the asset manifest can be interpreted differently by the publishing infrastructure. For the most part, everything is identical. But in order to sign after the build, we'll need the signing info available, so we also need to archive the manifest itself (since builds will go away in AzDO after N days), To recap: So version 1 of publishing, the current state, knows nothing about adding signing info to the asset manifest, and version 2, which the new PushToAzureDevOps task will generate, will include the signing info from the provided signing manifest. Publishing will be altered to know about both asset versions. For version 1, it does what it does today. For version 2, it knows that the artifacts are not signed, will need to be signed later, and thus archives the manifest in blob storage or wherever for use by the post-build prep stages where signing is being moved to. |
| - task: PublishBuildArtifacts@1 | ||
| displayName: Publish Build Manifest | ||
| inputs: | ||
| PathToPublish: '$(Build.SourcesDirectory)/artifacts/log/$(_BuildConfig)/manifest.props' |
There was a problem hiding this comment.
IMHO we really need a better name than just "manifest.props"
There was a problem hiding this comment.
what do you think it should be?
There was a problem hiding this comment.
I'll post a more general observation in the PR.
|
Latest changes validated with https://dev.azure.com/dnceng/internal/_build/results?buildId=593679&view=results |
| options.InputFiles = inputFiles.ToArray(); | ||
|
|
||
| LogVerbosity verbosity; | ||
| if(Enum.TryParse<LogVerbosity>(Verbosity, out verbosity)) |
There was a problem hiding this comment.
NIT: can inline the verbosity var declaration
|
@chcosta I'd like to get this merged into the asset manifest during the publish to BAR stage as outlined above (with a BAR manifest verison), rather than carrying along a separate manifest. What would it take to get that done? |
|
Shouldn't be too difficult. I'm hoping to have this merged today but I think I need to make one more change to this pr before merging |
|
I fully agree with what @mmitche said. IMHO having two manifests isn't ideal.
I think we need to have a plan for this. What other information will this manifest contains in the future? Right now I see that it only contains signing information. |
|
I think we all agree that two manifests isn't ideal. |
ghost
mentioned this pull request
|
@chcosta Did you open an issue to merge the manifests? |
|
I asked @jonfortescue to open one, as he's working on that right now |
|
@chcosta I am indeed working on it and I also forgot to open the issue to show that so i'll go do that now! |
5 participants
Creates a build manifest during the build which later stages / jobs (or different pipelines) can consume.
Modifies signing validation to be an MSBuild task which can consume the manifest and is more extensible than the console tool previously used
Splits out many signing property defaults from the Arcade Sdk so they can be imported as needed
Replaces outdated Microsoft.Bcl.JsonSources with System.Text.Json
_UseBuildManifestineng\common\variables.ymlis currently set toFalsebecause we need to produce a version of the SDK with these changes before we can enable the build manifest.Validated with https://dnceng.visualstudio.com/internal/_build/results?buildId=590601&view=results