Improve performance of Package testing leg

[ericstj]

Related dotnet/corefx#35693

A couple ideas we can use to shorten this:

1. Do a sparse matrix of RIDs for all the older frameworks. The self-contained restores and tests are really the longest part of this work. We end up doing a dozen tests for each framework, all of which have to download the entire shared framework for those RIDs.
2. Pre-restore the framework packages on the build machine and add them to a package folder that gets shared with all helix clients. This has the potential of saving multiple clients from hitting nuget servers to download old packages. They still end up downloading through helix though, so I don't this alone is a sufficient without also doing 1.

[ViktorHofer]

I'm not a fan of 2). Can't we instead use a http cache for these assets?

[ericstj]

Yeah, i'm not so much a fan either. I think 1 is a clear win (by relaxing test requirements). Maybe a better option for 2 would be to share the packages folder for multiple test runs on the same helix client machine rather than starting every run from scratch?

[ViktorHofer]

That sounds like a good idea.

[ViktorHofer]

The leg just timed out during the restore: https://dev.azure.com/dnceng/public/_build/results?buildId=438614&view=ms.vss-test-web.build-test-results-tab&runId=13949022&resultId=100071&paneView=attachments. As part of the CI council effort, we should find a way to avoid unnecessary restore invocations.

[safern]

I'll work on this after the live-live CI effort. @ericstj and I had some ideas, but I think the best one for now will be to implement an spare matrix.

We also thought about doing package testing per RID, rather than per package. That way we would restore the runtime assets just one, which seems a good idea as well.

[ViktorHofer]

When doing this, should we also off-load the per-RID validation into the other build legs, out of the allconfigurations leg?

[safern]

We only build the Private.CoreFX packages (Which are going away) as part of the RID build legs and one runtime package (IO.Ports native package in Linux builds). All the OOB packages are built as part of the allconfigurations leg... so, is it worth it?

[ViktorHofer]

And I assume we can't build them as part of the main legs? I was thinking of repackaging the nuget packages from the different legs. Not sure if that makes any sense.

[safern]

You need to build all the build configurations in order to get the package assets.

I would think of the other way around where all packages are built in the all configurations leg, where other legs publish assets and then we package them -- but then you can't do builds in parallel. You would have to move the packaging step out in order to achieve it.

[ViktorHofer]

You need to build all the build configurations in order to get the package assets.

Why?

[safern]

because that's on the only build where we expand and build all the configurations of a project. i.e: we don't have a vertical build for netcoreapp2.0 or for netcoreapp2.1 so all the assets for those configurations would be lost if we package on whatever our verticals built.

[ViktorHofer]

That's why I mentioned "re-packaging" of nuget packages.

[safern]

That's why I mentioned "re-packaging" of nuget packages.

I see. Then it doesn't make sense to me how this would work. We would have an all configurations leg and then re-package each package on the main legs? Could you be more explicit? I am missing something I guess.