Download smoke test prereqs in prep script by mthalman · Pull Request #16359 · dotnet/installer

mthalman · 2023-05-10T14:49:24Z

This provides a way for people to download the packages required to smoke tests in order to fulfill two scenarios:

Pre-downloading the packages allows the tests to be run offline.
Provides a way to download prerelease packages from a private NuGet feed.

The prep script is modified to include options that, when provided, will download the necessary packages required by the smoke tests. Unfortunately, becomes NuGet's restore logic only allows a project to reference a single version of a NuGet package. But some of the packages required by the smoke tests require multiple versions of a given package name. So a custom MSBuild target is needed which iteratively runs restore once for each individual package version.

The list of packages is defined in a prereqs.csproj file. The accuracy of this list is ensured by a newly added test. This test gets run after all the other smoke tests have finished execution as it is the execution of those tests which determines which packages are required. The packages which were loaded as a result of running the smoke tests is then validated against the list of packages defined in the prereqs.csproj to ensure its accuracy.

Breaks things up into smaller methods and adds a couple data types

MichaelSimons · 2023-05-25T20:42:16Z

        OutputHelper = outputHelper;
    }
+
+    public static void EnumerateTarball(string tarballPath, Func<TarEntry, bool> continueEnumeration)


Was there a particular reason this wasn't put in the Utilities class? Seems like a better fit.

MichaelSimons · 2023-05-25T21:20:00Z

+###   --runtime-source-feed         URL of a remote server or a local directory, from which SDKs and
+###                                 runtimes can be downloaded
+###   --runtime-source-feed-key     Key for accessing the above server, if necessary
+###   --smoke-test-prereqs-path     Directory where the smoke test prereqs packages should be downloaded to.


Depending on the resolution to my previous comments, this may the the only new parameter. If that's the case I think this script would be more dev friendly if there was a 1. a flag to enable smoke test prereq gathering 2.the path was defaulted with an env override?

I don't know how we could have a prep script that didn't include a feed option. There has to be a way to specify the DSP feed.

Makes sense. I still find the smoke-test-prereqs-path option to be a bit inconsistent - there is no equivalent option to specify the artifacts path today. Because of that, I towards this should be defaulted. This would require the addition of a n option to indicate you want the prereqs to be gathered.

MichaelSimons · 2023-05-30T13:51:36Z

Regarding ad23764 and 4731319, what do you see as the frequency for these updates? Is this something we are going to need to update each release?

mthalman · 2023-06-06T19:43:22Z

Latest updates:

Updates the smoke tests project so that when run offline, it will restore packages from the local prereqs feed location.
Updates prep script to use a get-smoke-test-prereqs option to be explicit about the action to take rather than implied from the path. This also removes the prereqs path location as an option and is now defaulted instead.
Treats errors of the prereqs as warnings so that we can evaluate how much churn is involved with keeping the baseline up-to-date before causing the tests to fail the build. This change, unfortunately, requires that we not run tests offline for now during this eval period because running offline would cause restore failures in scenario tests when baseline isn't correct.
Various bug fixes

mthalman · 2023-06-07T14:21:09Z

@MichaelSimons - This previous update is interesting. This dependency is defined by the project generated by the MSTest template and has this content:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>

    <IsPackable>false</IsPackable>
    <IsTestProject>true</IsTestProject>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.6.0" />
    <PackageReference Include="MSTest.TestAdapter" Version="3.0.4" />
    <PackageReference Include="MSTest.TestFramework" Version="3.0.4" />
    <PackageReference Include="coverlet.collector" Version="6.0.0" />
  </ItemGroup>

</Project>

That version change was updated by dotnet/test-templates#307. Those dependencies make it tricky if we want to change things to derive the package list via static analysis. It may be that we'd need to use the SDK to create projects from the templates and parse the generated project file to get the packages that they reference.

mthalman · 2023-06-07T14:29:06Z

It may be that we'd need to use the SDK to create projects from the templates and parse the generated project file to get the packages that they reference.

I guess that wouldn't really work when executing from the prep script since you don't yet have the SDK.

MichaelSimons · 2023-06-07T13:39:12Z

Is it possible to re-use the existing NuGet.config files that already exist in the assets folder? These NuGet.configs get used by the test projects the tests themselves build. At a minimum we should utilize the same naming convention.

MichaelSimons · 2023-06-07T13:40:54Z

+
+```bash
+./prep.sh --no-artifacts --no-prebuilts \
+    --smoke-test-prereqs-output prereqs/packages/smoke-test-prereqs


This cmd and the one later on need to be updated to use the latest prep options. e.g. get-smoke-test-prereqs

MichaelSimons · 2023-06-07T13:58:52Z

+      if [[ '${{ parameters.runOnline }}' == 'False' ]]; then
+        smokeTestFeed="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet8/nuget/v3/index.json"
+        smokeTestFeedKeyArg=''
+        # Enable this when we have a feed for internal builds (when .NET 8 enters servicing)


This can happen before servicing. It will be required at GoLive time whenever there is an MSRC.

I am a bit concerned about the maintainability of this. Ideally this would be able to share logic from eng/common/SetupNuGetSources.sh to be able to retrieve the feeds to add so that there is not a disconnect.

Are we at the point where we should have regular unit tests for the prep script? Those tests could verify the usage of the feed options and then here we could utilize the SetupNugetSources script to avoid introducing a maintenance item that needs to be updated for each release.

mthalman · 2023-06-08T18:16:13Z

Abandoning these changes because of the issues that we're going to have with maintenance of the baseline file (see dotnet/source-build#3386 (comment)).

Download smoke test prereqs in prep script

a64219c

mthalman requested a review from a team as a code owner May 10, 2023 14:49

MichaelSimons reviewed May 10, 2023

View reviewed changes

Import versions.props

668ad80

MichaelSimons added the Area-SourceBuild label May 18, 2023

mthalman added 5 commits May 18, 2023 14:59

Refactor test class

d77209e

Breaks things up into smaller methods and adds a couple data types

Run tests offline in offline build legs

3f40e98

Simplify prep script by having a default, pre-defined NuGet.config

bfa56d5

Update comments in prep script

6cdc89c

Update test comments

e573328

mthalman requested a review from MichaelSimons May 25, 2023 19:24

MichaelSimons reviewed May 25, 2023

View reviewed changes

mthalman added 4 commits May 26, 2023 09:55

Updates from review comments

6512e2b

Improve test output for diagnosing failures

8cffb9b

Update expected packages

4731319

Update package versions

ad23764

MichaelSimons mentioned this pull request May 30, 2023

source-build: bundle runtime packs with the sdk. #16499

Merged

mthalman added 2 commits June 6, 2023 14:31

Updates from review comments

c8dbaba

Merge branch 'main' into sb3386-smoke-tests

07a7f6b

mthalman requested a review from MichaelSimons June 6, 2023 19:43

Update package versions

0436c5f

MichaelSimons reviewed Jun 7, 2023

View reviewed changes

mthalman mentioned this pull request Jun 8, 2023

Provide a mechanism to gather prereq packages required to run smoke-tests offline dotnet/source-build#3386

Closed

mthalman closed this Jun 8, 2023

mthalman deleted the sb3386-smoke-tests branch July 17, 2023 14:34

mthalman mentioned this pull request Jul 17, 2023

bootstrapped smoke-test running unable to find package Microsoft.DotNet.Arcade.Sdk dotnet/source-build#2958

Closed

Conversation

mthalman commented May 10, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MichaelSimons commented May 30, 2023

Uh oh!

mthalman commented Jun 6, 2023

Uh oh!

mthalman commented Jun 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mthalman commented Jun 7, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mthalman commented Jun 8, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mthalman commented Jun 7, 2023 •

edited

Loading