Insertion of master to CLI is failing

[wli3]

dotnet/cli#8901

stack trace:

D:\j\workspace\debug_windows---8a02d2bd\build\BundledDotnetTools.proj(24,5): error MSB4018: The "DotNetRestore" task failed unexpectedly. [D:\j\workspace\debug_windows---8a02d2bd\build.proj]
23:12:07          D:\j\workspace\debug_windows---8a02d2bd\build\BundledDotnetTools.proj(24,5): error MSB4018: System.IndexOutOfRangeException: Index was outside the bounds of the array. [D:\j\workspace\debug_windows---8a02d2bd\build.proj]
23:12:07          D:\j\workspace\debug_windows---8a02d2bd\build\BundledDotnetTools.proj(24,5): error MSB4018: at System.Collections.Generic.Dictionary`2.KeyCollection.CopyTo(TKey[] array, Int32 index) [D:\j\workspace\debug_windows---8a02d2bd\build.proj]
23:12:07          D:\j\workspace\debug_windows---8a02d2bd\build\BundledDotnetTools.proj(24,5): error MSB4018: at System.Diagnostics.Process.GetEnvironmentVariablesBlock(IDictionary`2 sd) [D:\j\workspace\debug_windows---8a02d2bd\build.proj]
23:12:07          D:\j\workspace\debug_windows---8a02d2bd\build\BundledDotnetTools.proj(24,5): error MSB4018: at System.Diagnostics.Process.StartWithCreateProcess(ProcessStartInfo startInfo) [D:\j\workspace\debug_windows---8a02d2bd\build.proj]
23:12:07          D:\j\workspace\debug_windows---8a02d2bd\build\BundledDotnetTools.proj(24,5): error MSB4018: at System.Diagnostics.Process.Start() [D:\j\workspace\debug_windows---8a02d2bd\build.proj]
23:12:07          D:\j\workspace\debug_windows---8a02d2bd\build\BundledDotnetTools.proj(24,5): error MSB4018: at Microsoft.Build.Utilities.ToolTask.ExecuteTool(String pathToTool, String responseFileCommands, String commandLineCommands) [D:\j\workspace\debug_windows---8a02d2bd\build.proj]
23:12:07          D:\j\workspace\debug_windows---8a02d2bd\build\BundledDotnetTools.proj(24,5): error MSB4018: at Microsoft.Build.Utilities.ToolTask.Execute() [D:\j\workspace\debug_windows---8a02d2bd\build.proj]

[wli3]

this could also affect 2.1 branch

[danmoseley]

@wil3 sorry about that. This was broken for 3 days by dotnet/coreclr@922c5a4 and fixed by dotnet/coreclr@fe867dd.

[danmoseley]

@benaadams we had a test in corefx that broke for this right?

@eerhardt when CLI pick up the runtime, has it gone through corefx first? If not, then breaks in mscorlib are not necessarily going to stopped before it goes up because we rely mostly on corefx tests for mscorlib.

[wil3]

@danmosemsft wrong username

[danmoseley]

Oops. I'm guessing that happened before! I mean @wli3 then.

[eerhardt]

@eerhardt when CLI pick up the runtime, has it gone through corefx first?

No. If coreclr produces a build, it goes to core-setup. After core-setup builds, it goes to cli. We don't wait for tests to pass before we flow builds.

[danmoseley]

OK I guess the onus is always on the dev changing mscorlib to privately run corefx tests on their PR.

[eerhardt]

OK I guess the onus is always on the dev changing mscorlib to privately run corefx tests on their PR.

I think that is good practice.

The idea is that if a change got through CI, it must be good. Or else, why did it get through CI?

[AndyAyersMS]

In CoreCLR we don't run the CI CoreFX tests on every PR because they tend suffer from various integration issues.

But if a PR changes anything under mscorlib then perhaps the CI should add in the CoreFx tests automatically.

[livarcocc]

In which build can we expect this fix to be in? We haven't had a runtime insertion into master for a while now due to various breaks. This seems to be the last one. When will this change surface in core-setup master?

[danmoseley]

cc @stephentoub @jkotas for opinions

The CoreFX tests when run in CoreCLR CI are very slow - I assume they are not super-parallelized across machines as they are in CoreFX. That means in my opinion it should stay selective (also because they are not always perfectly in sync)

[danmoseley]

When will this change surface in core-setup master?

It is not there yet. It will surface soon after we get a build out of coreclr now that this is fixed. https://github.com/dotnet/corefx/issues/28602 I just started a build. If all goes well, could be in core-setup tonight.

[jkotas]

But if a PR changes anything under mscorlib then perhaps the CI should add in the CoreFx tests automatically.

The corefx tests were triggered in the CoreCLR CI manually on the offending change: dotnet/coreclr#17196 (comment)

Neither Ben or me have noticed the break because of it was buried behind other failures and noise.

I think the plan has to be to make the infrastucture changes so that the CoreFX tests can be run in CoreCLR unconditionally on every PR, and the run is fast, reliable and always green. It has been discussed number of times, e.g.: https://github.com/dotnet/coreclr/issues/12782#issuecomment-348626206. We just need to find a way to fund this.

[danmoseley]

@jkotas you did not miss the tests behind noise, the tests did not fail.

The commit in the PR that introduced the bug was dotnet/coreclr@c5bb5ba and you can see that this https://ci.dot.net/job/dotnet_coreclr/job/master/job/jitstress/job/x64_checked_windows_nt_corefx_baseline_prtest/320/consoleFull#213780152433fe3402-0c0f-45a7-a707-47164965bab5 ran on that commit specifically and yet

17:36:03      System.Collections.Tests  Total: 26542, Errors: 0, Failed: 0, Skipped: 0, Time: 131.240s
18:38:53      System.Runtime.Extensions.Tests  Total: 5779, Errors: 0, Failed: 0, Skipped: 15, Time: 165.646s

I don't know why. I do see

17:26:55 [D:\j\workspace\x64_checked_w---d7295605\tests\scripts\run-corefx-tests.py]: Copy: D:\j\workspace\x64_checked_w---d7295605\bin\Product\Windows_NT.x64.Checked\System.Private.CoreLib.dll => D:\j\workspace\x64_checked_w---d7295605\_\fx\bin\testhost\netcoreapp-Windows_NT-Release-x64\shared\Microsoft.NETCore.App\9.9.9\System.Private.CoreLib.dll

then

17:29:33   Using D:\j\workspace\x64_checked_w---d7295605\_\fx\bin\testhost\netcoreapp-Windows_NT-Release-x64\ as the test runtime folder.