Building the repo is creating a bunch of host instances and using all the CPU

[safern]

As of 3 days ago we received an arcade update PR which updated the SDK version that we use to build in the repo to: 5.0.100-preview.6.20266.3.

e0e752a

Some people including myself have noted weird behaviors like 56 instances of dotnet.exe being created when building the repo, the build having some weird behaviors and taking a long time. I reverted the global.json back to the older version we were using and everything works fine.

I think we should revert to the preview5 sdk we were using.

The new updated SDK was for packaging performance and it was done in: dotnet/arcade#5497

cc: @mmitche @dotnet/runtime-infrastructure @joperezr

Tagging subscribers to this area: @ViktorHofer
Notify danmosemsft if you want to be subscribed.

[trylek]

I completely concur, my local Windows builds of the runtime repo take hours, spawn 3K .NET Host process instances, consume 60 GB RAM and brick my devbox completely, that's something I've never seen before. Thanks Santi for following up so quickly!

[safern]

cc: @dsplaisted @wli3

[trylek]

@safern - this is hell on Earth, can you please share steps how to undo the SDK update locally? I can locally revert the entire PR #37200 but that's going to pollute my local git clone, is it sufficient to undo the changes to global.json or do we also need the change to eng/Versions.props?

[safern]

@trylek you can revert the changes in global.json only, that would be enough. I'll put up a PR to revert this soon and I'll open an issue on the SDK repo.

@mmitche should we revert this in arcade?

[trylek]

It works! Life's back to normal! Reminded me of long ago in .NET Native in early introduction of nugets when the network usage for enormous and my local build took 12 hours over the VPN.

[mmitche]

Reverting in arcade is non-trivial as some repos will have to do work to back up. It also won't auto-revert in all the repos (since the version there is newer). I'd recommend we isolate the issue and resolve it rather than hide it.

In the meantime we can update the sdk version in runtime, and ensure that it doesn't get pulled forward in a new update by disabling the arcade subscription or just editing global.json in any auto-created PR (which are sparse these days)

[mmitche]

@rrelyea @dsplaisted @rainersigwald @safern has some new info here.

If the p6 SDK is installed globally (matching what is in runtime's global.json) 5.0.100-preview.6.20266.3, then it appears that during project restore early in the build he's getting hundreds of dotnet hosts launched, effectively killing the machine.

This does not happen if the sdk is not installed globally.

[safern]

@rainersigwald I see that most of all the processes created are just doing:

"C:\Program Files\dotnet\dotnet.exe" "C:\Program Files\dotnet\sdk\5.0.100-preview.6.20266.3\MSBuild.dll" C:\Program Files\dotnet\sdk\5.0.100-preview.6.20266.3\MSBuild.dll /nologo /nodemode:1 /nodeReuse:true /low:false

The parent process is:

"dotnet.exe" exec "C:\Program Files\dotnet\sdk\5.0.100-preview.6.20266.3\MSBuild.dll" -maxcpucount -verbosity:m /m /nologo /clp:Summary /v:minimal /nr:True /p:ContinuousIntegrationBuild=False /warnaserror /p:TreatWarningsAsErrors=true C:\Users\Santiago\.nuget\packages\microsoft.dotnet.arcade.sdk\5.0.0-beta.20278.4\tools\Build.proj /p:Configuration=Debug /p:RepoRoot=C:\repos\runtime /p:Restore=True /p:DeployDeps=False /p:Build=True /p:Rebuild=False /p:Deploy=False /p:Test=False /p:Pack=False /p:IntegrationTest=False /p:PerformanceTest=False /p:Sign=False /p:Publish=False /p:TargetArchitecture=x64 "-distributedlogger:Microsoft.DotNet.Tools.MSBuild.MSBuildLogger,C:\Program Files\dotnet\sdk\5.0.100-preview.6.20266.3\dotnet.dll*Microsoft.DotNet.Tools.MSBuild.MSBuildForwardingLogger,C:\Program Files\dotnet\sdk\5.0.100-preview.6.20266.3\dotnet.dll"

[dsplaisted]

Sounds like it's failing to connect to the child processes it creates, so it keeps retrying.

Can you set the MSBUILDDEBUGCOMM environment variable to 1, run the build, and then look at the files starting with MSBuild_CommTrace_PID_ in the temp folder?

[safern]

Sounds like it's failing to connect to the child processes it creates, so it keeps retrying.

So potentially a fall back from: dotnet/msbuild@fa3df44 ?

Can you set the MSBUILDDEBUGCOMM environment variable to 1, run the build, and then look at the files starting with MSBuild_CommTrace_PID_ in the temp folder?

Sure. Let me try that and I'll update the issue.

[safern]

Yeah, a bunch of files are created and they have:

OutOfProc Endpoint Packet Pump (TID 5) 637268249724763835 +    1.5585ms: Waiting for connection 900000 ms...
OutOfProc Endpoint Packet Pump (TID 5) 637268249724918563 +   15.4728ms: Parent started connecting. Reading handshake from parent
OutOfProc Endpoint Packet Pump (TID 5) 637268249724941539 +    2.2976ms: Client connection failed but we will wait for another connection. Exception: Client: rejected old host. Received byte 0 but this matched a byte to reject.
OutOfProc Endpoint Packet Pump (TID 5) 637268249724949461 +    0.7922ms: Waiting for connection 899979 ms...