Out-of-proc builds in a custom .NET Core process are not possible

[jeffkl]

Issue Description

If you write a custom application in .NET Core, use MSBuildLocator to find .NET Core MSBuild, and attempt to do an out-of-process build, then the MSBuild BuildManager will attempt to launch the node as the user's custom application.

Steps to Reproduce

• Create a custom .NET Core application named Foo.exe
• Use MSBuildLocator to find MSBuild in .NET Core
• Build a project with DisableInProcNode set to true

Expected Behavior

MSBuild APIs correctly build project out-of-proc

Actual Behavior

MSBuild APIs attempt to launch an out-of-proc node with Foo.exe instead of MSBuild.exe

Analysis

This is happening because this call:

msbuild/src/Build/BackEnd/Components/Communications/NodeProviderOutOfProcBase.cs

Line 483 in aac64bb

exeName = GetCurrentHost();

But GetCurrentHost assumes that MSBuild.exe is the entry application which is not always correct:

msbuild/src/Build/BackEnd/Components/Communications/NodeProviderOutOfProcBase.cs

Lines 594 to 608 in aac64bb

	private static string GetCurrentHost()
	{
	#if RUNTIME_TYPE_NETCORE || MONO
	if (CurrentHost == null)
	{
	using (Process currentProcess = Process.GetCurrentProcess())
	{
	CurrentHost = currentProcess.MainModule.FileName;
	}
	}
	return CurrentHost;
	#else
	return null;
	#endif

In some cases, the current process is not MSBuild.exe.

Instead, this code should use the current dotnet.exe that is running that application and use the MSBuild location which is already done here:

msbuild/src/Build/BackEnd/Components/Communications/NodeProviderOutOfProcBase.cs

Lines 435 to 437 in aac64bb

	// Repeat the executable name as the first token of the command line because the command line
	// parser logic expects it and will otherwise skip the first argument
	commandLineArgs = msbuildLocation + " " + commandLineArgs;

There is another bug where we prepend the path to MSBuild.dll twice:

msbuild/src/Build/BackEnd/Components/Communications/NodeProviderOutOfProcBase.cs

Line 484 in aac64bb

commandLineArgs = "\"" + msbuildLocation + "\" " + commandLineArgs;

But that doesn't seem to break anything as dotnet.exe correctly ignores that it was given two applications to run!

Versions & Configurations

Any version of .NET Core MSBuild

[Forgind]

Good news and bad news:
I made a console app as you described and tried running it. Mine happened to be named 6782, so it made 6782.exe as you described.
It also made more than 1. In fact, it made hundreds very quickly and kept making more until my computer completely died. Everything started going really slowly after just a minute or so, and then the screen turned black, and I had to force it to shut down. That was last week, and I decided to take the rest of the day off after that. This week, I'd forgotten (blocked out?) that memory, so I tried it again, but I did manage to kill the hundreds of 6782.exes before my computer was completely overwhelmed, though it was touch and go there for a little bit. Task Manager was no help at all because it couldn't kill them fast enough. I'm guessing they came in at about 5/second.
On the positive side, I don't want to touch my repro because now we have a reliable repro for "hundreds of MSBuild processes" (probably?). It would be great to know what's going on there.

[jeffkl]

@Forgind I did the same exact thing, glad I'm not the only sucker 😛 😆

Some good news is that I was able to work around it via reflection to set the private member:

https://github.com/jeffkl/MSBuildProjectCreator/blob/1f262b528bd03297e5967aff8f3268589dd55410/src/Microsoft.Build.Utilities.ProjectCreation/BuildManagerHost.cs#L80

[Forgind]

So having looked into this a little more, I think the problem is that there is no dotnet.exe process involved. In my case, 6782.exe was acting as dotnet.exe in loading msbuild.dll and executing its code, but that means I can't just look for a dotnet.exe nearby and replace GetCurrentHost() with that. I think the best option is to use <path_to_MSBuild.dll>.upOneLevel() + "dotnet.exe".