Make "child node exited prematurely" more diagnosable from CI like Jenkins

[danmoseley]

In a typical CI system like Jenkins only the console output and expected build logs are easily available. By the time the result is examined the VM may have even been recycled. So when a build fails this way:

error MSB4166: Child node "4" exited prematurely. Shutting down. Diagnostic information may be found in files in the temporary files directory named MSBuild_*.failure.txt.

There is no easy way to diagnose the problem, so it cannot be fixed. Note that it's likely this isn't a bug in MSBuild proper - it can happen when a task runs out of stack space.

Suggestion: there is no reason why MSBuild.exe can't read the content of such a file and dump it to the console before it terminates. It knows the child process's PID and when the build started (or, it could give the child process a unique ID to put in the file name, but it's not necessary to do that work). It doesn't need to be a pretty solution.

Hopefully in this situation everything has been drained from the child node's logging events as well, such as the last step it wsa executing.

This situation has been happening in the dotnet/corefx repo.

[danmoseley]

If there is no such file, that is worth logging also since that suggests other causes eg., an AV or external termination.

[danmoseley]

If the child node is alive but unresponsive to the parent node, that is also worth knowing

[natemcmaster]

Is there anyway to configure where these temporary files are generated? I'm running into this too while running in Docker, but can't get to the log files because /tmp is cleaned up automatically.

[cdmihai]

Workaround: tell the CI job to archive those dumps on error.

[danmoseley]

Happened again - ought to be just a few line change to dump the contents of those files at the point we log the "exiting prematurely"

[cdmihai]

@danmosemsft
Can't log it if the exception comes at a point where the event bus is not working. And if it's happening on a child node, then stdout might not point to the console.
Can you configure Jenkins to archive those files?

[danmoseley]

@cdmihai

I think this is entirely on the parent node. It generates this message here and here. At that point it could then look in ExceptionHandling.DebugDumpPath for any files named MSBuild_pid-_.failure.txt whose write time was later than the start of the build, and dump the content of those to the console. Messy, yes, but the build just actually crashed.

For sure Jenkins should grab the log, but that would mean educating every CI system in the world that uses MSBuild to know about these MSBuild-specific crash files.