[build] Shut down the build server after 'make all' as well.#22300
[build] Shut down the build server after 'make all' as well.#22300rolfbjarne merged 1 commit intomainfrom
Conversation
Hopefully this will fix the build randomly hanging at the end of 'make all'.
✅ [PR Build #4147fe5] Build passed (Build packages) ✅Pipeline on Agent |
✅ [PR Build #4147fe5] Build passed (Detect API changes) ✅Pipeline on Agent |
✅ [PR Build #4147fe5] Build passed (Build macOS tests) ✅Pipeline on Agent |
💻 [PR Build #4147fe5] Tests on macOS M1 - Mac Monterey (12) passed 💻✅ All tests on macOS M1 - Mac Monterey (12) passed. Pipeline on Agent |
💻 [PR Build #4147fe5] Tests on macOS X64 - Mac Sonoma (14) passed 💻✅ All tests on macOS X64 - Mac Sonoma (14) passed. Pipeline on Agent |
💻 [PR Build #4147fe5] Tests on macOS M1 - Mac Ventura (13) passed 💻✅ All tests on macOS M1 - Mac Ventura (13) passed. Pipeline on Agent |
💻 [PR Build #4147fe5] Tests on macOS arm64 - Mac Sequoia (15) passed 💻✅ All tests on macOS arm64 - Mac Sequoia (15) passed. Pipeline on Agent |
✅ API diff for current PR / commit.NET ( No breaking changes )❗ API diff vs stable (Breaking changes).NET ( ❗ Breaking changes ❗ )ℹ️ Generator diffGenerator Diff: vsdrops (html) vsdrops (raw diff) gist (raw diff) - Please review changes) Pipeline on Agent |
💻 [CI Build #4147fe5] Windows Integration Tests passed 💻✅ All Windows Integration Tests passed. Pipeline on Agent |
🚀 [CI Build #4147fe5] Test results 🚀Test results✅ All tests passed on VSTS: test results. 🎉 All 112 tests passed 🎉 Tests counts✅ cecil: All 1 tests passed. Html Report (VSDrops) Download Pipeline on Agent |
Parallel make (e.g. 'make all -j8', 'make world') has been hanging indefinitely at the end of the build. This is a long-standing issue (#13355) that has been patched three times (#15407, #21315, #22300) without fully fixing the root cause. The problem: when using parallel make, GNU Make uses a jobserver with pipe-based file descriptors to coordinate sub-makes. The dotnet CLI can start background build servers (MSBuild server, Roslyn/VBCSCompiler) that inherit these file descriptors but never close them. Make then waits indefinitely for those file descriptors to close, thinking there are still active jobs. The previous workaround attempted to shut down and force-kill dotnet processes after the build via a 'shutdown-build-server' target. This approach was unreliable because: - The shutdown ran from a double-colon all-hook:: rule with no prerequisites, so with -j it could execute in parallel with (or before) the actual build, killing nothing. - Build servers started by later subdirectories (e.g. tests/) after the dotnet/ shutdown were never killed. - The process-matching regex pattern might not match all server processes. The fix: disable build servers entirely via environment variables in Make.config: - DOTNET_CLI_USE_MSBUILD_SERVER=0: prevents the MSBuild server - UseSharedCompilation=false: prevents the Roslyn compiler server - MSBUILDDISABLENODEREUSE=1: prevents MSBuild node reuse This eliminates the root cause - no background servers means no inherited file descriptors means no hang. The shutdown-build-server target and its invocations are removed as they are no longer needed. Additionally, 'make world' now prints the installed workloads at the end of the build for visibility. Build without changes: make world 2149.57s user 258.32s system 107% cpu 37:30.19 total Build with changes: make world 2242.74s user 286.38s system 354% cpu 11:52.55 total Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Parallel make (e.g. 'make all -j8', 'make world') has been hanging indefinitely at the end of the build. This is a long-standing issue (#13355) that has been patched three times (#15407, #21315, #22300) without fully fixing the root cause. The problem: when using parallel make, GNU Make uses a jobserver with pipe-based file descriptors to coordinate sub-makes. The dotnet CLI can start background build servers (MSBuild server, Roslyn/VBCSCompiler) that inherit these file descriptors but never close them. Make then waits indefinitely for those file descriptors to close, thinking there are still active jobs. The previous workaround attempted to shut down and force-kill dotnet processes after the build via a 'shutdown-build-server' target. This approach was unreliable because: - The shutdown ran from a double-colon all-hook:: rule with no prerequisites, so with -j it could execute in parallel with (or before) the actual build, killing nothing. - Build servers started by later subdirectories (e.g. tests/) after the dotnet/ shutdown were never killed. - The process-matching regex pattern might not match all server processes. The fix: disable build servers entirely via environment variables in Make.config: - DOTNET_CLI_USE_MSBUILD_SERVER=0: prevents the MSBuild server https://learn.microsoft.com/en-us/visualstudio/msbuild/msbuild-server https://github.com/dotnet/msbuild/blob/main/documentation/MSBuild-Server.md - UseSharedCompilation=false: prevents the Roslyn compiler server (VBCSCompiler) dotnet/roslyn#27975 - MSBUILDDISABLENODEREUSE=1: prevents MSBuild node reuse https://github.com/dotnet/msbuild/wiki/MSBuild-Tips-&-Tricks This eliminates the root cause - no background servers means no inherited file descriptors means no hang. The shutdown-build-server target and its invocations are removed as they are no longer needed. Additionally, 'make world' now prints the installed workloads at the end of the build for visibility. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
) Parallel make (e.g. 'make all -j8', 'make world') has been hanging for a while at the end of the build. This is a long-standing issue (#13355) that has been patched three times (#15407, #21315, #22300) without fully fixing the root cause. The problem: when using parallel make, GNU Make uses a jobserver with pipe-based file descriptors to coordinate sub-makes. The dotnet CLI can start background build servers (MSBuild server, Roslyn/VBCSCompiler) that inherit these file descriptors but never close them. Make then waits for those file descriptors to close (which won't happen until the servers exit - which they typically do about 10 minutes without activity), thinking there are still active jobs. The previous workaround attempted to shut down and force-kill dotnet processes after the build via a 'shutdown-build-server' target. This approach was unreliable because: - The shutdown ran from a double-colon all-hook:: rule with no prerequisites, so with -j it could execute in parallel with (or before) the actual build, killing nothing. - Build servers started by later subdirectories (e.g. tests/) after the dotnet/ shutdown were never killed. - The process-matching regex pattern might not match all server processes. Ideally this would be fixed in when launching the build servers, by making them not inherit handles. Unfortunately this is currently not possible: dotnet/runtime#13943 (although this might change in a not so distant future: dotnet/runtime#123959) The workaround: disable build servers entirely via environment variables in Make.config: - DOTNET_CLI_USE_MSBUILD_SERVER=0: prevents the MSBuild server https://learn.microsoft.com/en-us/visualstudio/msbuild/msbuild-server https://github.com/dotnet/msbuild/blob/main/documentation/MSBuild-Server.md - UseSharedCompilation=false: prevents the Roslyn compiler server (VBCSCompiler) dotnet/roslyn#27975 - MSBUILDDISABLENODEREUSE=1: prevents MSBuild node reuse https://github.com/dotnet/msbuild/wiki/MSBuild-Tips-&-Tricks This eliminates the root cause - no background servers means no inherited file descriptors means no hang. The shutdown-build-server target and its invocations are removed as they are no longer needed. Additionally, 'make world' now prints the installed workloads at the end of the build for visibility. Build without changes: > make world 2149.57s user 258.32s system 107% cpu 37:30.19 total Build with changes: > make world 2242.74s user 286.38s system 354% cpu 11:52.55 total Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
4 participants
Hopefully this will fix the build randomly hanging at the end of 'make all'.