[build] Fix parallel make hang by disabling dotnet build servers#24903
[build] Fix parallel make hang by disabling dotnet build servers#24903
Conversation
Parallel make (e.g. 'make all -j8', 'make world') has been hanging indefinitely at the end of the build. This is a long-standing issue (#13355) that has been patched three times (#15407, #21315, #22300) without fully fixing the root cause. The problem: when using parallel make, GNU Make uses a jobserver with pipe-based file descriptors to coordinate sub-makes. The dotnet CLI can start background build servers (MSBuild server, Roslyn/VBCSCompiler) that inherit these file descriptors but never close them. Make then waits indefinitely for those file descriptors to close, thinking there are still active jobs. The previous workaround attempted to shut down and force-kill dotnet processes after the build via a 'shutdown-build-server' target. This approach was unreliable because: - The shutdown ran from a double-colon all-hook:: rule with no prerequisites, so with -j it could execute in parallel with (or before) the actual build, killing nothing. - Build servers started by later subdirectories (e.g. tests/) after the dotnet/ shutdown were never killed. - The process-matching regex pattern might not match all server processes. The fix: disable build servers entirely via environment variables in Make.config: - DOTNET_CLI_USE_MSBUILD_SERVER=0: prevents the MSBuild server https://learn.microsoft.com/en-us/visualstudio/msbuild/msbuild-server https://github.com/dotnet/msbuild/blob/main/documentation/MSBuild-Server.md - UseSharedCompilation=false: prevents the Roslyn compiler server (VBCSCompiler) dotnet/roslyn#27975 - MSBUILDDISABLENODEREUSE=1: prevents MSBuild node reuse https://github.com/dotnet/msbuild/wiki/MSBuild-Tips-&-Tricks This eliminates the root cause - no background servers means no inherited file descriptors means no hang. The shutdown-build-server target and its invocations are removed as they are no longer needed. Additionally, 'make world' now prints the installed workloads at the end of the build for visibility. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
✅ [CI Build #d7a5d48] Build passed (Build packages) ✅Pipeline on Agent |
✅ [PR Build #d7a5d48] Build passed (Detect API changes) ✅Pipeline on Agent |
✅ [CI Build #d7a5d48] Build passed (Build macOS tests) ✅Pipeline on Agent |
✅ API diff for current PR / commitNET (empty diffs)✅ API diff vs stableNET (empty diffs)ℹ️ Generator diffGenerator Diff: vsdrops (html) vsdrops (raw diff) gist (raw diff) - Please review changes) Pipeline on Agent |
This comment has been minimized.
This comment has been minimized.
🚀 [CI Build #d7a5d48] Test results 🚀Test results✅ All tests passed on VSTS: test results. 🎉 All 156 tests passed 🎉 Tests counts✅ cecil: All 1 tests passed. Html Report (VSDrops) Download macOS tests✅ Tests on macOS Monterey (12): All 5 tests passed. Html Report (VSDrops) Download Pipeline on Agent |
3 participants
Parallel make (e.g. 'make all -j8', 'make world') has been hanging
for a while at the end of the build. This is a long-standing issue
(#13355) that has been patched three times
(#15407, #21315, #22300) without fully fixing the root cause.
The problem: when using parallel make, GNU Make uses a jobserver with
pipe-based file descriptors to coordinate sub-makes. The dotnet CLI
can start background build servers (MSBuild server, Roslyn/VBCSCompiler)
that inherit these file descriptors but never close them. Make then
waits for those file descriptors to close (which won't happen until
the servers exit - which they typically do about 10 minutes without
activity), thinking there are still active jobs.
The previous workaround attempted to shut down and force-kill dotnet
processes after the build via a 'shutdown-build-server' target. This
approach was unreliable because:
prerequisites, so with -j it could execute in parallel with (or
before) the actual build, killing nothing.
the dotnet/ shutdown were never killed.
Ideally this would be fixed in when launching the build servers, by
making them not inherit handles. Unfortunately this is currently not
possible: dotnet/runtime#13943 (although this might change in a not so
distant future: dotnet/runtime#123959)
The workaround: disable build servers entirely via environment variables in
Make.config:
https://learn.microsoft.com/en-us/visualstudio/msbuild/msbuild-server
https://github.com/dotnet/msbuild/blob/main/documentation/MSBuild-Server.md
Turn on UseSharedCompilation by default in the Microsoft.NET*.Compilers packages roslyn#27975
https://github.com/dotnet/msbuild/wiki/MSBuild-Tips-&-Tricks
This eliminates the root cause - no background servers means no
inherited file descriptors means no hang. The shutdown-build-server
target and its invocations are removed as they are no longer needed.
Additionally, 'make world' now prints the installed workloads at the
end of the build for visibility.
Build without changes:
Build with changes: