Remove Homebrew LLVM during macOS CI setup by steveisok · Pull Request #125763 · dotnet/runtime

steveisok · 2026-03-19T05:04:09Z

The macos-15 CI runner image ships with Homebrew LLVM 18, which installs libraries (e.g., an x86_64-only libunwind.dylib) into /usr/local/lib. The macOS linker searches /usr/local/lib by default, finds the wrong-architecture library, rejects it, and leaves symbols like _unw_step and __Unwind_Resume unresolved — breaking all native linking on macOS. In addition, it'll use default to the wrong clang during compile and that causes additional havoc.

The build uses Apple clang from /usr/bin/clang exclusively and does not need Homebrew LLVM. This PR removes it during CI setup via brew uninstall in install-dependencies.sh.

Fixes #125757

The macos-15 runner image (20260317.0278) upgraded LLVM from 17 to 18, placing clang-18 at /usr/local/bin/clang-18. The init-compiler.sh version search loop finds this upstream LLVM before falling through to unversioned clang (AppleClang), causing three classes of build failures: - NativeAOT: invalid CFI advance_loc assembly errors (LLVM 18 stricter) - Native libs: Swift linker undefined symbols (missing Apple Swift paths) - CoreCLR: CMake LINK_GROUP RESCAN unsupported (LLVM linker vs Apple ld) On Darwin, use xcrun --find clang to resolve the Xcode toolchain clang (AppleClang) instead of searching PATH for versioned LLVM binaries. This matches the pattern Mono already uses in mono.proj. The existing CLR_CC/CLR_CXX override and explicit version requests (e.g. clang-18) are preserved as escape hatches. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The previous commit used xcrun --find clang to resolve the Xcode toolchain clang directly. While this correctly avoids Homebrew LLVM, the raw Xcode clang does not inject -isysroot automatically, causing the linker to fail to find system libraries (libdl, libobjc, libswiftCore, etc.) on macOS 26 with Xcode 26. Switch to /usr/bin/clang, the Apple-provided shim that always delegates to the active Xcode's AppleClang and automatically injects -isysroot pointing to the macOS SDK. This preserves the Homebrew LLVM avoidance while ensuring native linking works correctly.

Copilot

Copilot wasn't able to review any files in this pull request.

You can also share your feedback on Copilot code review. Take the survey.

dotnet-policy-service · 2026-03-19T05:05:12Z

Tagging subscribers to this area: @agocke, @dotnet/ilc-contrib
See info in area-owners.md if you want to be subscribed.

jkotas · 2026-03-19T09:38:07Z

Looks like the fix is incomplete - the tests are failing due to references to llvm@18 libraries:

  dlopen(/private/tmp/helix/working/AB0A0904/w/BE7B0A98/e/test_artifacts/104eqhzt.4df/HelloWorld/extract-1/HelloWorld/AU5QZNzEE-Ya/libmockhostpolicy.dylib, 0x0001): Library not loaded: /usr/local/opt/llvm@18/lib/libunwind.1.dylib
    Referenced from: <E5013D17-9628-3D53-B876-6B996E88F598> /private/tmp/helix/working/AB0A0904/w/BE7B0A98/e/test_artifacts/104eqhzt.4df/HelloWorld/extract-1/HelloWorld/AU5QZNzEE-Ya/libmockhostpolicy.dylib
    Reason: tried: '/usr/local/opt/llvm@18/lib/libunwind.1.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/usr/local/opt/llvm@18/lib/libunwind.1.dylib' (no such file), '/usr/local/opt/llvm@18/lib/libunwind.1.dylib' (no such file)

eng/common/native/init-compiler.sh

gen-buildsys.sh sources init-compiler.sh which resolves the correct compiler into CC/CXX, but only CCC_CC/CCC_CXX (scan-build variables) were exported. CMake ignores those and falls back to PATH-based detection, which can find Homebrew LLVM 18 before Apple clang on macOS CI runners. This caused native host test libraries like libmockhostpolicy.dylib to link against Homebrew's libunwind.

Copilot

Pull request overview

This PR adjusts macOS native build configuration to prefer Apple’s /usr/bin/clang shim and avoid accidentally picking up Homebrew LLVM/linker behavior, reducing recent CI failures on macOS runners.

Changes:

Stop adding /usr/local/include for Apple targets to avoid pulling in Homebrew headers/libs.
Force a macOS SDK sysroot for CMake on osx builds and prefer /usr/bin/clang on Darwin when using default clang.
Propagate compiler env vars from gen-buildsys.sh to improve consistency of compiler selection.

Reviewed changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/native/libs/configure.cmake	Avoids injecting `/usr/local/include` on Apple targets to prevent Homebrew header/library conflicts.
src/coreclr/pal/src/CMakeLists.txt	Skips `/usr/local/include` for Apple targets in PAL build.
src/coreclr/debug/createdump/CMakeLists.txt	Skips `/usr/local/include` for Apple targets in createdump build.
eng/native/gen-buildsys.sh	Exports compiler variables for downstream CMake invocation.
eng/native/build-commons.sh	Forces `CMAKE_OSX_SYSROOT=macosx` for `osx` builds to stabilize header/lib resolution.
eng/common/native/init-compiler.sh	Prefers `/usr/bin/clang` shim on Darwin for default clang and clears `LIBRARY_PATH` to avoid Homebrew leakage.

You can also share your feedback on Copilot code review. Take the survey.

eng/native/gen-buildsys.sh

The CI runner image may ship with a Homebrew LLVM whose libraries (e.g., an x86_64-only libunwind.dylib in /usr/local/lib) conflict with the Apple SDK and break native linking. The build uses Apple clang from /usr/bin/clang exclusively and does not need Homebrew LLVM.

eng/common/native/install-dependencies.sh

danmoseley · 2026-03-19T15:23:05Z

still getting " Feature 'RESCAN', specified through generator-expression '$<LINK_GROUP>' to"

akoeplinger · 2026-03-19T15:28:39Z

yeah. we'd need to brew uninstall llvm@18 but that still leaves clang pointing to homebrew clang for some reason which doesn't exist then - Steve created an IcM: https://portal.microsofticm.com/imp/v5/incidents/details/765018880/summary

Copilot

Copilot wasn't able to review any files in this pull request.

You can also share your feedback on Copilot code review. Take the survey.

akoeplinger · 2026-03-19T16:01:42Z

but that still leaves clang pointing to homebrew clang for some reason which doesn't exist then

ok this was just due to how I tested - if you run clang -v before brew uninstall then bash will cache the original path to clang

steveisok · 2026-03-19T16:43:16Z

/backport to release/9.0

steveisok · 2026-03-19T16:43:24Z

/backport to release/8.0

github-actions · 2026-03-19T16:43:25Z

Started backporting to release/9.0 (link to workflow run)

github-actions · 2026-03-19T16:43:34Z

Started backporting to release/8.0 (link to workflow run)

github-actions · 2026-03-19T16:45:22Z

@steveisok backporting to release/9.0 failed, the patch most likely resulted in conflicts. Please backport manually!

git am output

$ git am --3way --empty=keep --ignore-whitespace --keep-non-patch changes.patch

Applying: Prefer AppleClang from Xcode toolchain on macOS
Applying: Use Apple toolchain shim instead of raw Xcode clang on macOS
Applying: Export CC and CXX in gen-buildsys.sh so CMake uses the resolved compiler
Applying: Remove Homebrew LLVM during macOS CI setup
error: sha1 information is lacking or useless (eng/native/gen-buildsys.sh).
error: could not build fake ancestor
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config set advice.mergeConflict false"
Patch failed at 0004 Remove Homebrew LLVM during macOS CI setup
Error: The process '/usr/bin/git' failed with exit code 128

Link to workflow output

github-actions · 2026-03-19T16:45:32Z

@steveisok backporting to release/8.0 failed, the patch most likely resulted in conflicts. Please backport manually!

git am output

$ git am --3way --empty=keep --ignore-whitespace --keep-non-patch changes.patch

Applying: Prefer AppleClang from Xcode toolchain on macOS
Using index info to reconstruct a base tree...
M	eng/common/native/init-compiler.sh
Falling back to patching base and 3-way merge...
Auto-merging eng/common/native/init-compiler.sh
CONFLICT (content): Merge conflict in eng/common/native/init-compiler.sh
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config set advice.mergeConflict false"
Patch failed at 0001 Prefer AppleClang from Xcode toolchain on macOS
Error: The process '/usr/bin/git' failed with exit code 128

Link to workflow output

The CI runner image may ship with a Homebrew LLVM whose libraries (e.g., an x86_64-only libunwind.dylib in /usr/local/lib) conflict with the Apple SDK and break native linking. The build uses Apple clang from /usr/bin/clang exclusively and does not need Homebrew LLVM. Backport of dotnet#125763

The CI runner image may ship with a Homebrew LLVM whose libraries (e.g., an x86_64-only libunwind.dylib in /usr/local/lib) conflict with the Apple SDK and break native linking. The build uses Apple clang from /usr/bin/clang exclusively and does not need Homebrew LLVM. Backport of #125763

The CI runner image may ship with a Homebrew LLVM whose libraries (e.g., an x86_64-only libunwind.dylib in /usr/local/lib) conflict with the Apple SDK and break native linking. The build uses Apple clang from /usr/bin/clang exclusively and does not need Homebrew LLVM. Backport of dotnet#125763

…25790) The CI runner image may ship with a Homebrew LLVM whose libraries (e.g., an x86_64-only libunwind.dylib in /usr/local/lib) conflict with the Apple SDK and break native linking. The build uses Apple clang from /usr/bin/clang exclusively and does not need Homebrew LLVM. Backport of #125763 Co-authored-by: vseanreesermsft <78103370+vseanreesermsft@users.noreply.github.com>

…25789) The CI runner image may ship with a Homebrew LLVM whose libraries (e.g., an x86_64-only libunwind.dylib in /usr/local/lib) conflict with the Apple SDK and break native linking. The build uses Apple clang from /usr/bin/clang exclusively and does not need Homebrew LLVM. Backport of #125763 Co-authored-by: vseanreesermsft <78103370+vseanreesermsft@users.noreply.github.com>

steveisok and others added 2 commits March 18, 2026 23:06

Copilot AI review requested due to automatic review settings March 19, 2026 05:04

Copilot AI reviewed Mar 19, 2026

View reviewed changes

github-actions bot added the area-NativeAOT-coreclr label Mar 19, 2026

dotnet-policy-service bot assigned steveisok Mar 19, 2026

steveisok requested a review from a team March 19, 2026 05:04

jkoritzinsky approved these changes Mar 19, 2026

View reviewed changes

hoyosjs approved these changes Mar 19, 2026

View reviewed changes

This was referenced Mar 19, 2026

wasm build failure in CI #117017

Open

MsQuic fails with QUIC_STATUS_OUT_OF_MEMORY on AzureLinux #123216

Open

jkotas mentioned this pull request Mar 19, 2026

Try checking for Darwin and Clang in addition to AppleClang #125758

Closed

jkotas reviewed Mar 19, 2026

View reviewed changes

eng/common/native/init-compiler.sh Show resolved Hide resolved

kotlarmilos mentioned this pull request Mar 19, 2026

[Mono] mini-llvm.c build failure: LLVMMono1CallConv undeclared after LLVM SDK version bump to 26159.1 #125769

Open

Copilot AI review requested due to automatic review settings March 19, 2026 13:23

Copilot AI reviewed Mar 19, 2026

View reviewed changes

eng/native/gen-buildsys.sh Outdated Show resolved Hide resolved

steveisok force-pushed the fix/macos-explicit-appleclang branch from b32c396 to 5c5fdb4 Compare March 19, 2026 14:04

steveisok changed the title ~~Prefer Apple clang shim on macOS to avoid Homebrew LLVM and linker failures~~ Remove Homebrew LLVM during macOS CI setup Mar 19, 2026

vcsjones reviewed Mar 19, 2026

View reviewed changes

eng/common/native/install-dependencies.sh Outdated Show resolved Hide resolved

Change Homebrew LLVM uninstallation to llvm@18

5791617

Copilot AI review requested due to automatic review settings March 19, 2026 15:59

Copilot AI reviewed Mar 19, 2026

View reviewed changes

build-analysis bot mentioned this pull request Mar 19, 2026

osx builds failing with CMake error #125757

Closed

This was referenced Mar 19, 2026

[release/8.0] Remove Homebrew LLVM during macOS CI setup #125783

Merged

[release/8.0-staging] Remove Homebrew LLVM during macOS CI setup - Part 2 #125784

Open

[release/9.0] Remove Homebrew LLVM during macOS CI setup #125785

Merged

This was referenced Mar 19, 2026

[release/9.0] Remove Homebrew LLVM during macOS CI setup - Part 2 #125789

Merged

[release/8.0] Remove Homebrew LLVM during macOS CI setup - Part 2 #125790

Merged

dotnet-maestro bot mentioned this pull request Mar 20, 2026

[main] Source code updates from dotnet/runtime dotnet/dotnet#5588

Open

Conversation

steveisok commented Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

dotnet-policy-service bot commented Mar 19, 2026

Uh oh!

jkotas commented Mar 19, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

danmoseley commented Mar 19, 2026

Uh oh!

akoeplinger commented Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

akoeplinger commented Mar 19, 2026

Uh oh!

steveisok commented Mar 19, 2026

Uh oh!

steveisok commented Mar 19, 2026

Uh oh!

github-actions bot commented Mar 19, 2026

Uh oh!

github-actions bot commented Mar 19, 2026

Uh oh!

github-actions bot commented Mar 19, 2026

Uh oh!

github-actions bot commented Mar 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

steveisok commented Mar 19, 2026 •

edited

Loading

akoeplinger commented Mar 19, 2026 •

edited

Loading