Skip to content

[CI] macOS build improvements#2884

Merged
fhanau merged 3 commits into
mainfrom
felix/100924-macOS-ci
Oct 10, 2024
Merged

[CI] macOS build improvements#2884
fhanau merged 3 commits into
mainfrom
felix/100924-macOS-ci

Conversation

@fhanau
Copy link
Copy Markdown
Contributor

@fhanau fhanau commented Oct 10, 2024

  • [nfc] Drop duplicated macOS experimental_objc_fastbuild_options flag

  • [CI] Use macos-15 image for test job, implicitly transitioning to Apple Silicon

    To mitigate our problems with macOS runner rate limiting, switch to the
    macos-15 runner image. This means we are implicitly transitioning the test job
    to Apple Silicon (macos-13 is x86, macos-15 is Apple Silicon with no x86 option
    available outside of paid runners).
    This allows us to take advantage of the latest compiler features available in
    Xcode 16 (based on LLVM 17), which is not available for macOS 13. Xcode has
    good backwards compatibility; based on the macos_minimum_os flag we will
    continue to support macOS 13 and later.

    Interestingly, the macos-15 build is much faster (~2x based on limited testing),
    so this will help avoid having stalled macOS build jobs. This is likely due to
    the Apple Silicon CPUs being more performant, although the new compiler version
    could also be helping.

    The release job continues to use the macos-13/x86 image, so we continue to have
    coverage for x86. If this proves to be stable, we can also add an Apple Silicon
    release build and decommission the internal arm64 macOS build.

  • Use lld to reduce macOS release binary size

@anonrig
Copy link
Copy Markdown
Contributor

anonrig commented Oct 10, 2024

I've heard about examples where latest apple silicon runners that use macOS 14 can produce binaries that segfault when run on macOS 11/12/13. I don't know if this statement still holds, but FYI. For example: astral-sh/ruff#9834

run: |
sudo xcode-select -s "/Applications/Xcode_15.1.app"
# Install lld
brew install lld
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just fyi, this installs llvm 19. i'm not sure if it will cause any compatibility problems. leaving it as a small note.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right. This should be fine as the release build job doesn't use homebrew LLVM at all, the bazel toolchain is configured solely based on the clang/llvm available in the Xcode installation. Installing homebrew LLVM 19 instead of the default version should not be an issue (I initially included this command in the test job too to try it out, seems to have worked flawlessly. I primarily want to install LLD to make release builds smaller, so this is not needed for the regular test run).

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to add this install to the README?

Copy link
Copy Markdown
Contributor Author

@fhanau fhanau Oct 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will only be needed for macOS release builds. Developers generally shouldn't need to build those regularly, but I think it should still be supported without needing more stuff – ended up removing LLD from the release_macos config and manually adding it in the release job instead.

@fhanau
Copy link
Copy Markdown
Contributor Author

fhanau commented Oct 10, 2024

I've heard about examples where latest apple silicon runners that use macOS 14 can produce binaries that segfault when run on macOS 11/12/13. I don't know if this statement still holds, but FYI. For example: astral-sh/ruff#9834

Yes, I can explain why that happened and why it won't be happening here:
Xcode 15 introduced a linker bug that can cause C++ programs to crash on macOS 12 or older (the linked issue mentions "macOS 11 (at least)" and only hypothesizes that this also affects macOS 13). This was fixed in Xcode 15.1 https://developer.apple.com/documentation/xcode-release-notes/xcode-15_1-release-notes#Resolved-Issues. After Xcode 15.1 was released, I manually updated from Xcode 14.3 (previous release, unaffected) to that Xcode version, which is already available on macOS 13 (#1496).
Unfortunately, Xcode is not well maintained on the Github Actions runner image, so the default Xcode version on the macos-14 image used in the linked issue stayed at the buggy 15.0.1 version until July, 7 months after the fixed version was released actions/runner-images@cea9d0e. That explains why crashes were encountered for binaries built on that image.

With this PR, we will be updating from the fixed version (15.1, the last Xcode version with toolchain updates that is supported on macOS 13) to the latest available version 16.0 (available on the macOS 14 and 15, later updates will only be available on 15). The Xcode 16.1 Beta release notes do not include major bug fixes, suggesting that Xcode 16 is reasonably stable.

@fhanau fhanau force-pushed the felix/100924-macOS-ci branch from 0ca6200 to e25f1b6 Compare October 10, 2024 01:58
@anonrig
Copy link
Copy Markdown
Contributor

anonrig commented Oct 10, 2024

That was educational and fun to read. Thanks @fhanau!

@fhanau fhanau force-pushed the felix/100924-macOS-ci branch 2 times, most recently from 96cb9f1 to 0a65d79 Compare October 10, 2024 15:34
@fhanau fhanau marked this pull request as ready for review October 10, 2024 15:35
@fhanau fhanau requested review from a team as code owners October 10, 2024 15:35
@fhanau fhanau changed the title [DRAFT][CI] macOS build improvements [CI] macOS build improvements Oct 10, 2024
…le Silicon

To mitigate our problems with macOS runner rate limiting, switch to the
macos-15 runner image. This means we are implicitly transitioning the test job
to Apple Silicon (macos-13 is x86, macos-15 is Apple Silicon with no x86 option
available outside of paid runners).
This allows us to take advantage of the latest compiler features available in
Xcode 16 (based on LLVM 17), which is not available for macOS 13. Xcode has
good backwards compatibility; based on the macos_minimum_os flag we will
continue to support macOS 13 and later.

Interestingly, the macos-15 build is much faster (~2x based on limited testing),
so this will help avoid having stalled macOS build jobs. This is likely due to
the Apple Silicon CPUs being more performant, although the new compiler version
could also be helping.

The release job continues to use the macos-13/x86 image, so we continue to have
coverage for x86. If this proves to be stable, we can also add an Apple Silicon
release build and decommission the internal arm64 macOS build.
@fhanau fhanau force-pushed the felix/100924-macOS-ci branch from 0a65d79 to a9f950b Compare October 10, 2024 18:23
@fhanau fhanau merged commit e4a4d4d into main Oct 10, 2024
@fhanau fhanau deleted the felix/100924-macOS-ci branch October 10, 2024 18:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants