tools/ci/platforms/darwin.sh: Port NuttX CI Job to macOS-14 #14691

simbit18 · 2024-11-07T14:27:54Z

Summary

Now you can use any of these runners

macos-13
https://github.com/actions/runner-images/blob/main/images/macos/macos-13-Readme.md

or

macos-14 (arm64)
https://github.com/actions/runner-images/blob/main/images/macos/macos-14-Readme.md

added autoconf
needed to build kconfig_frontends on macos-14

bloaty
speeded build with cmake + ninja

if we upgrade to macos-14 we need to remember to clear the caches (macOS-tools-xxxxxxxx) first
https://github.com/apache/nuttx/actions/caches

Impact

Impact on user: No changes to user-facing functionality
Impact on build: Build process remains the same

Testing

tested only on github workflow with only [macos] because sim01, sim02, sim03 are broken !!!

macos-13
github macos-13

macos-14
github macos-14

also runs old CI Job macOS-13

simbit18 · 2024-11-07T14:33:53Z

Hi @lupyuen can you run a test on real macOS devices ?

lupyuen · 2024-11-07T14:35:42Z

@simbit18 I don't have an Intel Mac though. This will work on M1 Mac?

simbit18 · 2024-11-07T14:42:41Z

macos-14 Today, GitHub is excited to announce the launch of a new M1 macOS runner!
https://github.blog/changelog/2024-01-30-github-actions-introducing-the-new-m1-macos-runner-available-to-open-source/

lupyuen · 2024-11-07T15:50:31Z

Thanks @simbit18! On my Mac Mini M2 Pro, it fails while building gperf:

g++ -g -O2 -I. -I./../lib -c ./main.cc
g++ -g -O2  -o gperf version.o positions.o options.o keyword.o keyword-list.o input.o bool-array.o hash-table.o search.o output.o main.o ../lib/libgp.a -lm
ld: archive member '/' not a mach-o file in '/Users/luppy/simbit18-macos-arm64/tools/gperf-3.1/lib/libgp.a'
clang++: error: linker command failed with exit code 1 (use -v to see invocation)

https://gist.github.com/lupyuen/9e49ad8f91e2f2e9b76c41eb583664d9

simbit18 · 2024-11-07T16:16:30Z

@lupyuen maybe related to this ?
https://stackoverflow.com/a/77630733

I see this is a common problem :)
https://forums.developer.apple.com/forums/thread/741149

possible solution
pyenv/pyenv#2862

lupyuen · 2024-11-07T17:57:36Z

I just restarted the job so it will pick up the fix from apache/nuttx-apps#2829. And it won't run all 6 hours :-)

lupyuen · 2024-11-08T02:17:23Z

OK making good progress! I changed this in tools/ci/platforms/darwin.sh

add_path() {
  PATH=$1:${PATH}
  ## TODO: Remove Homebrew ar from PATH
  PATH=$(
    echo $PATH \
      | tr ':' '\n' \
      | grep -v "/opt/homebrew/opt/make/libexec/gnubin" \
      | grep -v "/opt/homebrew/opt/coreutils/libexec/gnubin" \
      | grep -v "/opt/homebrew/opt/binutils/bin" \
      | tr '\n' ':'
  )
  echo "**** PATH=$PATH" | tr ':' '\n'
  ## Should show /usr/bin/ar
  which ar 
}

Now it fails while installing Cython:

+ python3 -m venv --system-site-packages /opt/homebrew
+ pip3 install 'Cython<3.0'
[notice] A new release of pip is available: 24.2 -> 24.3.1
[notice] To update, run: python3.13 -m pip install --upgrade pip
error: externally-managed-environment
× This environment is externally managed
╰─> To install Python packages system-wide, try brew install

https://gist.github.com/lupyuen/012515d912b8524d29a01793a256b485

simbit18 · 2024-11-08T08:46:15Z

Now it fails while installing Cython:

+ python3 -m venv --system-site-packages /opt/homebrew
+ pip3 install 'Cython<3.0'
[notice] A new release of pip is available: 24.2 -> 24.3.1
[notice] To update, run: python3.13 -m pip install --upgrade pip
error: externally-managed-environment
× This environment is externally managed
╰─> To install Python packages system-wide, try brew install

@lupyuen for this problem i try to test on runner macos-14

lupyuen · 2024-11-08T08:50:35Z

Thank you so much! I see 3 different styles of venv in our CI Workflows, wondering which one we should use: https://github.com/search?q=repo%3Aapache%2Fnuttx%20venv&type=code

simbit18 · 2024-11-08T09:20:46Z

this is what I use locally (msys2 and Alpine) for create a virtual environment in Python
python3 -m venv --system-site-packages /usr/local

the problem is the path if you change it it should work

on macos-14
Error: [Errno 13] Permission denied: '/usr/local/include'

simbit18 · 2024-11-08T10:24:55Z

If you change path the test on github macos-14 is ok

workflow
https://github.com/simbit18/nuttx_test_pr/actions/runs/11740209647/workflow

https://github.com/simbit18/nuttx_test_pr/actions/runs/11740209647/job/32706315729

we need to define a path that is ok for both (github and real macOS devices)

lupyuen · 2024-11-08T11:03:37Z

I don't quite understand what the code below is doing. PIP_USER and PYTHONUSERBASE seem to be interfering with the venv? This runs OK:

python_tools() {
  #### TODO: Previously failed to install Cython
  # # Python User Env
  # export PIP_USER=yes
  # export PYTHONUSERBASE=${NUTTXTOOLS}/pylocal
  # echo "export PIP_USER=yes" >> "${NUTTXTOOLS}"/env.sh
  # echo "export PYTHONUSERBASE=${NUTTXTOOLS}/pylocal" >> "${NUTTXTOOLS}"/env.sh
  # add_path "${PYTHONUSERBASE}"/bin
  
  # if [ "X$osarch" == "Xarm64" ]; then
  #   python3 -m venv --system-site-packages /opt/homebrew
  # fi
  
  #### TODO: Now it installs Cython OK!
  python3 -m venv .venv
  source .venv/bin/activate

  # workaround for Cython issue
  # https://github.com/yaml/pyyaml/pull/702#issuecomment-1638930830
  pip3 install "Cython<3.0"

lupyuen · 2024-11-08T11:07:13Z

CI Build on Mac M2 Pro works OK yay! (with the above fixes)

$ ./cibuild.sh -i -c -A -R testlist/macos.dat
...
====================================================================================
Configuration/Tool: stm32f4discovery/nsh,CONFIG_ARM_TOOLCHAIN_GNU_EABI
2024-11-08 19:04:16
------------------------------------------------------------------------------------
  Cleaning...
  Configuring...
  Disabling CONFIG_ARM_TOOLCHAIN_GNU_EABI
  Enabling CONFIG_ARM_TOOLCHAIN_GNU_EABI
  Building NuttX...

https://gist.github.com/lupyuen/6d25cfa16b8920b515dcc14835e7a479

Here's the modified script: darwin.sh.txt

simbit18 · 2024-11-08T11:14:07Z

I don't quite understand what the code below is doing. PIP_USER and PYTHONUSERBASE seem to be interfering with the venv? This runs OK:

python_tools() {
  #### TODO: Previously failed to install Cython
  # # Python User Env
  # export PIP_USER=yes
  # export PYTHONUSERBASE=${NUTTXTOOLS}/pylocal
  # echo "export PIP_USER=yes" >> "${NUTTXTOOLS}"/env.sh
  # echo "export PYTHONUSERBASE=${NUTTXTOOLS}/pylocal" >> "${NUTTXTOOLS}"/env.sh
  # add_path "${PYTHONUSERBASE}"/bin
  
  # if [ "X$osarch" == "Xarm64" ]; then
  #   python3 -m venv --system-site-packages /opt/homebrew
  # fi
  
  #### TODO: Now it installs Cython OK!
  python3 -m venv .venv
  source .venv/bin/activate

  # workaround for Cython issue
  # https://github.com/yaml/pyyaml/pull/702#issuecomment-1638930830
  pip3 install "Cython<3.0"

This should be used for caches on github
https://github.com/simbit18/nuttx/blob/5609c3d4166905b224020ef70c5847367616889a/.github/workflows/build.yml#L227C7-L234C103

name: Restore Tools Cache
id: cache-tools
uses: actions/cache@v4
env:
cache-name: ${{ runner.os }}-cache-tools
with:
path: ./sources/tools
key: ${{ runner.os }}-tools-${{ hashFiles('./sources/nuttx/tools/ci/platforms/darwin.sh') }}

lupyuen · 2024-11-08T11:18:28Z

OK the CI Build is done! https://gist.github.com/lupyuen/6d25cfa16b8920b515dcc14835e7a479

The downloading wasn't that slow, and the GitHub Cache won't work on my Mac anyway. I'll probably disable that chunk of code when I run my Mac Build Farm :-)

lupyuen · 2024-11-08T13:15:54Z

I'm super impressed: arm-01 will actually build on Mac M2 Pro!

 $ ./cibuild.sh -i -c -A -R testlist/arm-01.dat
  Enabling CONFIG_ARM_TOOLCHAIN_GNU_EABI
  Building NuttX...
  Normalize pcduino-a10/nsh

https://gist.github.com/lupyuen/e2babbcad9a102fcb002ca6dcf361895

Build for arm-01 completes in 1 hour. That's the same speed as GitHub Actions!

Finally all my CPUs are getting purposefully busy :-)

lupyuen · 2024-11-08T23:20:34Z

FYI I'm running the script below to patch darwin.sh for my Mac Mini M2 Pro:

Change the toolchains from x86_64 macOS to Arm64 macOS
Remove Homebrew ar from PATH
Fix the Python Environment

The patched darwin.sh looks like this: darwin.sh.txt

When we change the Toolchains from x86_64 macOS to Arm64 macOS:

Build for arm-01 completes in 1 hour, same as earlier (See the log)
CPU Utilisation drops a little (compare with above)

https://github.com/lupyuen/nuttx-build-farm/blob/main/patch-ci-macos.sh

## How to run CI Build:
## rm -rf NUTTX_PARENT_FOLDER/tools
## ./cibuild.sh -i -c -A -R testlist/macos.dat
## ./cibuild.sh -i -c -A -R testlist/arm-01.dat

set -e  #  Exit when any command fails
set -x  #  Echo commands

## Create the Temp Folder
tmp_dir=/tmp/macos-build-farm
rm -rf $tmp_dir
mkdir $tmp_dir

## We shall rewrite darwin.sh
file=tools/ci/platforms/darwin.sh
tmp_file=$tmp_dir/darwin.yml

## Search and replace in the file
function rewrite_file {
  cat $file \
    | sed "s/$search/$replace/g" \
    >$tmp_file
  mv $tmp_file $file
}

## Change: basefile=arm-gnu-toolchain-13.2.rel1-darwin-x86_64-arm-none-eabi
## To:     basefile=arm-gnu-toolchain-13.2.rel1-darwin-arm64-arm-none-eabi #### TODO
search='-darwin-x86_64-arm-none-eabi'
replace='-darwin-arm64-arm-none-eabi #### TODO: We changed to Arm64 macOS'
rewrite_file

## Change: basefile=arm-gnu-toolchain-13.2.Rel1-darwin-x86_64-aarch64-none-elf
## To:     basefile=arm-gnu-toolchain-13.2.Rel1-darwin-arm64-aarch64-none-elf #### TODO
search='-darwin-x86_64-aarch64-none-elf'
replace='-darwin-arm64-aarch64-none-elf #### TODO: We changed to Arm64 macOS'
rewrite_file

## Change: basefile=xpack-riscv-none-elf-gcc-13.2.0-2-darwin-x64
## To:     basefile=xpack-riscv-none-elf-gcc-13.2.0-2-darwin-arm64 #### TODO
search='-darwin-x64'
replace='-darwin-arm64 #### TODO: We changed to Arm64 macOS'
rewrite_file

## Change: basefile=xtensa-esp32-elf-12.2.0_20230208-x86_64-apple-darwin
## To:     basefile=xtensa-esp32-elf-12.2.0_20230208-aarch64-apple-darwin #### TODO
search='-x86_64-apple-darwin'
replace='-aarch64-apple-darwin #### TODO: We changed to Arm64 macOS'
rewrite_file

## Change: add_path() {
## To:     add_path() { ... } \r NOTUSED_add_path() {
search='^add_path() {'
replace=$(
cat <<'EOF' | tr '\n' '\r'
set -x  #  Echo commands
add_path() {
  PATH=$1:${PATH}
  #### TODO: We removed Homebrew ar from PATH
  PATH=$(
    echo $PATH \\
      | tr ':' '\\n' \\
      | grep -v "\/opt\/homebrew\/opt\/make\/libexec\/gnubin" \\
      | grep -v "\/opt\/homebrew\/opt\/coreutils\/libexec\/gnubin" \\
      | grep -v "\/opt\/homebrew\/opt\/binutils\/bin" \\
      | tr '\\n' ':'
  )
  echo "**** PATH=$PATH" | tr ':' '\\n'
  which ar ## Should show \/usr\/bin\/ar
}

NOTUSED_add_path() {
EOF
)
rewrite_file

## Change: python_tools() {
## To:     NOTUSED_python_tools() {
search='^python_tools() {'
replace='NOTUSED_python_tools() {'
rewrite_file

## Change: # workaround for Cython issue
## To:     } \r python_tools() { \r ... \r # workaround for Cython issue
search='^  # workaround for Cython issue'
replace=$(
cat <<'EOF' | tr '\n' '\r'
}

python_tools() {
  #### TODO: We fixed the Python Environment
  python3 -m venv .venv
  source .venv\/bin\/activate

  # workaround for Cython issue
EOF
)
rewrite_file

## Change \r back to \n
cat $file \
  | tr '\r' '\n' \
  >$tmp_file
mv $tmp_file $file
chmod +x $file

(Called by run-job-macos.sh and run-ci-macos.sh)

lupyuen · 2024-11-10T00:11:59Z

Hmmm macOS fails to build arm-05, arm-08 ... arm-13 because of clang. I'll have to disable the clang builds for our macOS Build Farm:

Configuration/Tool: mps3-an547/clang,CONFIG_ARM_TOOLCHAIN_CLANG
clang++: error: configuration file 'armv8.1m.main_hard_fp.cfg' cannot be found
clang++: note: was searched for in the directory: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
clang++: error: unsupported argument '-mthumb' to option '-Wa,'
clang++: error: unsupported argument '-mimplicit-it=always' to option '-Wa,'
clang++: error: unsupported argument 'armv8.1-m.main+mve.fp+fp.dp' to option '-march='
clang++: error: configuration file 'armv8.1m.main_hard_fp.cfg' cannot be found

https://gist.github.com/lupyuen/f5eb30774e7359b0bfd07100a07eb252#file-gistfile1-txt-L1374

xiaoxiang781216 · 2024-11-10T02:27:45Z

you can download macOS arm clang from:
https://github.com/ARM-software/LLVM-embedded-toolchain-for-Arm/releases/tag/release-19.1.1

lupyuen · 2024-11-10T02:33:37Z

Thanks! Need to figure out how to add this to our macOS CI Job 🤔

lupyuen · 2024-11-10T12:18:07Z

Our macOS Build Farm is up! 🎉 We skipped these CI Jobs for macOS: run-ci-macos.sh

## Repeat forever for All CI Jobs, excluding:
## arm-05: "nrf5340-dk/rpmsghci_nimble_cpuapp: apps/wireless/bluetooth/nimble/mynewt-nimble/nimble/host/services/gatt/src/ble_svc_gatt.c:174:9: error: variable 'rc' set but not used"
## arm-07: "ucans32k146/se05x: mv: illegal option -- T"
## xtensa-02: "xtensa-esp32s2-elf-gcc: Command not found"
## arm64-01: "imx93-evk/bootloader: ld: library not found for -lcrt0.o"
## sim-01, 02, 03: "clang: error: invalid argument 'medium' to -mcmodel="
## other: "micropendous3/hello: make: avr-objcopy: Bad CPU type in executable"
for (( ; ; )); do
  for job in \
    arm-01 arm-02 arm-03 arm-04 \
    arm-06 arm-08 arm-09 arm-10 \
    arm-11 arm-12 arm-13 arm-14 \
    risc-v-01 risc-v-02 risc-v-03 \
    risc-v-04 risc-v-05 risc-v-06 \
    xtensa-01 \
    x86_64-01
  do
    ## Run the CI Job and find errors / warnings
    run_job $job
    clean_log
    find_messages

And we skipped all clang targets: run-job-macos.sh

## Exclude clang Targets from macOS Build, because they will fail due to unknown arch
## "/arm/lpc54xx,CONFIG_ARM_TOOLCHAIN_CLANG"
## https://github.com/apache/nuttx/pull/14691#issuecomment-2466518544
tmp_file=$tmp_dir/rewrite-testlist.dat
for file in nuttx-patched/tools/ci/testlist/*.dat; do
  grep -v "CLANG" \
    $file \
    >$tmp_file
  mv $tmp_file $file
done

## Run the CI Job in "nuttx-patched"
## ./cibuild.sh -i -c -A -R testlist/macos.dat
## ./cibuild.sh -i -c -A -R testlist/arm-01.dat
pushd nuttx-patched/tools/ci
(
  ./cibuild.sh -i -c -A -R testlist/$job.dat \
    || echo '***** BUILD FAILED'
)
popd

NuttX Build Logs for macOS Build Farm

simbit18 · 2024-11-11T11:51:48Z

hi @lupyuen I made some changes see branch
https://github.com/simbit18/nuttx/tree/simbit18-macos14

added the darwin_arm64.sh shell script only for macOS Apple Silicon (macos-14)

updated the toolchains for macOS on Apple Silicon

arm64
aarch64
xpack-riscv
xtensa-esp32

these will lighten your patch
https://github.com/lupyuen/nuttx-build-farm/blob/main/patch-ci-macos.sh
let me know if it is ok so I will create a new PR

lupyuen · 2024-11-11T12:11:22Z

Thanks @simbit18, yep just go ahead and create the PR :-)

simbit18 · 2024-11-25T14:03:30Z

hi @lupyuen I added toolchains xtensa-esp32s2 and xtensa-esp32s3 for macOS on Apple Intel and Silicon

PR #14934

simbit18 · 2024-12-11T15:22:31Z

hi @lupyuen I added added LLVM toolchain for Arm on Apple Intel release-17.0.1

on Apple Silicon release-19.1.1

PR #15146

tools/ci/platforms/darwin.sh: Port NuttX CI Job to macOS-14

2f0e72d

also runs old CI Job macOS-13

github-actions bot added Area: Tooling Area: CI Size: S The size of the change in this PR is small labels Nov 7, 2024

xiaoxiang781216 approved these changes Nov 7, 2024

View reviewed changes

lupyuen linked an issue Nov 7, 2024 that may be closed by this pull request

[FEATURE] Porting NuttX CI Job from Intel Mac to Apple Silicon Mac #14526

Closed

1 task

xiaoxiang781216 approved these changes Nov 7, 2024

View reviewed changes

xiaoxiang781216 merged commit 5080ab7 into apache:master Nov 7, 2024

lupyuen mentioned this pull request Nov 8, 2024

[FEATURE] Porting NuttX CI Job from Intel Mac to Apple Silicon Mac #14526

Closed

1 task

simbit18 deleted the simbit18-macos-arm64 branch November 11, 2024 09:14

simbit18 mentioned this pull request Nov 11, 2024

[darwin] CI macos-14 : updated toolchains for macOS Apple Silicon #14723

Merged

tools/ci/platforms/darwin.sh: Port NuttX CI Job to macOS-14 #14691

tools/ci/platforms/darwin.sh: Port NuttX CI Job to macOS-14 #14691

Uh oh!

Conversation

simbit18 commented Nov 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Impact

Testing

Uh oh!

simbit18 commented Nov 7, 2024

Uh oh!

lupyuen commented Nov 7, 2024

Uh oh!

simbit18 commented Nov 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lupyuen commented Nov 7, 2024

Uh oh!

simbit18 commented Nov 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lupyuen commented Nov 7, 2024

Uh oh!

lupyuen commented Nov 8, 2024

Uh oh!

simbit18 commented Nov 8, 2024

Uh oh!

lupyuen commented Nov 8, 2024

Uh oh!

simbit18 commented Nov 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

simbit18 commented Nov 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lupyuen commented Nov 8, 2024

Uh oh!

lupyuen commented Nov 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

simbit18 commented Nov 8, 2024

Uh oh!

lupyuen commented Nov 8, 2024

Uh oh!

lupyuen commented Nov 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lupyuen commented Nov 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lupyuen commented Nov 10, 2024

Uh oh!

xiaoxiang781216 commented Nov 10, 2024

Uh oh!

lupyuen commented Nov 10, 2024

Uh oh!

lupyuen commented Nov 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

simbit18 commented Nov 11, 2024

Uh oh!

lupyuen commented Nov 11, 2024

Uh oh!

simbit18 commented Nov 25, 2024

Uh oh!

simbit18 commented Dec 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

simbit18 commented Nov 7, 2024 •

edited

Loading

simbit18 commented Nov 7, 2024 •

edited

Loading

simbit18 commented Nov 7, 2024 •

edited

Loading

simbit18 commented Nov 8, 2024 •

edited

Loading

simbit18 commented Nov 8, 2024 •

edited

Loading

lupyuen commented Nov 8, 2024 •

edited

Loading

lupyuen commented Nov 8, 2024 •

edited

Loading

lupyuen commented Nov 8, 2024 •

edited

Loading

lupyuen commented Nov 10, 2024 •

edited

Loading

simbit18 commented Dec 11, 2024 •

edited

Loading