Building shared ESMF on macOS fails in linker stage

[climbfuji]

Describe the bug

See PR #953 for the code changes to trigger this error (in CI and on my macOS):

     9748    /Users/heinzell/work/spack-stack/spack-stack-geos-gcm/envs/unified-env/install/apple-clang/13.1.6/openmpi-4.1.6-atso6dm/bin/mpic++ -c  -std=c++11 -O -DNDEBUG -fPIC -x c++ -mmacosx-version-min=10.7 -stdlib=libc++ -pthread -I/Users/heinzell/work/sp
             ack-stack/spack-stack-geos-gcm/cache/build_stage/spack-stage-esmf-8.6.0-w7q65v7wblrc7xwb52al7l5smpg2r2yx/spack-src/src/Infrastructure/Trace/preload -I/Users/heinzell/work/spack-stack/spack-stack-geos-gcm/cache/build_stage/spack-stage-esmf-8.6.0-w
             7q65v7wblrc7xwb52al7l5smpg2r2yx/spack-src/src/Infrastructure/Trace/preload/../include -I/Users/heinzell/work/spack-stack/spack-stack-geos-gcm/cache/build_stage/spack-stage-esmf-8.6.0-w7q65v7wblrc7xwb52al7l5smpg2r2yx/spack-src/build_config/Darwin.
             gfortranclang.default -I/Users/heinzell/work/spack-stack/spack-stack-geos-gcm/cache/build_stage/spack-stage-esmf-8.6.0-w7q65v7wblrc7xwb52al7l5smpg2r2yx/spack-src/src/Infrastructure -I/Users/heinzell/work/spack-stack/spack-stack-geos-gcm/cache/bui
             ld_stage/spack-stage-esmf-8.6.0-w7q65v7wblrc7xwb52al7l5smpg2r2yx/spack-src/src/Superstructure -I/Users/heinzell/work/spack-stack/spack-stack-geos-gcm/cache/build_stage/spack-stage-esmf-8.6.0-w7q65v7wblrc7xwb52al7l5smpg2r2yx/spack-src/src/Infrastr
             ucture/stubs/pthread  -I/Users/heinzell/work/spack-stack/spack-stack-geos-gcm/cache/build_stage/spack-stage-esmf-8.6.0-w7q65v7wblrc7xwb52al7l5smpg2r2yx/spack-src/src/include  -I/Users/heinzell/work/spack-stack/spack-stack-geos-gcm/envs/unified-en
             v/install/apple-clang/13.1.6/netcdf-c-4.9.2-vufz7vj/include -I/Users/heinzell/work/spack-stack/spack-stack-geos-gcm/envs/unified-env/install/apple-clang/13.1.6/parallelio-2.5.10-nkc6ohk/include -I/Users/heinzell/work/spack-stack/spack-stack-geos-
             gcm/cache/build_stage/spack-stage-esmf-8.6.0-w7q65v7wblrc7xwb52al7l5smpg2r2yx/spack-src/src/prologue/yaml-cpp/include -DESMF_NO_INTEGER_1_BYTE -DESMF_NO_INTEGER_2_BYTE -DESMF_LOWERCASE_SINGLEUNDERSCORE -DESMF_MOAB=1 -DESMF_LAPACK=1 -DESMF_LAPACK_
             INTERNAL=1 -DESMF_NO_ACC_SOFTWARE_STACK=1 -DESMF_NETCDF=1 -DESMF_YAMLCPP=1 -DESMF_YAML=1 -DESMF_PIO=1 -DESMF_NO_OPENMP -DESMF_NO_OPENACC -DESMF_BOPT_O -DESMF_TESTCOMPTUNNEL -DS64=1 -DESMF_OS_Darwin=1 -DESMF_COMM=openmpi -DESMF_DIR=/Users/heinzell
             /work/spack-stack/spack-stack-geos-gcm/cache/build_stage/spack-stage-esmf-8.6.0-w7q65v7wblrc7xwb52al7l5smpg2r2yx/spack-src -D__SDIR__='"src/Infrastructure/Trace/preload"' -DESMF_CXXSTD=11 -DESMF_NO_SIGUSR2 preload.C -o preload.o
     9749    /Users/heinzell/work/spack-stack/spack-stack-geos-gcm/envs/unified-env/install/apple-clang/13.1.6/openmpi-4.1.6-atso6dm/bin/mpic++ -c  -std=c++11 -O -DNDEBUG -fPIC -x c++ -mmacosx-version-min=10.7 -stdlib=libc++ -pthread -I/Users/heinzell/work/sp
             ack-stack/spack-stack-geos-gcm/cache/build_stage/spack-stage-esmf-8.6.0-w7q65v7wblrc7xwb52al7l5smpg2r2yx/spack-src/src/Infrastructure/Trace/preload -I/Users/heinzell/work/spack-stack/spack-stack-geos-gcm/cache/build_stage/spack-stage-esmf-8.6.0-w
             7q65v7wblrc7xwb52al7l5smpg2r2yx/spack-src/src/Infrastructure/Trace/preload/../include -I/Users/heinzell/work/spack-stack/spack-stack-geos-gcm/cache/build_stage/spack-stage-esmf-8.6.0-w7q65v7wblrc7xwb52al7l5smpg2r2yx/spack-src/build_config/Darwin.
             gfortranclang.default -I/Users/heinzell/work/spack-stack/spack-stack-geos-gcm/cache/build_stage/spack-stage-esmf-8.6.0-w7q65v7wblrc7xwb52al7l5smpg2r2yx/spack-src/src/Infrastructure -I/Users/heinzell/work/spack-stack/spack-stack-geos-gcm/cache/bui
             ld_stage/spack-stage-esmf-8.6.0-w7q65v7wblrc7xwb52al7l5smpg2r2yx/spack-src/src/Superstructure -I/Users/heinzell/work/spack-stack/spack-stack-geos-gcm/cache/build_stage/spack-stage-esmf-8.6.0-w7q65v7wblrc7xwb52al7l5smpg2r2yx/spack-src/src/Infrastr
             ucture/stubs/pthread  -I/Users/heinzell/work/spack-stack/spack-stack-geos-gcm/cache/build_stage/spack-stage-esmf-8.6.0-w7q65v7wblrc7xwb52al7l5smpg2r2yx/spack-src/src/include  -I/Users/heinzell/work/spack-stack/spack-stack-geos-gcm/envs/unified-en
             v/install/apple-clang/13.1.6/netcdf-c-4.9.2-vufz7vj/include -I/Users/heinzell/work/spack-stack/spack-stack-geos-gcm/envs/unified-env/install/apple-clang/13.1.6/parallelio-2.5.10-nkc6ohk/include -I/Users/heinzell/work/spack-stack/spack-stack-geos-
             gcm/cache/build_stage/spack-stage-esmf-8.6.0-w7q65v7wblrc7xwb52al7l5smpg2r2yx/spack-src/src/prologue/yaml-cpp/include -DESMF_NO_INTEGER_1_BYTE -DESMF_NO_INTEGER_2_BYTE -DESMF_LOWERCASE_SINGLEUNDERSCORE -DESMF_MOAB=1 -DESMF_LAPACK=1 -DESMF_LAPACK_
             INTERNAL=1 -DESMF_NO_ACC_SOFTWARE_STACK=1 -DESMF_NETCDF=1 -DESMF_YAMLCPP=1 -DESMF_YAML=1 -DESMF_PIO=1 -DESMF_NO_OPENMP -DESMF_NO_OPENACC -DESMF_BOPT_O -DESMF_TESTCOMPTUNNEL -DS64=1 -DESMF_OS_Darwin=1 -DESMF_COMM=openmpi -DESMF_DIR=/Users/heinzell
             /work/spack-stack/spack-stack-geos-gcm/cache/build_stage/spack-stage-esmf-8.6.0-w7q65v7wblrc7xwb52al7l5smpg2r2yx/spack-src -D__SDIR__='"src/Infrastructure/Trace/preload"' -DESMF_CXXSTD=11 -DESMF_NO_SIGUSR2 preload_mpi.C -o preload_mpi.o
     9750    /Users/heinzell/work/spack-stack/spack-stack-geos-gcm/envs/unified-env/install/apple-clang/13.1.6/openmpi-4.1.6-atso6dm/bin/mpif90  -dynamiclib -o /Users/heinzell/work/spack-stack/spack-stack-geos-gcm/cache/build_stage/spack-stage-esmf-8.6.0-w7q6
             5v7wblrc7xwb52al7l5smpg2r2yx/spack-src/lib/libO/Darwin.gfortranclang.64.openmpi.default/libesmftrace_preload.dylib preload.o preload_mpi.o wrappers.o wrappers_mpi.o   -pthread -L/Users/heinzell/work/spack-stack/spack-stack-geos-gcm/cache/build_st
             age/spack-stage-esmf-8.6.0-w7q65v7wblrc7xwb52al7l5smpg2r2yx/spack-src/lib/libO/Darwin.gfortranclang.64.openmpi.default -L/Users/heinzell/work/spack-stack/spack-stack-geos-gcm/envs/unified-env/install/apple-clang/13.1.6/netcdf-c-4.9.2-vufz7vj/lib
             -L/Users/heinzell/work/spack-stack/spack-stack-geos-gcm/envs/unified-env/install/apple-clang/13.1.6/netcdf-fortran-4.6.1-zmvqkmu/lib -L/Users/heinzell/work/spack-stack/spack-stack-geos-gcm/envs/unified-env/install/apple-clang/13.1.6/parallelio-2.
             5.10-nkc6ohk/lib -L/opt/homebrew/Cellar/gcc@12/12.3.0/bin/../lib/gcc/12/gcc/aarch64-apple-darwin21/12/../../../ -Wl,-rpath,/Users/heinzell/work/spack-stack/spack-stack-geos-gcm/cache/build_stage/spack-stage-esmf-8.6.0-w7q65v7wblrc7xwb52al7l5smpg2
             r2yx/spack-src/lib/libO/Darwin.gfortranclang.64.openmpi.default -Wl,-rpath,/Users/heinzell/work/spack-stack/spack-stack-geos-gcm/envs/unified-env/install/apple-clang/13.1.6/netcdf-c-4.9.2-vufz7vj/lib -Wl,-rpath,/Users/heinzell/work/spack-stack/sp
             ack-stack-geos-gcm/envs/unified-env/install/apple-clang/13.1.6/netcdf-fortran-4.6.1-zmvqkmu/lib  -Wl,-rpath,/Users/heinzell/work/spack-stack/spack-stack-geos-gcm/envs/unified-env/install/apple-clang/13.1.6/parallelio-2.5.10-nkc6ohk/lib -Wl,-rpath
             ,/opt/homebrew/Cellar/gcc@12/12.3.0/bin/../lib/gcc/12/gcc/aarch64-apple-darwin21/12/../../../  -lgfortran -lnetcdf -lnetcdff -lnetcdf -lnetcdf -lm -lpioc -lesmf
  >> 9751    Undefined symbols for architecture arm64:
     9752      "__ZNKSt3__121__basic_string_commonILb1EE20__throw_length_errorEv", referenced from:
     9753          __ZNKSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE20__throw_length_errorEv in wrappers_mpi.o
     9754      "__ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE6__initEPKcm", referenced from:
     9755          __ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEC2IDnEEPKc in wrappers_mpi.o
     9756      "__ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEED1Ev", referenced from:
     9757          ___wrap_MPI_Allgather in wrappers_mpi.o

     ...

     9773          ___wrap_MPI_Alltoall in wrappers_mpi.o
     9774          ___wrap_MPI_Alltoallv in wrappers_mpi.o
     9775          ___wrap_MPI_Alltoallw in wrappers_mpi.o
     9776          ___wrap_MPI_Barrier in wrappers_mpi.o
     9777          ...
     9778    ld: symbol(s) not found for architecture arm64
  >> 9779    collect2: error: ld returned 1 exit status
  >> 9780    make[2]: *** [tracelib_preload] Error 1
  >> 9781    make[1]: *** [build_tracelibs] Error 2
  >> 9782    make: *** [lib] Error 2

To Reproduce
See #953

Expected behavior
It builds!

System:
macOS (CI, @climbfuji's macOS)

Additional context
openmpi is built by/in spack, not as an external package

[climbfuji]

@AlexanderRichert-NOAA @ulmononian @mathomp4 FYI

[mathomp4]

Interesting. As @oehmke said in esmf-org/esmf#209, try setting ESMF_TRACE_LIB_BUILD=OFF in the build. We've only had to that in Baselibs when using NAG + ESMF 8.6.0 + macOS, but maybe it is more general?

[climbfuji]

That worked! I'll submit a PR to spack with the change for macOS when building shared ESMF. Sorry for the late reply, all-day meeting today ...

[climbfuji]

See:

spack/spack#42133 # issue in spack develop
spack/spack#42134 # pull request in spack develop (cherry-picked from PR to our spack fork)
JCSDA/spack#393 # pull request to our spack fork