November 2023 main Spack merge + updates

[AlexanderRichert-NOAA]

Summary

I'll keep the notes for this update, which includes JCSDA/spack#371, in this PR.

This PR pulls in Spack release 0.21.0. Additionally, I've reconciled NCEPLIBS recipes with those in the library repos themselves (very few changes), as well as added in changes to support NCO's stack. Ran style checks/fixes on all packages. This PR also includes a proposed solution for the MySQL/Boost problem (#705).

Misc. notes:

• wgrib2 now creates limitation for gmake version; this may be fixable with patching (modeled on wgrib2@3.1.1+ which is appropriately corrected)
• MySQL now has an option to use internal boost libraries (this approach could probably be improved by creating resources)
• I added a fix for a random gmake build issue
• I tweaked a few default version settings, including removing a few; obviously these can be re-added as needed but so far so good based on my testing.
• I tested out the new spack deconcretize command. Pretty nifty :)
• Still need external flex to avoid duplicates.
• Adding rust_bootstrap variants to py-cryptography and py-setuptools-rust
• Looks like py-numpy and py-setuptools versions should stay pinned

Waiting on:

• Add cxx17_flag to intel.py spack/spack#41207

Testing

• Tested concretizing and some building on PC.
• Tested building NCO's stack (needed some changes, incl. supporting deprecated packages).
• Tested building UFS WM & GW deps on Hera; no duplicates

Tests to do:

• Do another round of diffing Spack core (non-recipe) changes to inventory our changes (@AlexanderRichert-NOAA; there appears to be one unintentional difference in lib/spack/spack/build_environment.py that can get fixed whenever)
• Hercules (@climbfuji)
• Orion (@ulmononian)
• Discover (@climbfuji)
• Casper (@climbfuji)
• Derecho (@climbfuji)
• Acorn (@AlexanderRichert-NOAA)
• Gaea C5 (@RatkoVasic-NOAA)
• Hera (@RatkoVasic-NOAA)
• Jet (@RatkoVasic-NOAA)
• Narwhal (@climbfuji)
• Nautilus (@climbfuji)
• S4 (@climbfuji)
• AWS AMI RH JCSDA (@climbfuji)
• AWS ParallelCluster JCSDA (@climbfuji)
• RDHPCS Cloud (ParallelWorks) EPIC
• RDHPCS Cloud (ParallelWorks) JCSDA (deferred to https://github.com/JCSDA-internal/jedi-tools/issues/325)
• JCSDA CI container builds (@climbfuji)
  • docker-ubuntu-clang-mpich
  • docker-ubuntu-gnu-openmpi
  • docker-ubuntu-intel-impi
• EMC RHEL (@AlexanderRichert-NOAA)

New site configs:

• @climbfuji's macOS Monterey with apple-clang@13.1.6 and openmpi@4.1.6

Applications affected

All.

Systems affected

All.

Dependencies

• Nov 2023 spack merge spack#371

Issue(s) addressed

Addresses #705 (w.r.t. MySQL)

Checklist

• This PR addresses one issue/problem/enhancement, or has a very good reason for not doing so.
• These changes have been tested on the affected systems and applications.
• All dependency PRs/issues have been resolved and this PR can be merged.

[AlexanderRichert-NOAA]

@climbfuji have you run into Undefined symbols for architecture arm64: / ld: symbol(s) not found for architecture arm64 (see https://github.com/JCSDA/spack-stack/actions/runs/6951315212/job/18913085682?pr=884)? It's failing on gsl, but it's the same package/version/spec as the current one.

[climbfuji]

Details

That's new to me. If I had to guess I'd say that the binary cache needs to be cleared. Let me do that and rerun the failing tests.

[climbfuji]

Same error when building from scratch.

[climbfuji]

@AlexanderRichert-NOAA I am trying to fix the macOS and Ubuntu CI errors. Given the problems we are seeing I would like to have a good testing coverage across the various HPCs that we support, ideally we build this branch on all of the tier-1 platforms before merging?

[AlexanderRichert-NOAA]

Yep agreed. I'll fill out the test list in a bit.

[climbfuji]

Error on Narwhal on the home stretch, for mapl with GNU (worked with Intel):

work1/heinzell/spack-stack-nov2023_spackmerge/cache/build_stage/spack-stage-mapl-2.40.3-bpfups3oqez3cqbg5wqu4zn6ofoxalty/spack-src'

1 error found in build log:
     22    -- Cray Programming Environment 2.7.19 C
     23    -- Detecting C compiler ABI info
     24    -- Detecting C compiler ABI info - done
     25    -- Check for working C compiler: /p/work1/heinzell/spack-stack-nov2023_spackmerge/spack/lib/spack/env/gcc/gcc - skipped
     26    -- Detecting C compile features
     27    -- Detecting C compile features - done
  >> 28    CMake Error at ESMA_cmake/esma.cmake:5 (esma_check_install_prefix):
     29      Unknown CMake command "esma_check_install_prefix".
     30    Call Stack (most recent call first):
     31      CMakeLists.txt:29 (include)
     32
     33
     34    -- Configuring incomplete, errors occurred!

See build log for details:
  /p/work1/heinzell/spack-stack-nov2023_spackmerge/cache/build_stage/spack-stage-mapl-2.40.3-bpfups3oqez3cqbg5wqu4zn6ofoxalty/spack-build-out.txt

I can look into it later but if @mathomp4 has an idea right away (hopefully!) ...

[AlexanderRichert-NOAA]

Noted. That looks suspiciously like an error I ran into ages ago involving the esma_cmake resource and the caching thereof.

[mathomp4]

That macro is fairly benign:

https://github.com/GEOS-ESM/ESMA_cmake/blob/main/esma_support/esma_check_install_prefix.cmake

it just checks to make sure you aren't installing to /usr/local, you don't have a comma in the directory path¹, and where you are installing, you have write permissions.

Footnotes

1. That one was a fun bug to figure out. ↩

[climbfuji]

Noted. That looks suspiciously like an error I ran into ages ago involving the esma_cmake resource and the caching thereof.

Hmm, but given that it installed fine a few hours beforehand with Intel that seems rather unlikely. And there's no difference between jcsda_emc_spack_stack mapl and this PR, and it previously built just fine. Grrrrrr...

[AlexanderRichert-NOAA]

Try relocating the cached version so it redownloads? But don't delete it, in case I'm wrong :)

[climbfuji]

Try relocating the cached version so it redownloads? But don't delete it, in case I'm wrong :)

Solution, a lot easier: run spack clean -a once, try, it fails. Wait a few hours, run spack clean -a again, try, it installs just fine.

[climbfuji]

I am happy with all of this, and very thankful to @AlexanderRichert-NOAA for shepherding these PRs through. I will approve once the spack PR is merged and the submodule updated as usual.

[climbfuji]

spack PR was merged, new hash is JCSDA/spack@8683cc4

[climbfuji]

I went ahead and updated the spack submodule / reverted .gitmodules. Only one question left from my side, other than that it's good to go!

[natalie-perlin]

@AlexanderRichert-NOAA @climbfuji -
let me update or include README.md files for noaa- platforms

[climbfuji]

@AlexanderRichert-NOAA @climbfuji - let me update or include README.md files for noaa- platforms

@natalie-perlin How quickly can you do that? If you need a bit longer, we can make it a small follow-up PR (since no testing needed for adding/updating README.md files).

[climbfuji]

Merging this now to avoid container build problems (since the spack PR was merged this morning).