add support in init script for using 'archdetect' alternative to archspec (only if $EESSI_USE_ARCHDETECT is set to '1')

[hmeiland]

alternative for archspec

• removes requirement for python
• be able to maintain which flags are important for architecture determination
• provides detection for all EESSI supported cpu archs
• breaks graviton2 detection in favor of neoverse-n1
• breaks graviton3 detection in favor of neoverse-v1

not enabled by default: require EESSI_USE_ARCHDETECT to be used (for now)

[lexming]

@hmeiland I've gone through this PR one more time and it looks fine on my side. All I can think of as improvement at this point is to enhance the logging and comments in the script.

I made the following changes in PR hmeiland#3:

• add a log function to simplify logging, we can now easily use different logging levels
• add -h (print help) and -d (debug level) arguments to the script
• add more comments and fix formatting
• fix get_cpuinfo() to be fully case insensitive
• generalize check_flags() to check_allinfirst(), the new function works in the same way
• add version tag and -v option to print it

You will see that the diff looks like I rewrote most of the script, but that's mainly because I standardize the indentation to 4 spaces.

[lexming]

LGTM

[bartoldeman]

lgtm

[lexming]

LGTM

[boegel]

I've tested this in various systems, works as expected:

• default of using archspec is unchanged, for now;
• eessi_archdetect.sh bash script is used instead of archspec only if $EESSI_USE_ARCHDETECT is set to 1;
• on an Azure AMD Milan(-X) system (HB120rs_v3), archdetect correctly returns x86_64/amd/zen3, while the archspec version we have in the EESSI pilot version 2021.12 returns x86-64/amd/zen2 (see also archspec 0.1.3 identies AMD Milan in Azure as zen2 due to pku feature that is only listed for zen3 archspec/archspec-json#38):
  • using archspec (default):

    Singularity> source /cvmfs/pilot.eessi-hpc.org/versions/2021.12/init/bash
    Found EESSI pilot repo @ /cvmfs/pilot.eessi-hpc.org/versions/2021.12!
    archspec says x86_64/amd/zen2
    Using x86_64/amd/zen2 as software subdirectory.
    Using /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/modules/all as the directory to be added to MODULEPATH.
    Found Lmod configuration file at /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/.lmod/lmodrc.lua
    Initializing Lmod...
    Prepending /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen2/modules/all to $MODULEPATH...
    Environment set up to use EESSI pilot software stack, have fun!
    [EESSI pilot 2021.12] $
    

  • using archdetect (opt-in via $EESSI_USE_ARCHDETECT):

    Singularity> EESSI_USE_ARCHDETECT=1 source /cvmfs/pilot.eessi-hpc.org/versions/2021.12/init/bash
    Found EESSI pilot repo @ /cvmfs/pilot.eessi-hpc.org/versions/2021.12!
    2022-11-08 17:17:59 [INFO] cpupath: best match for host CPU: x86_64/amd/zen3
    archdetect says x86_64/amd/zen3
    Using x86_64/amd/zen3 as software subdirectory.
    Using /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen3/modules/all as the directory to be added to MODULEPATH.
    Found Lmod configuration file at /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen3/.lmod/lmodrc.lua
    Initializing Lmod...
    Prepending /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen3/modules/all to $MODULEPATH...
    Environment set up to use EESSI pilot software stack, have fun!
    [EESSI pilot 2021.12] $
    

• On Graviton 3, archdetect returns aarch64/arm/neoverse-v1 (similar to what the next version of archspec will do, cfr. Rename graviton products with ARM uarch name archspec/archspec-json#57), which requires that we symlink that subdir to aarch64/graviton3:

  Singularity> EESSI_USE_ARCHDETECT=1 source /cvmfs/pilot.eessi-hpc.org/versions/2021.12/init/bash
  Found EESSI pilot repo @ /cvmfs/pilot.eessi-hpc.org/versions/2021.12!
  2022-11-08 16:58:55 [INFO] cpupath: best match for host CPU: aarch64/arm/neoverse-v1
  archdetect says aarch64/arm/neoverse-v1
  Using aarch64/arm/neoverse-v1 as software subdirectory.
  ERROR: EESSI software layer at /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/aarch64/arm/neoverse-v1 not found!
  

  Singularity> cd /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/aarch64
  Singularity> ls
  generic  graviton2  graviton3
  Singularity> mkdir arm
  Singularity> cd arm
  Singularity> ln -s ../graviton3 neoverse-v1
  

  Singularity> EESSI_USE_ARCHDETECT=1 source /cvmfs/pilot.eessi-hpc.org/versions/2021.12/init/bash
  Found EESSI pilot repo @ /cvmfs/pilot.eessi-hpc.org/versions/2021.12!
  2022-11-08 16:59:39 [INFO] cpupath: best match for host CPU: aarch64/arm/neoverse-v1
  archdetect says aarch64/arm/neoverse-v1
  Using aarch64/arm/neoverse-v1 as software subdirectory.
  Using /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/aarch64/arm/neoverse-v1/modules/all as the directory to be added to MODULEPATH.
  Found Lmod configuration file at /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/aarch64/arm/neoverse-v1/.lmod/lmodrc.lua
  Initializing Lmod...
  Prepending /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/aarch64/arm/neoverse-v1/modules/all to $MODULEPATH...
  Environment set up to use EESSI pilot software stack, have fun!
  [EESSI pilot 2021.12] $
  

  • With the archspec version in the 2021.12 EESSI compat layer, we're falling back to aarch64/graviton2:

    Found EESSI pilot repo @ /cvmfs/pilot.eessi-hpc.org/versions/2021.12!
    WARNING: Ignoring unknown target "graviton3"
    archspec says aarch64/graviton2
    Using aarch64/graviton2 as software subdirectory.
    Using /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/aarch64/graviton2/modules/all as the directory to be added to MODULEPATH.
    Found Lmod configuration file at /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/aarch64/graviton2/.lmod/lmodrc.lua
    Initializing Lmod...
    Prepending /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/aarch64/graviton2/modules/all to $MODULEPATH...
    Environment set up to use EESSI pilot software stack, have fun!
    [EESSI pilot 2021.12] $
    

I've also tested this on:

• on Intel Skylake, archdetect returns x86_64/intel/skylake_avx512;
• on AMD Rome, archdetect returns x86_64/amd/zen2;
• on Graviton 2, archdetect returns aarch64/arm/neoverse-n1 (which should symlink to aarch64/graviton2 in EESSI pilot 2021.12);
• on POWER9, archdetect returns ppc64le/power9le;