In a chained environment, setup-meta-modules doesn't find python from the upstream environment

[srherbener]

Describe the bug

When building a chained environment and the upstream environment satisfies the python dependency, the spack stack setup-meta-modules command doesn't find the python provider for building the stack-python meta module.

To Reproduce

How I ran into this was trying to use chained environments to build a debug version of the hdf5 library so I can run the debugger on the hdf5 library code. I did the following using spack-stack-1.5.1 as the upstream environment. I did this on my arm64 based Mac, using apple-clang@14.0.3, gfortran@12.2.0 and openmpi@4.1.5.

spack stack create env --site macos.default --template empty --name spack-stack-1.5.1-debug-hdf5  --upstream $PWD/envs/spack-stack-1.5.1/install

cd envs/spack-stack-1.5.1-debug-hdf5
spack env activate .
spack add ewok-env%apple-clang@14.0.3 ^hdf5
spack add jedi-fv3-env%apple-clang@14.0.3 ^hdf5

spack concretize --force 2>&1 | tee log.concretize
../../util/show_duplicate_packages.py -d log.concretize
spack install -v 2>&1 | tee log.install

spack module lmod refresh --upstream-modules
spack stack setup-meta-modules

And the setup-meta-modules returned this:

Configuring basic directory information ...
  ... script directory: /Users/stephenh/spack-stack/spack/lib/jcsda-emc/spack-stack/stack
  ... base directory: /Users/stephenh/spack-stack/spack/lib/jcsda-emc/spack-stack
  ... spack directory: /Users/stephenh/spack-stack/spack
Configuring active spack environment ...
  ... environment directory: /Users/stephenh/spack-stack/envs/spack-stack-1.5.1-debug-hdf5
Parsing spack environment main config ...
  ... install directory: /Users/stephenh/spack-stack/envs/spack-stack-1.5.1-debug-hdf5/install
Parsing spack environment modules config ...
  ... configured to use lmod modules
  ... module directory: /Users/stephenh/spack-stack/envs/spack-stack-1.5.1-debug-hdf5/install/modulefiles
Parsing spack environment package config ...
  ... list of possible compilers: '['apple-clang', 'clang', 'gcc', 'intel', 'pgi', 'xl', 'nag', 'fj', 'aocc']'
  ... list of possible mpi providers: '['openmpi', 'mpich']'
['module-index.yaml', 'Core', 'apple-clang', 'openmpi']
 ... stack compilers: '{'apple-clang': ['14.0.3']}'
 ... stack mpi providers: '{'openmpi': {'4.1.5': {'apple-clang': ['14.0.3']}}}'
  ... core compilers: ['gcc@4.6']
Preparing meta module directory ...
  ... meta module directory : /Users/stephenh/spack-stack/envs/spack-stack-1.5.1-debug-hdf5/install/modulefiles/Core
Creating compiler modules ...
  ... configuring stack compiler apple-clang@14.0.3
  ... ... CC  : /usr/bin/clang
  ... ... CXX : /usr/bin/clang++
  ... ... F77 : /opt/homebrew/bin/gfortran
  ... ... FC' : /opt/homebrew/bin/gfortran
  ... ... COMPFLAGS: 
  ... ... MODULELOADS: 
  ... ... MODULEPREREQS: 
  ... ... MODULEPATH  : /Users/stephenh/spack-stack/envs/spack-stack-1.5.1-debug-hdf5/install/modulefiles/apple-clang/14.0.3
  ... writing /Users/stephenh/spack-stack/envs/spack-stack-1.5.1-debug-hdf5/install/modulefiles/Core/stack-apple-clang/14.0.3.lua
  ... configuring stack mpi library openmpi@4.1.5 for compiler apple-clang@14.0.3
  ... ... MODULELOADS: load("openmpi/4.1.5")
  ... ... MODULEPREREQS: prereq("openmpi/4.1.5")
  ... ... MODULEPATH  : /Users/stephenh/spack-stack/envs/spack-stack-1.5.1-debug-hdf5/install/modulefiles/openmpi/4.1.5/apple-clang/14.0.3
  ... writing /Users/stephenh/spack-stack/envs/spack-stack-1.5.1-debug-hdf5/install/modulefiles/apple-clang/14.0.3/stack-openmpi/4.1.5.lua
  ... using spack-built python version 3.10.8
 ... stack python providers: '{}'
No matching Python version found found, proceed without creating Python modules? (yes/no): yes
Metamodule generation completed successfully in /Users/stephenh/spack-stack/envs/spack-stack-1.5.1-debug-hdf5/install/modulefiles/Core

My spack.yaml file has these entries:

...
  specs:
  - ewok-env%apple-clang@14.0.3 ^hdf5
  - jedi-fv3-env%apple-clang@14.0.3 ^hdf5
  upstreams:
    install:
      install_tree: /Users/stephenh/spack-stack/envs/spack-stack-1.5.1/install
...

And I enabled the debug build of hdf5 by modifying common/packages.py:

...
    hdf5:
      version: ['1.14.0']
      variants: build_type=Debug +hl +fortran +mpi ~threadsafe +szip
...

Everything appeared to work up to the setup-meta-modules command. I think the only thing that is missing is the stack-python module lmod script.

Expected behavior
The setup-meta-modules command finds the python provided by the upstream environment and construct a stack-python lua module script accordingly.

System:
arm64 MacOS

Additional context
Add any other context about the problem here.

[srherbener]

I'm new to the chained environment process, so I realize that I might have done something wrong that caused this fault to occur.

[climbfuji]

@srherbener I am having trouble even getting to the problem you are having. My chained env builds hdf w/ debug fine, but then netcdf-c and anything else depending on hdf5 doesn't find the library and fails building. Working on it ...

[climbfuji]

@srherbener This is what I am seeing: in the original unified-env, netcdf-c gets built as follows:

==> [2023-11-06-10:37:20.268227] './configure' '--prefix=/Users/heinzell/work/spack-stack/spack-stack-1.5.1-derecho/envs/unified-env/install/apple-clang/13.1.6/netcdf-c-4.9.2-3kb6jx2' '--enable-v2' '--enable-utilities' '--enable-static' '--enable-largefile' '--enable-netcdf-4' '--enable-nczarr' '--enable-plugins' '--with-plugin-dir=/Users/heinzell/work/spack-stack/spack-stack-1.5.1-derecho/envs/unified-env/install/apple-clang/13.1.6/netcdf-c-4.9.2-3kb6jx2/plugins' '--enable-parallel4' '--disable-pnetcdf' '--disable-hdf4' '--enable-shared' '--enable-dap' '--enable-libxml2' '--disable-byterange' '--disable-jna' '--disable-fsync' 'CC=/Users/heinzell/work/spack-stack/spack-stack-1.5.1-derecho/envs/unified-env/install/apple-clang/13.1.6/openmpi-4.1.5-zhlaupe/bin/mpicc' 'ac_cv_search_zip_open=no' 'ac_cv_prog_NC_M4=false' 'LDFLAGS=-L/Users/heinzell/work/spack-stack/spack-stack-1.5.1-derecho/envs/unified-env/install/apple-clang/13.1.6/hdf5-1.14.0-he7uscp/lib -L/Users/heinzell/work/spack-stack/spack-stack-1.5.1-derecho/envs/unified-env/install/apple-clang/13.1.6/zlib-1.2.13-3rwvvpy/lib -L/Users/heinzell/work/spack-stack/spack-stack-1.5.1-derecho/envs/unified-env/install/apple-clang/13.1.6/libaec-1.0.6-grtvkjm/lib -L/Users/heinzell/work/spack-stack/spack-stack-1.5.1-derecho/envs/unified-env/install/apple-clang/13.1.6/bzip2-1.0.8-vr4opg5/lib -L/Users/heinzell/work/spack-stack/spack-stack-1.5.1-derecho/envs/unified-env/install/apple-clang/13.1.6/zstd-1.5.2-brsyot2/lib -L/Users/heinzell/work/spack-stack/spack-stack-1.5.1-derecho/envs/unified-env/install/apple-clang/13.1.6/c-blosc-1.21.4-bkylaxc/lib -L/Users/heinzell/work/spack-stack/spack-stack-1.5.1-derecho/envs/unified-env/install/apple-clang/13.1.6/curl-8.1.2-j7aqucy/lib' 'LIBS='

In the chained env:

'./configure' '--prefix=/Users/heinzell/work/spack-stack/spack-stack-1.5.1-derecho/envs/chained-env/install/apple-clang/13.1.6/netcdf-c-4.9.2-ml4apgw' '--enable-v2' '--enable-utilities' '--enable-static' '--enable-largefile' '--enable-netcdf-4' '--enable-nczarr' '--enable-plugins' '--with-plugin-dir=/Users/heinzell/work/spack-stack/spack-stack-1.5.1-derecho/envs/chained-env/install/apple-clang/13.1.6/netcdf-c-4.9.2-ml4apgw/plugins' '--enable-parallel4' '--disable-pnetcdf' '--disable-hdf4' '--enable-shared' '--enable-dap' '--enable-libxml2' '--disable-byterange' '--disable-jna' '--disable-fsync' 'CC=/Users/heinzell/work/spack-stack/spack-stack-1.5.1-derecho/envs/unified-env/install/apple-clang/13.1.6/openmpi-4.1.5-zhlaupe/bin/mpicc' 'ac_cv_search_zip_open=no' 'ac_cv_prog_NC_M4=false' 'LDFLAGS=-L/Users/heinzell/work/spack-stack/spack-stack-1.5.1-derecho/envs/unified-env/install/apple-clang/13.1.6/zlib-1.2.13-3rwvvpy/lib -L/Users/heinzell/work/spack-stack/spack-stack-1.5.1-derecho/envs/unified-env/install/apple-clang/13.1.6/libaec-1.0.6-grtvkjm/lib -L/Users/heinzell/work/spack-stack/spack-stack-1.5.1-derecho/envs/unified-env/install/apple-clang/13.1.6/bzip2-1.0.8-vr4opg5/lib -L/Users/heinzell/work/spack-stack/spack-stack-1.5.1-derecho/envs/unified-env/install/apple-clang/13.1.6/zstd-1.5.2-brsyot2/lib -L/Users/heinzell/work/spack-stack/spack-stack-1.5.1-derecho/envs/unified-env/install/apple-clang/13.1.6/c-blosc-1.21.4-bkylaxc/lib -L/Users/heinzell/work/spack-stack/spack-stack-1.5.1-derecho/envs/unified-env/install/apple-clang/13.1.6/curl-8.1.2-j7aqucy/lib' 'LIBS='

It's a little hard to see, but the first entry in LD_LFLAGS for unified-env is for hdf5 and then the rest. For chained env, the first entry (hdf5) is missing, all others from the unified-env are there. That looks like a bug.

[srherbener]

Sorry @climbfuji, I didn't add this detail but the hdf5 library, when build_type=Debug variant is used, will name all of the installed library files with a "_debug" suffix. libhdf5_debug.dylib instead of libhdf5.dylib. I worked around this by manually adding symlinks in the install directory to make the names without the "_debug" suffix available. Then the spack-stack install completed successfully.

I'm not sure how to coerce the hdf5 cmake config to not add the "_debug" suffixes. There wasn't an obvious control. It would be nice if that was available.

[srherbener]

By the way, I successfully built a full environment this morning using the build_type=Debug for the hdf5 libraries. And by using the symlink workaround. So the urgency of this is not as high.

[climbfuji]

@srherbener I submitted a PR with a bug fix for the hdf5 _debug suffix in the library names to spack develop: spack/spack#40965 - once that gets merged (in this or an amended form), I'll bring it back to our spack fork.

[climbfuji]

@srherbener Also, I think the proper way to fix this is to address flaws in the setup-meta-modules extension (which I wrote and am to blame for). There is some very complicated logic in the code that tries to find installed packages and match them to what is in the package config. I think we can replace most of this by doing the equivalent of spack find in the code and then use that information. That will take a bit of time, but I understand it's not that urgent.

[srherbener]

@climbfuji thank you for addressing this so quickly and submitting the PR to the authoritative spack repo. I agree that this isn't urgent since I have a workaround.