Lmod missing .version file in module tree

[nlvw]

Describe the bug
Following the getting started guide, as well as any of the install instructions in the registries, fails when performing any referenced module load command. This is due to how shpc is creating the directory tree. Lmod expects the .lua file to be the version and the directory to be the name. Since shpc is creating the version as a directory, and then a module.lua file, Lmod is getting confused. To solve this shpc should be creating an empty .version file in the same folder as the version directory (see https://lmod.readthedocs.io/en/latest/055_module_names.html#module-naming-scheme-name-version-version-n-v-v.

To Reproduce
Install any container and try to load it following the provided commands. Following the getting started guide in the docs works too.

Expected behavior
The name should be separate from the software version. So for the following

$ module avail

---------------------------------- /software/singularity/shpc/shpc/modules -----------------------------------
   biocontainers/samtools/v1.9-4-deb_cv1/module (L)

'biocontaners/samtools' should be the name and 'v1.9-4-deb_cv1/module' should be the version. This allows for module load biocontainers/samtools to work.

Other
lmod showing '/module' as part of the full module name bugs me but I see no way to suppress this other then an overhaul of how shpc organizing things.

Software Versions

• Lmod = 8.2.7
• shpc = 0.0.34
• singularity = 3.8.0
• OS = CentOS 7.9

[vsoch]

@marcodelapierre is this a version issue with LMOD? Why haven't we seen this before?

[vsoch]

@nlvw we can certainly add an empty .version file in the top level folder, or after the container name and before the tag, if that solves some of your problem. Would you care to open a PR to test?

[nlvw]

@vsoch probably a bit beyond my skill set. I can however give you a screenshot of where the .version needs to be placed.

Any idea on why Lmod is appending a '/module'. Not sure if that showed up in your testing but I can't find anything in the Lmod docs for ignoring the name of the lua file.

[vsoch]

I'm not an expert so I'm not sure - I suspect usually the module file is named according to a tag or version, but for shpc we just needed the filename to be predictable without that (hence the naming). Let's try the .version file and see how that does (or does not) change things.

I should have some time this evening to work on this.

[nlvw]

This also seems to fix a subsequent compatibility issue with Lmod where multiple versions of the same container are not grouped together with the newest version getting marked as default. The default is important or your have to always use the full module name, which is generally what we tell users to do anyways to be fair.

Before:

After adding .version:

I'll leave it at that.

[marcodelapierre]

Hi @vsoch , @nlvw ,
I would say that both Lmod and EnvModules allow to configure the behaviour regarding default module versions in various ways:

• explicit choice of a default version (formatted info inside .version)
• default version according to Lmod/EnvModules rules (I think it's generally the largest version number)
• no default - what shpc does right now

One can tune this behaviour through the module directory tree structure (eg version as file name or directory name) and/or the use of .version.

As Nicholas mentions, with the current shpc setup, this works:

module load <tool>/<version>

but this does not:

module load <tool>

In our centre, we like the idea of not having a default and forcing users to specify a version, as we find this better for reproducibility of the workflows.

I don't think there's a right and wrong way, but it would be definitely beneficial if shpc allowed more flexibility in the configuration to accommodate various needs.

For instance, having an option to create or not create the .version file, eg default_version = true/false

This would

[marcodelapierre]

A further level would be to allow the choice of which version is the default.
However, this gets tricky to implement, because it is a choice that is both site specific and package specific.
It boils down to a trade-off between simplicity and flexibility.

For now, the above suggestion of a default_module toggle may be a good trade off, extended functionality yet simple to handle in the configuration.

What do you guys think?

[vsoch]

@marcodelapierre the customization might be a better option - and I can tweak the PR I opened for it. But what should be the default?

[marcodelapierre]

In the case default = true, I think you could just create an empty .version in the location Nicholas suggestion suggested.
That way, Lmod will pick the highest version number as default (info is buried somewhere in the link he provided above)

[vsoch]

Done! Just updated the PR. 6dc57aa

[vsoch]

This is all set. Thanks y'all!