Frank's feedback on using Slurm

[ffelten]

Version

• Not used the latest version -- branch HPC-Zaratan needs to update to Geri's latest version of the Slurm API.

Experience with the Slurm API

• The API seems to work quite well once you understand it.
• Did not test the reduce part.

Pain points

• Singularity to Apptainer Support Apptainer #148
• Use reset with different seeds to create different engibench_studies in Airfoil. Why is the study_None generated when init is called? (see Frank's branch)
• More (advanced) examples of usage for instance using the Slurm API would be useful to understand the API. (see actions)
• Container should be cached and not redownloaded by each job. Slurm scripts caching #140

Gotchas

• Formula to compute total number of jobs from group size can drop some jobs due to rounding. (see Frank's branch)
• MPI related tmp files were being written in a memory bounded folder. Solved by using Apptainer instead of Singularity. Can be solved by mounting your own tmp dir but it will clash because all jobs write to the same tmp. (see Actions)
• Amount of requested resources is different than the job array cap. The fix was to request the job arrays sequentially. (see Frank's PR)

Actions

• Merge Frank's branch into Slurm

• Merge Slurm branch into main

• Add documentation and example

• @ffelten check if Euler supports Apptainer: Yes they do https://scicomp.ethz.ch/wiki/Apptainer -- move on with Support Apptainer #148

[fgvangessel-umd]

Note I've merged slurm into my hpc-zaratan branch. Should I now go the other way and merge hpc-zaratan into Slurm? Not exactly sure what the git workflow is here.

[ffelten]

@fgvangessel-umd that's the right git flow yes. But we said @g-braeunlich will do it :). Enjoy your holidays !

[g-braeunlich]

Just to say: there is no "right" here. There are at least 2 philosophies:

• The "merge" approach (your approach)
• The "rebase" approach (dont merge, but rebase)

The latter is my favorite: it leads to clean atomic commits which can be reviewed much more easily. Also, this approach leads to a clean linear history where it is transparent where changes come from.

Also: It looks to me, that the current state of your branch will lead to some potential runtime crashes. E.g.
here: I cannot see a definition of sif_image and therefore would expect a crash at this line.

I will try to fix that, but that also means that the logic might change and behavior will change.

[fgvangessel-umd]

THanks for the clarification Geri. Regarding the sif_image, it looks like I removed the line below when merging:

# Get sif filename
sif_image = cls.sif_filename(image)

Which ensures we are using the correct Mach_Aero container name as it appears when pulled using singularity/apptainer.

[ffelten]

@fgvangessel-umd did you have the chance to talk with @g-braeunlich to know which changes coming from your branch have not been merged into the slurm branch, i.e., what can we still scavenge from your work on Airfoil?

[fgvangessel-umd]

Hey Florian,

I emailed with Geri and he rebased all my changes into a subranch of SLURM (hpc-zaratan-rebased). However, when I try to run the data generation script it errors out. I believe the error results form this line in sbatch_map:

work_dir = tempfile.mkdtemp(dir=os.environ.get("SCRATCH"))

The environment variable SCRATCH is not consistently set across Zaratan (i.e. gives different or nonaccesible paths on login vs backend) and the created directories don't appear ot be populated with the appropriate pkl files for performing the mapping. I haven't been able to debug further than this. Previously this line looked like:

work_dir = tempfile.mkdtemp(dir='/home/fvangess/scratch/EngiBench/scratch')

Obvioulsy that is hard-coded and not a general solution either.

[g-braeunlich]

I cannot imagine that the "scratch" line is a problem:

work_dir = tempfile.mkdtemp(dir=os.environ.get("SCRATCH"))

If SCRATCH is not set, then os.environ.get("SCRATCH") evaluates to None which is the default value for the dir argument of tempfile.mkdtemp.

[g-braeunlich]

Btw.: the latest slurm API now has much better error handling: errors originating in a slurm process will be propagated to the script calling sbatch_map() - including a backtrace.

[fgvangessel-umd]

I'll try running this problem on the main slurm branch. @g-braeunlich you are likely correct, that was just the first divergence in the codebase between my version and the rebased version. I will say it is difficult to debug those errors though because I cannot view/access the SCRATCH directory of a backend node from the login nodes.

[fgvangessel-umd]

Had a chance this weekend to test out the airfoil dataset generator on main. Good news is that the dataset generation works! I do observe some strange behavior where this only works as long as the entire wokflow is run through slurm. What I mean by that is that the code calling slurm.sbatch_map must be running via a slurm job on a backend node (i.e. if we submit on a login node things won't work). I'm not sure exactly what is causing this but running everything on backend nodes is best practice, albeit less convenient for debugging.

My next step is to add the dataset generator python file (and helper functions) to the airfoil problem folder of engibench. I believe we wanted these Slurm examples included for the next major release. FInally, I'd like to also add the ability to output field data along with lift and drag. I can already do this on my own zaratn-hpc branch but I'll speak with the team on how to best integrate with main.