Slurm utils design

[ffelten]

Discussion with @markfuge and @g-braeunlich

Main use cases

• Dataset generation: calling optimize or simulate. Currently uses the config and design factories to create a parameter_space using slurm.Args.
• Starting gradient descent from different initial points: you only change the starting_point of the optimization (all other parameters kept unchanged).
• We want to run things other than optimize or simulate: render or even custom code could be run inside the job (e.g., using a callback). MapReduce like logic.
• Group small runs into a bigger job to reduce Slurm scheduling overhead. Might be useful for ML models evaluations.

Additional features

• Specify how long each runtime should be for Slurm for jobs that are too long (bad simulations). This timeout is job specific and Euler specific.
• How to check which run failed? How do you track back to the config/parameters/args that triggered this run? What we want: job id, error or timeout, args used to run this job.

Found bugs

• Job array size limit on Euler (kicked out by Euler)
• OOM killed for the reduce node (ask for a bigger node for reduce)

[cashend]

What was the job array size limit on Euler? If its too low, maybe we can petition the hpc IT people to increase it, like I did with Zaratan at UMD.

Also I think the default memory for Euler is 1GB per cpu core, which seems very low. You can try to get more by increasing the amount specified by --mem-per-cpu=MEM. ex 4GB: --mem-per-cpu=4G

[markfuge]

@cashend in my experience I was able to get job array lengths of around 1K with no issue. 2K requests were rejected. So somewhere in the 1-2LK range. Also, we need to be considerate with requesting both CPU and Memory, as they track whether the SLURM jobs are being efficient:
https://scicomp.ethz.ch/wiki/Best_Practices#How_to_Improve_Your_Resource_Usage
I mentioned this to Geri during our SLURM debrief and he will be looking into how to modify our script accordingly. In my case, the memory requirements were not so large and the job length was not so long, so Euler would prefer to batch the jobs together to reduce demand on the scheduler. I think this will be task/problem dependent and also dependent on the specific cluster, so this is not something we can mandate by fiat in the library and we should let the user configure this

[ffelten]

#144