Compatibility with snakemake version 8.0 and higher

[TBradley27]

Hello,

Snakemake was recently updated to version 8.0 with quite extensive changes to its API. I was wondering if there were any plans to update this repository so that it is compatible with the latest version of snakemake?

I attempted to use a profile generated from the latest version of this repo with snakemake 8.0 and received the following error:

snakemake: error: unrecognized arguments: --cluster-cancel=scancel --cluster=profile/test_profile_dir/slurm/slurm-submit.py --cluster-status=profile/test_profile_dir/slurm/slurm-status.py

As an alternative approach, I attempted to use the snakemake executor plug-in, but also encountered issues (cf: snakemake/snakemake-executor-plugin-slurm#11)

As a result, there is functionality in snakemake versions <8.0, which does not seem to exist in the latest version of snakemake

[percyfal]

Hi @TBradley27, I'm just catching up on this, so while the answer is yes to updating the repo, I need to look into some of the details first. Have you tried changing the option names --cluster-cancel, --cluster-status in your config file to --cluster-generic-cancel-cmd, --cluster-generic-status-cmd and so on? These options become available after installing snakemake-executor-plugin-cluster-generic and snakemake-executor-plugin-slurm. Here I don't know if the specific plugins are meant to invoke specific option names (e.g., --cluster-slurm-status), but that doesn't seem to be the case at present?

[TMAdams]

Hi @percyfal,

I've come across this same issue when new users have attempted to run our snakemake workflows in our recommended run mode with this profile.

I've done some experimentation based on your comment on this issue, and have found I need to make the following changes to force snakemake to run, alongside installation of the slurm and generic executor modules.

cluster: "slurm-submit.py" changed to cluster-generic-submit-cmd: "slurm-submit.py"
cluster-cancel: "scancel" changed to cluster-generic-cancel-cmd: "scancel"
cluster-status: "slurm-status.py" changed to cluster-generic-status-cmd: "slurm-status.py"
cluster-sidecar: "slurm-sidecar.py" changed to cluster-generic-sidecar-cmd; "slurm-sidecar.py"

Unfortunately the logging and job ID functionality that this profile used to provide still don't work with this fix, but it does at least mean it can run! The default sbatch settings also aren't working as before.

[blaiseli]

I generated a profile using cookiecutter, then edited it according to @TMAdams suggestion, and now snakemake is able to perform a dry-run.
However, the log annotates all the rules as "localrule". So I suppose they will not be executed via slurm, but on the submit node, which is wrong.

My snakemake command is the following:

snakemake -s ${snakefile} \
    --executor slurm \
    -j 144 \
    --configfile ${profile} \
   -n

Here is ${profile}/config.yaml

# Manually edited according to https://github.com/Snakemake-Profiles/slurm/issues/117#issuecomment-1906448548
#cluster-cancel: "scancel"
cluster-generic-cancel-cmd: "scancel"
restart-times: "3"
jobscript: "slurm-jobscript.sh"
#cluster: "slurm-submit.py"
cluster-generic-submit-cmd: "slurm-submit.py"
#cluster-status: "slurm-status.py"
cluster-generic-status-cmd: "slurm-status.py"
max-jobs-per-second: "10"
max-status-checks-per-second: "10"
local-cores: 1
latency-wait: "240"
use-conda: "False"
use-singularity: "False"
jobs: "144"
printshellcmds: "False"

What am I missing ?

I tried using --drmaa, as suggested in the documentation (https://snakemake.readthedocs.io/en/v8.4.8/tutorial/additional_features.html#cluster-execution), but I get a snakemake: error: unrecognized arguments: --drmaa.

[TMAdams]

I generated a profile using cookiecutter, then edited it according to @TMAdams suggestion, and now snakemake is able to perform a dry-run. However, the log annotates all the rules as "localrule". So I suppose they will not be executed via slurm, but on the submit node, which is wrong.

My snakemake command is the following:

snakemake -s ${snakefile} \
    --executor slurm \
    -j 144 \
    --configfile ${profile} \
   -n

Here is ${profile}/config.yaml

# Manually edited according to https://github.com/Snakemake-Profiles/slurm/issues/117#issuecomment-1906448548
#cluster-cancel: "scancel"
cluster-generic-cancel-cmd: "scancel"
restart-times: "3"
jobscript: "slurm-jobscript.sh"
#cluster: "slurm-submit.py"
cluster-generic-submit-cmd: "slurm-submit.py"
#cluster-status: "slurm-status.py"
cluster-generic-status-cmd: "slurm-status.py"
max-jobs-per-second: "10"
max-status-checks-per-second: "10"
local-cores: 1
latency-wait: "240"
use-conda: "False"
use-singularity: "False"
jobs: "144"
printshellcmds: "False"

What am I missing ?

I tried using --drmaa, as suggested in the documentation (https://snakemake.readthedocs.io/en/v8.4.8/tutorial/additional_features.html#cluster-execution), but I get a snakemake: error: unrecognized arguments: --drmaa.

I've not experimented directly, but looking at my config compared to yours, the only difference I can see is that yours doesn't appear to have a slurm-sidecar listed? I also specify my profile with the --profile option and reserve --configfile for the config.yaml that controls parameters for the workflow, though it may be the way our different workflows are set up meaning that.

Mine are also marked as "localrule" but run through the scheduler as expected, see an example from a dry-run below:

[Mon Feb 12 10:51:02 2024]
localrule blast_plot:
    input: /mnt/shared/scratch/tadams/DM_chromosomes/PGSC_DM_v4.03_pseudomolecules.fasta, results/H2_DM/blast/blast_results.txt, results/H2_DM/blast/interesting_nlrs.txt, results/H2_DM/blast/genome_sizes.txt
    output: results/H2_DM/blast/blast_plot.png
    log: logs/H2_DM/blast_plot/blast_plot.log
    jobid: 47
    reason: Input files updated by another job: results/H2_DM/blast/interesting_nlrs.txt
    wildcards: experiment=H2_DM
    resources: tmpdir=<TBD>, mem_mb=1000, mem_mib=954

[blaiseli]

Thanks for your quick answer.

I made a type when posting my command. This should be:

snakemake -s ${snakefile} \
    --executor slurm \
    -j 144 \
    --profile ${profile} \
    --configfile ${configfile} \
   -n

I didn't include slurm-sidecar when generating the profile, because I did not understand the documentation and I was not sure it would work. But maybe this is the default for some good reason.

I will try to actually run the pipeline with a very reduced set of inputs, to see if the rules are correctly submitted to compute nodes. In any case, the "localrule" annotation is at best misleading.

[blaiseli]

This seems to work (i.e., jobs are submitted to slurm when not performing a dry run), but the job names (as displayed by the squeue command) do not seem to comply with the pattern specified in CookieCutter.py.

In my case:

@staticmethod
def get_cluster_jobname() -> str:
     return "%r_%w"

I obtain stuff like "c31263ea-65ba-4" in the NAME column of squeue output.

Using sidecar didn't help with this.

[TMAdams]

Yeah I've observed the same behaviour with job names. Haven't been able to fix it or to get the logging back to how it was, but at least it runs!

[blaiseli]

Where should this job naming issue be reported ? Here, or to the main snakemake repo ?

[blaiseli]

I think I found how to display a better job name in squeue, based on this line of code:
https://github.com/snakemake/snakemake-executor-plugin-slurm/blob/5dfd514b3c246acbeab9257ae029c039105672a0/snakemake_executor_plugin_slurm/__init__.py#L77

The job name is stored as comment in sbatch, so to display it in squeue, you need to pass "%k" to the -o option.

[blaiseli]

@TMAdams In the following issue, someone suggests a hack to set job names: snakemake/snakemake-executor-plugin-slurm#18 (comment)