LINDA with singularity and batch processing with slurm

[tib-sim]

Hi - I wonder if someone already tried to used LINDA on singularity and use the BATCH processsing with slurm?

I tried the slurm code below with a R script encompassing the required lines of command to start LINDA.

So far, I have an error saying that the file does not exist while I have specified the right path.

#!/bin/bash

#SBATCH --job-name=linda
#SBATCH --output=./log/linda.out
#SBATCH --error=./log/linda.err
#SBATCH --cpus-per-task=10
#SBATCH --mem=40G

WD="/my/working/directory"

LINDA_SINGULARITY_IMG="/directory/to/linda/image/linda_latest.sif"
SCRIPT_R=${WD}"/LINDA_script.R"

srun singularity run --cleanenv --bind ${WD}:/home ${LINDA_SINGULARITY_IMG} Rscript SCRIPT_R

[dorianps]

I have not used it myself slurm, maybe SGE long time ago. I also have not used it through containers. Not sure if anyone else has done it.

You probably need to go step by step, make sure it works with singularity in a regular shell instead of slurm job. Your script must point to the container folders you are binding. If you manage to get it to work, sending slurm jobs will be a natural extension.

Out of curiosity, I am a big fan of containers for reproducibility purposes, but why are you trying to go through containers in this case? Might as well call the jobs through a regular Linda installation in the hpc.

[tib-sim]

Hi - Thanks for the prompt reply.

I understood my mistake the line srun singularity run --cleanenv --bind ${WD}:/home ${LINDA_SINGULARITY_IMG} Rscript SCRIPT_R has to be edited as follows srun singularity run --cleanenv --bind ${WD}:/home ${LINDA_SINGULARITY_IMG} Rscript ${SCRIPT_R}

Actually I am using the container because R is not installed on the hpc and seems it takes some time to the IT team of my instution to install it. I did not want to wait too long before starting to play around with LINDA (looks like a very nice tool btw!). That is the main reason. Also with all the admin rights I am not sure how easy it would be to install R libraries so it is possible I'll stick with the container because it will make my life easier.

Also, I saw in an other issue raised the possibility that LINDA could become a BIDS-app, is there any update on this side?

Best,
Simon

[dorianps]

No, sorry, I have not planned to work on the bids app for now, but will see about it later. Fyi, R packages do not require admin privileges, they will be installed in your home folder. You may not even need admin to install R, but I am not sure of that. Please let the community know imhere if you make it work, so others can learn.

…

On Sun, Sep 3, 2023, 12:52 PM Simon Thibault ***@***.***> wrote: Hi - Thanks for the prompt reply. I understood my mistake the line srun singularity run --cleanenv --bind ${WD}:/home ${LINDA_SINGULARITY_IMG} Rscript SCRIPT_R has to be edited as follows srun singularity run --cleanenv --bind ${WD}:/home ${LINDA_SINGULARITY_IMG} Rscript ${SCRIPT_R}``` Actually I am using the container because R is not installed on the hpc and seems it takes some time to the IT team of my instution to install it. I did not want to wait too long before starting to play around with LINDA (looks like a very nice tool btw!). That is the main reason. Also with all the admin rights I am not sure how easy it would be to install R libraries so it is possible I'll stick with the container because it will make my life easier. Also, I saw in an other issue raised the possibility that LINDA could become a BIDS-app, is there any update on this side? Best, Simon — Reply to this email directly, view it on GitHub <#36 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACFJU7IK5UOBT6PEUTLY56LXYSYVJANCNFSM6AAAAAA4IVH43U> . You are receiving this because you commented.Message ID: ***@***.***>

[dorianps]

One more question. Where did you find the singularity image? Converting from docker typically requires admin privileges.

…

On Sun, Sep 3, 2023, 12:52 PM Simon Thibault ***@***.***> wrote: Hi - Thanks for the prompt reply. I understood my mistake the line srun singularity run --cleanenv --bind ${WD}:/home ${LINDA_SINGULARITY_IMG} Rscript SCRIPT_R has to be edited as follows srun singularity run --cleanenv --bind ${WD}:/home ${LINDA_SINGULARITY_IMG} Rscript ${SCRIPT_R}``` Actually I am using the container because R is not installed on the hpc and seems it takes some time to the IT team of my instution to install it. I did not want to wait too long before starting to play around with LINDA (looks like a very nice tool btw!). That is the main reason. Also with all the admin rights I am not sure how easy it would be to install R libraries so it is possible I'll stick with the container because it will make my life easier. Also, I saw in an other issue raised the possibility that LINDA could become a BIDS-app, is there any update on this side? Best, Simon — Reply to this email directly, view it on GitHub <#36 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACFJU7IK5UOBT6PEUTLY56LXYSYVJANCNFSM6AAAAAA4IVH43U> . You are receiving this because you commented.Message ID: ***@***.***>

[tib-sim]

Ok thanks for the information, I am still not super familiar with the hpc. Actually in my former institution I was not allowed to use sudo on the hpc. It looks like I have some admin rights on the hpc of my current institution so maybe it is possible to install R on my own.

Given I have been able to use batch processing to run 2 subjects in parallel on the hpc I think I will stick using the containers.

To get the image I ran the command singularity pull docker://dorianaps/linda:latest. I don't think I need admin right to get it. At least I was able to get singularity images on the hpc of my former institution as well. Maybe they restricted the use of sudo only. I am not so sure to be honest.