docker

[aryarm]

we could simplify dependency installation even more using docker

Update
Actually, using docker does not simplify dependency installation, since when our docker container is created, it will have to install everything it needs using conda anyway.
But docker can improve reproducability by making the pipeline executable on other platforms (ex: Windows or Mac). That is the only reason why we chose to pursue this issue.

[umasstr]

Hi Arya,

Do you have a working docker image for this pipeline? I'm not familiar with snakemake, but the version specified fails to install locally using the setup instructions provided, even in the continuumio/miniconda3 container. Surprisingly, the CodeOcean Dockerfile in /environment fails to build as well.

I may have misinterpreted the above statement "it will have to install everything it needs using conda anyway." To me, the utility of docker is that end-users don't have to build the image and install from scratch. Once pulled, the container should be ready-to-use.

As is, it seems that other investigators would have an exceedingly difficult time reproducing this workflow on other published ATAC datasets

[aryarm]

Hi @umasstr,

Thanks for your interest in using VarCA! We don't yet have a docker container for this pipeline, but we try to make installation both reproducible and easy by utilizing Snakemake's automated dependency installation, which should install the required dependencies into isolated conda environments upon your first execution of the pipeline so long as snakemake and conda are installed.

In your case, it sounds like the installation of Snakemake itself is the issue. Can I inquire as to what sort of error messages or failed behavior you might be getting when you follow the setup instructions? You can also follow Snakemake's official installation instructions or just use the official Snakemake docker image.

[umasstr]

Thanks for your quick reply! In both cases (codeocean and straight miniconda), The following error led to timeout in the container build or install:

Solving environment: failed with initial frozen solve. Retrying with flexible solve

This issue was presented in conda/conda#9367, but I didn't see a definitive cause identified.

Also, I'd be interested to know if you can build CodeOcean's Dockerfile. If not, then the package manager (Conda) may no longer host that particular version of Snakemate or some dependency. I'll try to build from snakemate/snakemate as well.

[aryarm]

Ah! This is a common issue when installing Snakemake, actually. The issue originates from the fact that Snakemake has many dependencies and it can be tough for conda to resolve all of them quickly, depending on the compute power available. It turns out that dependency resolution is actually an NP-complete problem!

Are you using the --no-channel-priority parameter from the setup instructions when executing conda create or conda install? This usually helps to reduce the search space and has never failed me so far. Also, it always helps to install Snakemake into its own conda environment, as we recommend in the setup instructions, since this will relieve conda of the burden of having to resolve any currently installed dependencies with Snakemake's.

If neither of those options are helping, then I would recommend trying Snakemake's official installation instructions rather than the ones listed in our README. They recommend a solution which is even faster than the one we suggest but requires downloading a separate installer called mamba first. Otherwise, if you'd rather stick to a quick docker solution, I would recommend using the official Snakemake docker image which comes pre-installed with Snakemake and conda. As I mentioned earlier, all of our dependencies should be automatically installed upon initial execution of the pipeline, and this should be true even when executing the pipeline from within a container.

[umasstr]

Hi @aryarm, I did use the --no-channel-priority parameter, copied directly from your README install instructions. I also tried the snakemake/snakemake docker container.

Failed installs

Executing from snakemake/snakemake:latest (last updated on dockerhub the day of this post) after successful download of data dir as described in README, ./run.bash fails. In out dir:

(snakemake) root@f727ad8d337d:/varCA# ls -l

total 48
-rw-r--r-- 1 root root 1070 Apr 6 15:59 LICENSE
-rw-r--r-- 1 root root 6874 Apr 6 15:59 README.md
-rw-r--r-- 1 root root 2983 Apr 6 15:59 Snakefile
drwxr-xr-x 2 root root 4096 Apr 6 15:59 breakCA
drwxr-xr-x 2 root root 4096 Apr 6 15:59 callers
drwxr-xr-x 2 root root 4096 Apr 6 15:59 configs
drwxrwsr-x 2 8218 31441 4096 Jul 7 2020 data
drwxr-xr-x 2 root root 4096 Apr 6 15:59 envs
drwxr-xr-x 2 root root 4096 Apr 6 15:59 rules
-rwxr-xr-x 1 root root 1440 Apr 6 15:59 run.bash
drwxr-xr-x 2 root root 4096 Apr 6 15:59 scripts

(snakemake) root@f727ad8d337d:/varCA# ./run.bash

(snakemake) root@f727ad8d337d:/varCA# cat out/*

WorkflowError in line 5 of /varCA/Snakefile:
Expecting Snakemake version 5.18.0 or higher.
File "/varCA/Snakefile", line 5, in

I do not see a snakemake 5.18.0 docker container currently hosted on their hub site—varCA should've run in the "latest" container, but there may be something wrong with the version notation in the snakemake container. I didn't dig too deeply here.

Building container with snakemake 5.18.0 from miniconda3 docker (same base container used by CodeOcean):
conda create -n snakemake -c bioconda -c conda-forge --no-channel-priority 'snakemake==5.18.0'

Solving environment: failed with initial frozen solve. Retrying with flexible solve

Times out, with or without a -y flag.

Building a working container

First, we have to scrap the install command provided in the README and use instructions from the snakemake docs.

conda install -y -n base -c conda-forge mamba

To activate an environment, we need to overcome conda's cryptic init errors. Fortunately, there's a solution.
eval "$(conda shell.bash hook)"

Now, we can install using mamba, as directed.
conda activate base && mamba create -y -n snakemake -c bioconda -c conda-forge --no-channel-priority 'snakemake==5.18.0'

Finally, we clone your repo, download and unpack your data dir inside varCA. The container builds without issue!

We can now conda activate snakemate and execute ./run.bash inside the container without any install steps required by the end user.

Feel free to try out my container, umasstr/varca. You can also pull and re-tag to distribute as you please.

IMO, conda sucks as means of creating reproducible workflows. Even fans of conda+snakemate implementations note that this package manager regularly purges old package versions. For pipelines with abundant dependencies or stringent version requirements, this death blow comes sooner rather than later. At best, conda is a quick way to install multiple dependencies when building a container for public distribution.

Thanks again for your detailed responses!

[aryarm]

No, thank you for providing me with such an amazing response! This will all be super helpful for us, going forward. I really appreciate the time you put into this.

Failed installs

Executing from snakemake/snakemake:latest

I'm not sure, but it seems like the Snakemake docker container you used pulled in an older version of Snakemake? I can try to look into this and investigate why that might be happening.

Building container with snakemake 5.18.0 from miniconda3 docker

I'm really not sure what might be going on here. It might have something to do with the amount of memory, CPU usage, and cores the container is limited to? As you've already deduced, this worked when we tried it on Code Ocean, and that's the only possible difference I can think of.

Building a working container

I think I will change the instructions in our README based on your success with this option. Thank you so much for letting us know about this and for providing us with a container!

re your comment about conda being a poor solution for reproducibility: I'm still not yet entirely convinced. The blog post you reference says that packages from the Anaconda channel are purged. Even if that were the case, none of our packages come from that channel. For the most part, we use the open-source, community-maintained bioconda and conda-forge channels, which shouldn't ever remove old package versions (see source and source). My understanding is that this is actually one of the reasons that installation via conda has become so slow for packages from those channels (source and source).
At the same time, I think there is still an argument to be made for containerization. And I would like to make this tool as accessible as possible, including to those who prefer to use containers. So I will try to publicize a VarCA container on our README going forward!

[umasstr]

Re: snakemake/snakemake
I found the culprit, the version is untagged in their latest container. Maybe it's for their internal use only.
snakemake --version

0+untagged.1.g2e5294d.dirty

Re: miniconda build
My containers are allocated ~30G of RAM, so I'd be surprised if that's the issue. Also curious that the mamba solution worked, but I don't know enough about conda to say for sure.

Re: conda as package manager
Fair enough, I was unaware that they retained all versions. It hasn't been my experience that conda environments are as ready-to-execute as docker containers, but my problems with bioconda/conda-forge may just be due to poorly-made environments or my own unwillingness to read up on the subject...

Re: container
I also pushed a container umasstr/varca:lite sans the sample data

Very excited to execute this pipeline—great work!

Thanks again!