Errors #32
Description
Hello,
First of all, thanks a lot for the great pipeline.
I'm having a bit of trouble getting stainedGlass to work on my longest sequences however. I'm using the provided condo environments with snakemake and during figure generation I get a variety of different errors, but only for the longest sequences.
This is my command:
mamba activate stainedGlass
samtools faidx ${region_fasta}
snakemake --use-conda --cores ${task.cpus} all make_figures cooler --config sample=${name} fasta=${region_fasta} alnthreads=8 nbatch=16
These are some of the error messages:
[Tue Jan 2 00:47:58 2024]
Finished job 48.
46 of 49 steps (94%) done
Select jobs to execute...
[Tue Jan 2 00:47:58 2024]
localrule cooler:
input: results/chr2_centromere.2000.10000.strand.cool, results/chr2_centromere.2000.10000.identity.cool, results/chr2_centromere.2000.10000.strand.mcool, results/chr2_centromere.2000.10000.identity.mcool
jobid: 44
reason: Input files updated by another job: results/chr2_centromere.2000.10000.strand.mcool, results/chr2_centromere.2000.10000.identity.cool, results/chr2_centromere.2000.10000.identity.mcool, results/chr2_centromere.2000.10000.strand.cool
resources: tmpdir=/tmp
[Tue Jan 2 00:47:58 2024]
Finished job 44.
47 of 49 steps (96%) done
Error in checkForRemoteErrors(val) :
one node produced an error: there is no package called ‘labeling’
Calls: parLapply ... clusterApply -> staticClusterApply -> checkForRemoteErrors
Execution halted
Error: package ‘tools’ does not have a namespace
Error in unserialize(node$con) : error reading from connection
Calls: <Anonymous> ... doTryCatch -> recvData -> recvData.SOCKnode -> unserialize
Execution halted
Error: package ‘tools’ does not have a namespace
[Tue Jan 2 04:16:25 2024]
Error in rule make_R_figures:
jobid: 43
input: results/chr2_centromere.2000.10000.bed.gz
output: results/chr2_centromere.2000.10000_figures/pdfs/chr2_centromere.2000.10000.tri.TRUE__onecolorscale.FALSE__all.pdf, results/chr2_centromere.2000.10000_figures/pngs/chr2_centromere.2000.10000.tri.TRUE__onecolorscale.TRUE__all.png, results/chr2_centromere.2000.10000_figures/pngs/chr2_centromere.2000.10000.tri.TRUE__onecolorscale.FALSE__all.png, results/chr2_centromere.2000.10000_figures/pngs/chr2_centromere.2000.10000.tri.FALSE__onecolorscale.TRUE__all.png, results/chr2_centromere.2000.10000_figures/pngs/chr2_centromere.2000.10000.tri.FALSE__onecolorscale.FALSE__all.png, results/chr2_centromere.2000.10000_figures/pngs/chr2_centromere.2000.10000.facet.all.png
log: logs/make_R_figures.chr2_centromere.2000.10000.log (check log file(s) for error details)
conda-env: /scratch-cbe/users/diego.terrones/5_AxolotlT2T/6_centromereAnalysis/1_Ambystoma-mexicanum/1_StainedGlass/cb/309e7904e7bbeb60a61c79c1f50d37/.snakemake/conda/5d32f79cd3fb5bb0c333cf2c276a3eaf_
shell:
Rscript /users/diego.terrones/.cache/snakemake/snakemake/source-cache/runtime-cache/tmpqrqptiq2/file/scratch-cbe/users/diego.terrones/5_AxolotlT2T/6_centromereAnalysis/1_Ambystoma-mexicanum/1_StainedGlass/cb/309e7904e7bbeb60a61c79c1f50d37/workflow/scripts/aln_plot.R --bed results/chr2_centromere.2000.10000.bed.gz --threads 16 --prefix chr2_centromere.2000.10000
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2024-01-01T164206.294919.snakemake.log
Traceback (most recent call last):
File "/users/diego.terrones/miniforge3/envs/stainedGlass/lib/python3.9/weakref.py", line 667, in _exitfunc
f()
File "/users/diego.terrones/miniforge3/envs/stainedGlass/lib/python3.9/weakref.py", line 591, in __call__
return info.func(*info.args, **(info.kwargs or {}))
File "/users/diego.terrones/miniforge3/envs/stainedGlass/lib/python3.9/tempfile.py", line 820, in _cleanup
cls._rmtree(name)
File "/users/diego.terrones/miniforge3/envs/stainedGlass/lib/python3.9/tempfile.py", line 816, in _rmtree
_shutil.rmtree(name, onerror=onerror)
File "/users/diego.terrones/miniforge3/envs/stainedGlass/lib/python3.9/shutil.py", line 734, in rmtree
_rmtree_safe_fd(fd, path, onerror)
File "/users/diego.terrones/miniforge3/envs/stainedGlass/lib/python3.9/shutil.py", line 667, in _rmtree_safe_fd
_rmtree_safe_fd(dirfd, fullname, onerror)
File "/users/diego.terrones/miniforge3/envs/stainedGlass/lib/python3.9/shutil.py", line 667, in _rmtree_safe_fd
_rmtree_safe_fd(dirfd, fullname, onerror)
File "/users/diego.terrones/miniforge3/envs/stainedGlass/lib/python3.9/shutil.py", line 667, in _rmtree_safe_fd
_rmtree_safe_fd(dirfd, fullname, onerror)
[Previous line repeated 8 more times]
File "/users/diego.terrones/miniforge3/envs/stainedGlass/lib/python3.9/shutil.py", line 673, in _rmtree_safe_fd
onerror(os.rmdir, fullname, sys.exc_info())
File "/users/diego.terrones/miniforge3/envs/stainedGlass/lib/python3.9/shutil.py", line 671, in _rmtree_safe_fd
os.rmdir(entry.name, dir_fd=topfd)
OSError: [Errno 39] Directory not empty: 'scripts'
Or this one:
Activating conda environment: .snakemake/conda/284fb86d478e02947ecbd7c10b955f52_
[Tue Jan 2 01:45:06 2024]
localrule all:
input: results/chr4_centromere.2000.10000.sorted.bam, results/chr4_centromere.2000.10000.bed.gz, results/chr4_centromere.2000.10000.full.tbl.gz
jobid: 43
reason: Input files updated by another job: results/chr4_centromere.2000.10000.sorted.bam, results/chr4_centromere.2000.10000.full.tbl.gz, results/chr4_centromere.2000.10000.bed.gz
resources: tmpdir=/tmp
[Tue Jan 2 01:45:06 2024]
Finished job 43.
42 of 49 steps (86%) done
/scratch-cbe/users/diego.terrones/5_AxolotlT2T/6_centromereAnalysis/1_Ambystoma-mexicanum/1_StainedGlass/25/abdb34cc958c5259e797ff05e5cab3/.snakemake/conda/d959733dddc863405d2eb9dbb534ae0d_/lib/R/bin/R: line 272: /scratch-cbe/users/diego.terrones/5_AxolotlT2T/6_centromereAnalysis/1_Ambystoma-mexicanum/1_StainedGlass/25/abdb34cc958c5259e797ff05e5cab3/.snakemake/conda/d959733dddc863405d2eb9dbb534ae0d_/lib/R/bin/exec/R: No such file or directory
[Tue Jan 2 01:45:06 2024]
Error in rule make_R_figures:
jobid: 1
input: results/chr4_centromere.2000.10000.bed.gz
output: results/chr4_centromere.2000.10000_figures/pdfs/chr4_centromere.2000.10000.tri.TRUE__onecolorscale.FALSE__all.pdf, results/chr4_centromere.2000.10000_figures/pngs/chr4_centromere.2000.10000.tri.TRUE__onecolorscale.TRUE__all.png, results/chr4_centromere.2000.10000_figures/pngs/chr4_centromere.2000.10000.tri.TRUE__onecolorscale.FALSE__all.png, results/chr4_centromere.2000.10000_figures/pngs/chr4_centromere.2000.10000.tri.FALSE__onecolorscale.TRUE__all.png, results/chr4_centromere.2000.10000_figures/pngs/chr4_centromere.2000.10000.tri.FALSE__onecolorscale.FALSE__all.png, results/chr4_centromere.2000.10000_figures/pngs/chr4_centromere.2000.10000.facet.all.png
log: logs/make_R_figures.chr4_centromere.2000.10000.log (check log file(s) for error details)
conda-env: /scratch-cbe/users/diego.terrones/5_AxolotlT2T/6_centromereAnalysis/1_Ambystoma-mexicanum/1_StainedGlass/25/abdb34cc958c5259e797ff05e5cab3/.snakemake/conda/d959733dddc863405d2eb9dbb534ae0d_
shell:
Rscript /users/diego.terrones/.cache/snakemake/snakemake/source-cache/runtime-cache/tmp1vhq84v1/file/scratch-cbe/users/diego.terrones/5_AxolotlT2T/6_centromereAnalysis/1_Ambystoma-mexicanum/1_StainedGlass/25/abdb34cc958c5259e797ff05e5cab3/workflow/scripts/aln_plot.R --bed results/chr4_centromere.2000.10000.bed.gz --threads 16 --prefix chr4_centromere.2000.10000
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
Traceback (most recent call last):
Traceback (most recent call last):
File "/scratch-cbe/users/diego.terrones/5_AxolotlT2T/6_centromereAnalysis/1_Ambystoma-mexicanum/1_StainedGlass/25/abdb34cc958c5259e797ff05e5cab3/.snakemake/conda/284fb86d478e02947ecbd7c10b955f52_/bin/cooler", line 7, in <module>
File "/scratch-cbe/users/diego.terrones/5_AxolotlT2T/6_centromereAnalysis/1_Ambystoma-mexicanum/1_StainedGlass/25/abdb34cc958c5259e797ff05e5cab3/.snakemake/conda/284fb86d478e02947ecbd7c10b955f52_/bin/cooler", line 7, in <module>
from cooler.cli import cli
from cooler.cli import cli
ImportError: cannot import name 'cli' from 'cooler.cli' (unknown location)
ImportError: cannot import name 'cli' from 'cooler.cli' (unknown location)
[Tue Jan 2 01:45:07 2024]
Error in rule cooler_strand:
jobid: 45
input: results/chr4_centromere.2000.10000.bed.gz, /scratch-cbe/users/diego.terrones/5_AxolotlT2T/6_centromereAnalysis/1_Ambystoma-mexicanum/1_StainedGlass/25/abdb34cc958c5259e797ff05e5cab3/regions.part_004.fa.fai
output: results/chr4_centromere.2000.10000.strand.cool
log: logs/cooler_strand.chr4_centromere.2000.10000.log (check log file(s) for error details)
conda-env: /scratch-cbe/users/diego.terrones/5_AxolotlT2T/6_centromereAnalysis/1_Ambystoma-mexicanum/1_StainedGlass/25/abdb34cc958c5259e797ff05e5cab3/.snakemake/conda/284fb86d478e02947ecbd7c10b955f52_
shell:
gunzip -c results/chr4_centromere.2000.10000.bed.gz | tail -n +2 | sed 's/+$/100/g' | sed 's/-$/50/g' | cooler cload pairs -c1 1 -p1 2 -c2 4 -p2 5 --field count=8:agg=mean,dtype=float --chunksize 50000000000 /scratch-cbe/users/diego.terrones/5_AxolotlT2T/6_centromereAnalysis/1_Ambystoma-mexicanum/1_StainedGlass/25/abdb34cc958c5259e797ff05e5cab3/regions.part_004.fa.fai:2000 --zero-based - results/chr4_centromere.2000.10000.strand.cool
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
[Tue Jan 2 01:45:07 2024]
Error in rule cooler_identity:
jobid: 46
input: results/chr4_centromere.2000.10000.bed.gz, /scratch-cbe/users/diego.terrones/5_AxolotlT2T/6_centromereAnalysis/1_Ambystoma-mexicanum/1_StainedGlass/25/abdb34cc958c5259e797ff05e5cab3/regions.part_004.fa.fai
output: results/chr4_centromere.2000.10000.identity.cool
log: logs/cooler_identity.chr4_centromere.2000.10000.log (check log file(s) for error details)
conda-env: /scratch-cbe/users/diego.terrones/5_AxolotlT2T/6_centromereAnalysis/1_Ambystoma-mexicanum/1_StainedGlass/25/abdb34cc958c5259e797ff05e5cab3/.snakemake/conda/284fb86d478e02947ecbd7c10b955f52_
shell:
gunzip -c results/chr4_centromere.2000.10000.bed.gz | tail -n +2 | cooler cload pairs -c1 1 -p1 2 -c2 4 -p2 5 --field count=7:agg=mean,dtype=float --chunksize 50000000000 /scratch-cbe/users/diego.terrones/5_AxolotlT2T/6_centromereAnalysis/1_Ambystoma-mexicanum/1_StainedGlass/25/abdb34cc958c5259e797ff05e5cab3/regions.part_004.fa.fai:2000 --zero-based - results/chr4_centromere.2000.10000.identity.cool
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2024-01-01T172106.733496.snakemake.log
Weirdly, the shorter sequences work just fine, so I'm a bit confused. Any ideas?