Skip to content

[aws-batch] include .snakemake/storage #453

@joverlee521

Description

@joverlee521

Context

Originally discussed in nextstrain/zika#89 (comment)

We are migrating workflows to use Snakemake's storage plugins for supporting remote files. The remote files are downloaded and kept locally within .snakemake/storage. For users to be able to download these files from aws-batch builds, the path needs to be included in

included = path_matcher([
# But we do want the Snakemake logs to come over…
".snakemake/log/",
# …and the input/output metadata Snakemake tracks (akin to mtimes,
# which we also preserve).
".snakemake/metadata/",
])

Also consider if we need to support uploading the path as well. As @jameshadfield commented

Uploading local copies of remote storage will slow down job submission significantly. Maybe there's some use cases where this is what we want, but since local copies are liable to be overwritten at runtime anyway (via mtime cache invalidation) I think it'd be acceptable to not upload them.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions