Context
Originally discussed in nextstrain/zika#89 (comment)
We are migrating workflows to use Snakemake's storage plugins for supporting remote files. The remote files are downloaded and kept locally within .snakemake/storage. For users to be able to download these files from aws-batch builds, the path needs to be included in
|
included = path_matcher([ |
|
# But we do want the Snakemake logs to come over… |
|
".snakemake/log/", |
|
|
|
# …and the input/output metadata Snakemake tracks (akin to mtimes, |
|
# which we also preserve). |
|
".snakemake/metadata/", |
|
]) |
Also consider if we need to support uploading the path as well. As @jameshadfield commented
Uploading local copies of remote storage will slow down job submission significantly. Maybe there's some use cases where this is what we want, but since local copies are liable to be overwritten at runtime anyway (via mtime cache invalidation) I think it'd be acceptable to not upload them.
Context
Originally discussed in nextstrain/zika#89 (comment)
We are migrating workflows to use Snakemake's storage plugins for supporting remote files. The remote files are downloaded and kept locally within
.snakemake/storage. For users to be able to download these files from aws-batch builds, the path needs to be included incli/nextstrain/cli/runner/aws_batch/s3.py
Lines 139 to 146 in 4dbac26
Also consider if we need to support uploading the path as well. As @jameshadfield commented