When/how to modify Snakemake config?

[victorlin]

Description

This issue started out as discussion around ways to support nextstrain run in augur subsample config. The solution is clear: pass filepaths in subsample config through resolve_config_path before dumping the config variable in Snakemake.

The conversation has then moved to discuss whether/how to apply resolve_config_path to all values at the start of Snakemake (instead of within each individual rule), then to discuss when/how to modify Snakemake config. Options:

1. Before running Snakemake.
   • Example: using CUE to generate a phylogenetic/defaults/config.yaml.
   • We don't do this anywhere currently.
2. At Snakemake startup (i.e. before evaluating any rules).
   • Example: modifying a value in config.
     • Measles does this with resolve_filepaths.
   • Example: modifying the structure of config to resolve wildcards.
     • There is a measles PR to do this.
3. At Snakemake runtime.

   • Example: modifying a value from config while using it in a rule.

     rule foo:
         input:
             reference = resolve_config_path(config["files"]["reference"])

   • Most usage of resolve_config_path does this currently, and it's the only way to leverage Snakemake's built-in wildcards functionality.

   • This is not an option for augur subsample, which reads from the dump of config at startup.

Original issue: filter/subsample config compatibility with nextstrain run

I ran into this issue while updating the wnv repo to support nextstrain run (draft).

The issue stems from a fundamental change in working directory behavior between nextstrain build and nextstrain run:

• nextstrain build: Executes in the workflow directory
• nextstrain run: Executes in a user-specified analysis directory

This is a breaking change for all file paths in config, and the updated repos have addressed this using the resolve_config_path() helper function which searches for files in both directories. This works great for rules where file paths are passed directly to individual parameters, allowing each path to be wrapped with resolve_config_path().

augur filter

The WNV repo follows the flexible pattern of generalized subsampling where all augur filter arguments are stored in a literal string:

subsampling:
  region: >-
    --query "is_lab_host != 'true'"
    --query-columns is_lab_host:str
    --min-length '8200'
    --group-by region year
    --subsample-max-sequences 3000
    --exclude defaults/exclude.txt
    --include defaults/all-lineages/include.txt

Since file paths like defaults/exclude.txt are embedded within the literal string, they cannot be individually processed by resolve_config_path(). When executed from the analysis directory, these paths don't exist.

I can't think of a solution that doesn't involve breaking apart the config value into separate strings so that resolve_config_path can be used on the file paths. This is the pattern used in rule filter by other nextstrain run-compatible repos, but goes against the generalized subsampling pattern.

augur subsample

augur subsample has a similar situation. While file paths are accessed directly by config key, the access happens within augur subsample and not Snakemake, so resolve_config_path is not applicable.

A possible solution is to apply resolve_config_path to file paths in subsampling config before writing the config YAML that is then used by augur subsample (draft).

[joverlee521]

A possible solution is to apply resolve_config_path to file paths in subsampling config before writing the config YAML that is then used by augur subsample (draft).

+1 for this approach. I think it's inline with the ideas discussed in the issue re:standard composable config, specifically, the underlying workflow should be using the fully expanded config. Now that I'm thinking on this, maybe we should push to apply resolve_config_path to file paths at the start of all workflows instead of within each individual rule.

[victorlin]

We discussed this at dev chat today. Summary:

• On nextstrain run compatibility:
  • The config for augur filter is a string, but it doesn't have to be. YAML-based config is better for inheriting defaults.
  • I will prototype the solution for augur filter that involves breaking apart the config into separate YAML entries.
• On applying resolve_config_path to all file paths in config at the start of the workflow:
  • This doesn't work for wildcard-based config. It's fine for now, but should be considered later.
  • There are concerns about decoupled references to config entries. Config schemas that clearly mark file paths would improve this.

I've started a PR in the WNV repo to support nextstrain run with augur subsample and applying resolve_config_path at the start. Not ready to be merged, but ready to be reviewed with today's discussion points in mind.

[victorlin]

Prototype of nextstrain run + YAML-configured augur filter: nextstrain/WNV#106

[victorlin]

@tsibley said in nextstrain/WNV#105 (comment):

I would encourage a forward-looking implementation here early on rather than later, esp. since this is almost certainly going to be copied and modified in other repos.

There's a couple ways to make this more reusable and less of an unshared blob of imperative code:

• Make it more declarative by having resolve_config_paths() (or something like it) accept a list of config key locations to resolve (e.g. strings like .build_params[].reference or $.build_params.*.reference or tuples like ("build_params", ..., "reference")). This lets the code be shared while the set of values to resolve can be per-pathogen.
• Use a JSON Schema for the config to figure out what to resolve automatically and then do so. This lets the code be shared while each pathogen gets path resolution "for free" by adding a config schema. We want schemas for our configs anyway. Why not use this as another carrot and reduce copying at the same time?

I'll try these in the measles repo since we inadvertently broke nextstrain run compatibility there by switching to augur subsample.

[victorlin]

On applying resolve_config_path to all file paths in config at the start of the workflow:
This doesn't work for wildcard-based config. It's fine for now, but should be considered later.

Here is an example with measles, where wildcards are currently used: nextstrain/measles#75

[victorlin]

@jameshadfield said in nextstrain/measles#74 (comment):

For me the salient factor is that measles config values do not use wildcards, so wildcard expansion is not applicable. Were the configs to use wildcards within the values then we would have to use a different approach to resolving them.

# measles example - wildcards not in the (config) path value
subsample.genome.samples.global.include = "defaults/include_strains_genome.txt"
# avian-flu example of wildcards inside the (config) path value
colors.hardcoded = "config/{subtype}/colors_{subtype}.tsv:

subsample.genome.samples.global.include previously used wildcards but no longer does since it's not supported by augur subsample.

files.* still uses wildcards, but I'm proposing to replace them with the expanded versions in nextstrain/measles#75.

Ultimately, I don't think it's possible to support wildcards in config values if we wish to resolve filepaths at the start of the workflow (because Snakemake wildcards are only available at the rule level). To support a similar templated syntax, we could generate the config file dynamically.

[jameshadfield]

Ultimately, I don't think it's possible to support wildcards in config values if we wish to resolve filepaths at the start of the workflow (because Snakemake wildcards are only available at the rule level). To support a similar templated syntax, we could generate the config file dynamically.

Isn't generating the config file dynamically what we're proposing, and which you did in WNV? Or do you call #74 resolving not generating?

Could you not have a rule which iterated over all the known wildcard values and generated a fully expanded/resolved config like so:

# current config.yaml
files:
    reference: "measles_reference_{gene}.gb"

# dynamically generated config (in memory, or `results/configs.yaml` etc)
files:
  genome:
    reference: "measles_reference_genome.gb"
  N450:
    reference: "measles_reference_N450.gb"

The snakefile would look something like

configfile: ... # define / load configs as normal

# create an expanded config file by considering all wildcard combinations,
# (and resolving paths up-front, but that's not shown here)
# Load this into memory and write it out for debugging
# either overwrite the `config` variable or use a new one `c`
c = expand_configs()

...

rule foo:
    input:
        reference = c["files"][w.gene]["reference"]

(P.S. I'm keeping the config structure similar to the current status-quo for simplicity.)

[victorlin]

Isn't generating the config file dynamically what we're proposing, and which you did in WNV?

You're right that write_config is generating the config file for augur subsample, but the conversation around resolve_config_path is at a larger scope – Snakemake's config variable – which should transfer down into augur subsample config as long as the config variable is modified before calling write_config.

---

Or do you call #74 resolving not generating?

I think so, let me try to clarify and define clear terms. There are a few points in time where config values can be modified:

1. Before running Snakemake.
   • Example: using CUE to generate a phylogenetic/defaults/config.yaml.
   • We don't do this anywhere currently.
   • This is what I was referring to above with generating.
2. At Snakemake startup.
   • Example: modifying a value in config before it's used in rules (i.e. before Snakemake builds its DAG).
   • The measles workflow is now doing this with resolve_filepaths.
3. At Snakemake runtime.

   • Example: modifying a value from config while using it in a rule.

     rule foo:
         input:
             reference = resolve_config_path(config["files"]["reference"])

   • Most usage of resolve_config_path does this currently, and it's the only way to leverage Snakemake's built-in wildcards functionality.

   • This is not an option for augur subsample, which reads from the dump of config at startup.

I'm using the term resolving only for filepaths, such as in the names of the functions resolve_config_path and resolve_filepaths.

---

Could you not have a rule which iterated over all the known wildcard values and generated a fully expanded/resolved config like so:

Yes, that's possible, but I'm not sure about modifying the config structure (opposed to just values) at run time. That introduces a mapping from files.reference = "…{gene}…" to files.{gene}.reference which we'd have to keep track of both in the code and config schema (once we get to that). I don't have a working prototype, but I think something like this template to generate an expanded config.yaml for Snakemake would be more appropriate:

files:
  {gene}:
    reference: "measles_reference_{gene}.gb"