Add standard logging of config values

[joverlee521]

Context

With many layers of Snakemake configs provided via default configs and/or CLI options (--configfile/--config), it is helpful to have a standard way of logging the config values used for a workflow run.

Consensus in discussions below is to go with option 5 of the possible solutions, which has been implemented by @victorlin in WNV with discussion in nextstrain/WNV@d4831ac.

Previously listed possible solutions

1. This is done in the ncov workflow with a dump_config rule. Users must specify the target with the same configs as their workflow run to see the config output.

2. We could print out the config with each workflow run using the onstart handler. However, Snakemake docs note that these handlers are not triggered during dry-runs.

   onstart:
       import yaml, sys
       yaml.dump(config, sys.stdout, explicit_start = True, explicit_end = True)

3. We could print out the config with each workflow run using Snakemake's logger:

   import yaml
   from snakemake.logging import logger
   
   # Use default configuration values. Override with Snakemake's --configfile/--config options.
   configfile: "config/defaults.yaml"
   
   logger.info(f"Config is:\n{yaml.dump(config, explicit_start = True, explicit_end = True)}")

4. If the config output is too noisy, we can make it a debug level log that will only output if users provide the --verbose flag.

   import yaml
   from snakemake.logging import logger
   
   # Use default configuration values. Override with Snakemake's --configfile/--config options.
   configfile: "config/defaults.yaml"
   
   logger.debug(f"Config is:\n{yaml.dump(config, explicit_start = True, explicit_end = True)}")

5. Always dumping the final, fully-expanded, config-as-loaded to an output file, e.g. results/config.yaml. (comment)

TODOs

• Add config functions shared#61
• Write config to file #97
• vendor + include to use across pathogen workflows

[tsibley]

Option 3 is enticing because it means the actual config in use is always in build logs, so when something unexpectedly goes wrong you can inspect the config (without having to reconstruct it in a separate subsequent run).

[jameshadfield]

Option 3 Option 5 👍

[tsibley]

Option 5: Always dumping the final, fully-expanded, config-as-loaded to an output file, e.g. results/config.yaml.¹

This seems like the best option to me,

• It's written to disk, so not hard to find in all the logging output or lost to scrollback limits.

• It can be directly re-used as input config to reproduce/replicate the config for any given run (rather than having to be snipped out of logs, which can lead to errors).

and @jameshadfield and @joverlee521 +1'd this idea where I originally suggested it.

We could also do option 3 + 5 to get best of both worlds, which could be useful in rare cases when result files are lost (e.g. an out-of-disk-space Batch job), although 3 gets quite lengthy potentially.

[genehack]

+1 for option 5, with the "yes, but" addition of "don't call the output results/config.yaml, call it something like results/run_config.yaml or results/config_log.yaml. Consider including a timestamp in the name.

[jameshadfield]

Related thread where discussion about this is happening in parallel.

[joverlee521]

Chatted with @j23414 and @kimandrews on this, they both like the config output file that can easily be re-used for another workflow run.

[tsibley]

@genehack

don't call the output results/config.yaml, call it something like results/run_config.yaml or results/config_log.yaml.

Why? What makes the addition of run_ or _log important/useful?

(I can guess, but my guesses aren't compelling reasons (in my estimation), so I'm interested in your rationale.)

[genehack]

@genehack

don't call the output results/config.yaml, call it something like results/run_config.yaml or results/config_log.yaml.

Why? What makes the addition of run_ or _log important/useful?

If it's not named config.yaml, it can't get confused with defaults/config.yaml.

[victorlin]

I've implemented option 5 in an open PR on WNV: nextstrain/WNV#102

It works well, and will be useful for augur subsample.

[victorlin]

Update: option 5 was discussed and implemented in WNV.

import os
import sys
import yaml
from datetime import datetime

timestamp = datetime.now().astimezone().strftime("%Y-%m-%dT%H%M%S.%f")
path = f"results/run_configs/{timestamp}.yaml"

os.makedirs(os.path.dirname(path), exist_ok=True)

with open(path, 'w') as f:
    yaml.dump(config, f, sort_keys=False)

print(f"Saved current run config to {path!r}.", file=sys.stderr)

[victorlin]

This is helpful for debugging, but it opens the door to a new type of user error: accidentally editing the config dump file instead of the source config file. The natural debugging flow – run workflow → inspect dump → spot bad value → fix bad value – creates a trap, since the crucial step of switching back to the source config file is easily missed. I have already seen this happen in a screen sharing session.

Potential guardrails:

• Use a different file name (as discussed above)
• Add a prominent warning block at the top of the file (example)
• Mark the file as read-only (example)