Workflows as programs

[genehack]

This is a meta-issue for tracking work around "workflows as programs". Extracted from Q3 planning meeting notes by @genehack. Fleshed out by @tsibley.

Background

• 5 Nov 2024 lab meeting slides
• 9 Sept 2021 lab meeting slides
• Lots more in my personal notes and written up in Slack

Tooling

• Grab bag of bug fixes and improvements cli#419
• Workflows-as-programs: nextstrain run cli#407
• entrypoint-aws-batch: Keep ../ path parts in ZIP archive members during extraction docker-base#241
• Nextstrain CLI and nextstrain.org: endpoints for pathogen resolution
  • Endpoints to resolve pathogen repo versions for the CLI nextstrain.org#1153
  • pathogens: Resolve versions via a new nextstrain.org API cli#431
  • Send meaningful User-Agent with all requests cli#432
  • Record Plausible analytics events for endpoints intended for Nextstrain CLI nextstrain.org#1161
• Pathogen repo usage flow cli#504
• Document what "compatible with nextstrain run" means pathogen-repo-guide#102
• Lay down the way for pathogen documentation with an example or two
  • Per-pathogen docs prototyping zika#90
• Nextstrain CLI: commands for accessing version-specific pathogen docs? (see also, see also)
• nextstrain run: Define available workflows in the nextstrain-pathogen.yaml cli#454
• Add schema for nextstrain-pathogen.yaml cli#466
• Write up and document design notes with additions from my Nov 2024 lab meeting
• Nextstrain CLI: better UX for orgs with their own GitHub repos, e.g. my-org/my-pathogen (see also)
  • nextstrain run: third-party pathogen repo support cli#449
• nextstrain run: unmanaged pathogen support? cli#420
• Nextstrain CLI: roll back setups/updates that fail (e.g. checks fail)? (see also, see also)
• Nextstrain CLI: removal of set up pathogens (and runtimes); nextstrain teardown?
• Nextstrain CLI: auto-cleanup of old pathogens after upgrade?
• Nextstrain CLI: check-setup support for pathogens?
• Bundle runtimes with pathogen repos cli#448

Pathogens

• Workflows-as-programs support measles#55
• Allow running workflow from outside of repo avian-flu#103
• Multiple inputs / overriding inputs avian-flu#112
• Update workflows to support nextstrain run zika#85
• Add support for workflows as programs mumps#36
• Update workflows for nextstrain run pathogen-repo-guide#93
• Support nextstrain run (with augur subsample) WNV#105
• Grab bag of various fixups nipah#17
• mpox
  • multiple Snakefile version
  • alternative using nextstrain-pathogen.yaml to define workflow files
• work needed to allow users to run analyses separate to pathogen workflows pathogen-repo-guide#74
• Standardized composable configs pathogen-repo-guide#79

Testing

• Partner with INRB, WA DOH, and/or external ncov users to beta-test nextstrain run and gather feedback

[huddlej]

One question that occurred to me during @tsibley's lab meeting on this topic today is: do we want to guarantee that users can run our Nextstrain workflows with snakemake instead of with nextstrain?

I was thinking about how the workflow authors need to define shared functions (like shared/functions.smk from the measles PR) within and across pathogen repos and how nextstrain run could always inject shared logic into the workflows or provide a runtime environment that guarantees those functions will be available. But that kind of approach would break the workflows with vanilla snakemake. Even if code injection is a terrible idea, it made me wonder about our long-term guarantees that Nextstrain workflows will always work with Snakemake...

[genehack]

Copying this (and mildly cleaning it up) from a comment I sent to @tsibley on Slack after his lab meeting:

• I suspect for many current “authors” this is going to feel like more work for little-to-no return; I think one way to assuage some of that would be to try as hard as possible to make the requirements of extra “stuff” in config.yaml and the snakemake rules as minimal as possible — for example, instead of requiring helpers to do path translations, what if nextstrain run pre-parsed the workflow config and the snakerules and automatically applied those transformations? (or attempted to and introspected the filesystem with fallbacks, etc.)

• I wonder if something like the mpox clade I builds, or something WA DOH is working on, wouldn’t make for a better initial implementation target — a lot (most? all?) of the benefits to doing this will be to the “external consumer” part of the experience; would make sense to do prototype runs on something where we can get feedback from somebody in that external pool as quickly as possible

[tsibley]

To @huddlej and @genehack's suggestions re: adding more "magic": I think it'd be more complicating and mystifying for authors (and readers) if we use techniques like code injection or config/workflow rewriting or similar. I don't want to rule out using such "magic" techniques in the future—they may prove to be the best way to meet our eventual goals/expectations—but for now I think it would be way more complex and fragile than making clear guidelines that all file paths from the config that have defaults should be wrapped in a function call.

As @huddlej notes, adding magic also ruins the ability to run workflows directly with Snakemake, and at least for now, I think we want to preserve that ability. (Though again, I wouldn't rule it out for the future.)

To @genehack's explicit concern (and @huddlej's implicit concern, if I'm reading correct) around extra stuff to do in authoring, the extra burden for authoring new workflows doesn't feel high to me. The function-wrapped-config-values are going to be mostly copied/cribbed from existing workflows and/or our pathogen-repo-guide template. Our Snakemake best practices will include appropriate guidance as reference material.

While this does create new work to update existing workflows, that is paid once and is straightforward work to do. We also do not need to do it all at once; only as we want a pathogen's support for nextstrain run.

[tsibley]

For @genehack's thoughts around initial implementations, I agree that we should consider that and try to pick from existing partners. I was thinking of INRB (see also) in addition to WA DOH. There are also many external users of ncov that could likely benefit from this work and potentially be willing and able to provide valuable feedback if we make them aware of it and how to try it out (e.g. perhaps a post on the discussion forum).

Measles was merely a convenient example for this initial "how do we actually do this" phase (the same way zika or zika-tutorial has been that in the past, though much less so in recent years).