best practices guide

.vitepress/config.mts

@@ -81,6 +81,7 @@ export default defineConfig({
         text: "Getting Started",
         items: [
           { text: "Quickstart", link: "/getting-started/quickstart" },
+          { text: "Best Practices", link: "/getting-started/best-practices" },
           { text: "Ecosystem", link: "/getting-started/ecosystem" },
           { text: "Getting help", link: "/getting-started/getting-help" },
           { text: "Contributing", link: "/getting-started/contributing" },

getting-started/best-practices.md

@@ -0,0 +1,30 @@
+# WDL Best Practices
+
+- All tasks with multiple commands (including any pipes (`|`)) should have `set -euo pipefail` before any other commands.
+  - Tasks without multiple commands or pipes can omit this.
+  - These options will cause common classes of bugs in Bash scripts to fail immediately and loudly, instead of causing silent or subtle bugs in your task behavior.
+- All tasks should run in a persistently versioned container.
+  - e.g. do not use `latest` tags for Docker images.
+  - This helps ensure reproducibility across time and environments.
+- Check all assumptions made about workflow inputs before beginning long running executions.
+  - Common examples of assumptions that should be checked:
+    - valid `String` choice (for WDL 1.3 and later, an `enum` should be used in place of `String`s with a fixed set of valid options)
+    - mutually exclusive parameters
+    - missing optional file for selected parameters
+    - filename extensions
+  - Use `after` clauses in workflows to ensure that all these assumptions are valid before beginning tasks with heavy computation.
+- If the _contents_ of a `File` are not read or do not need to be localized for a task, try to coerce the `File` variable to a `Boolean` (with `defined()`) or a `String` (with `basename()`) to avoid unnecessary disk space usage and networking.
+- All requirement values are overridable at runtime. However, tasks should have easily configurable memory and disk space allocations.
+  - Often, tasks have a dynamic calculation for resource requirements based on input sizes. Users of a WDL should have an easy way to fine tune this calculation.
+  - This may mean incorporating an `Int` or `Float` in the inputs of the task that is applied to the dynamic calculation.
+  - For WDL 1.3 and later, WDL authors can change resource requirements between retry attempts. This enables mitigation of errors relating to resources limits, but users may inadvertantly disable these mitigations by introducing runtime overrides. WDL authors should expose resource fine tuning via the input section and incorporate those user values in any dynamic calculations to prevent runtime locking.
+- Tasks which assume a file and any accessory files (e.g. a BAM and a BAI) have specific extensions and/or are in the same directory should *always* create symlinks from the mounted inputs to the work directory of the task
+  - This is because individual `File` types are not guarenteed to be in the same mounted directory.
+  - The `command` may include something like: `ln -s "~{<input name>}" "./<expected name>"`
+- Tasks should `rm` any temporary or intermediate files created in the work directory (including symlinks).
+  - This helps reduce disk bloat from keeping unnecessary files around.
+  - This is especially important for any large or uncompressed files, such as reference FASTAs or databases.
+- Most tasks should have a default `maxRetries` of 1.
+  - This is because many WDL backends are prone to intermittent failures that can be recovered from with a retry.
+  - Certain tasks are especially prone to intermittent failure (often if any networking is involved) and can have a higher default `maxRetries`.
+  - Certain tasks with potentially high compute costs in cloud environments may default to `0`. This should be used in combination with call caching to aid rerunning while minimizing costs.