Deprecate `meta` and `parameter_meta` in favor of inline documentation comments

[claymcleod]

Summary

Deprecate the meta and parameter_meta sections in favor of inline documentation using a new comment delimiter (//) and markdown conventions.

Motivation

It is unusual for documentation to be part of a language's syntax itself. Most languages treat documentation as structured comments that are associated with the declarations they describe (e.g., Javadoc, rustdoc, Python docstrings). WDL's current approach has several drawbacks:

• Duplication: Parameter names must appear in both the declaration and the parameter_meta section, violating DRY principles.
• Disconnection: Documentation is physically separated from what it documents, making it harder to keep in sync.
• Maintenance burden: Renaming a parameter requires updating two locations.
• Parsing overhead: Engines must parse and validate structured metadata that is primarily intended for human consumption.

This would also address the confusion raised in #637 about parameter_meta documenting both inputs and outputs, since inline documentation would be directly associated with each declaration.

Proposed Approach

Introduce a line comment delimiter (//) that can be used anywhere in the document. When placed immediately before a declaration, it serves as documentation for that declaration. Documentation would use markdown conventions:

// # Wizard Spells
//
// A collection of tasks for casting magical spells.
//
// ## Author
//
// Merlin the Magnificent <merlin@camelot.gov>
version 1.4

// A task that casts a spell.
//
// # Inputs
//
// - `spell_name`: The name of the spell to cast.
// - `power_level`: The power level of the spell. Defaults to `10`.
//
// # Outputs
//
// - `incantation`: The resulting incantation.
task cast_spell {
  input {
    // The name of the spell to cast.
    String spell_name

    // The power level of the spell. Defaults to `10`.
    Int power_level = 10
  }

  command <<<
    echo "Casting ~{spell_name} at power level ~{power_level}!"
  >>>

  output {
    // The resulting incantation.
    String incantation = read_string(stdout())
  }
}

Structured metadata could be expressed through markdown conventions (e.g., headings, bulleted lists) rather than special syntax.

If there is strong opposition to introducing a new comment delimiter, # could work as well. However, I find that block comments look rather strange with that syntax, so // is my suggestion.

Related

• documenting outputs in **parameter**_meta isn't clear #637

[peterhuene]

I would like to not extend the existing grammar to support this, if possible.

In Sprocket's tooling, at least, we already use ## to mean a "document preamble comment" (i.e. a comment on the purpose of the document itself) and #@ to mean "directive comment" (i.e. something to direct the WDL tooling itself; the meaning of which is WDL engine specific).

I would propose we instead formalize the above in the spec and add that ## as the way of documenting items within a WDL document as well.

## that comes before the version statement is the document comment. Preceding ## comments for an item are the item's document comment.

[peterhuene]

I do agree, though, that if we support markdown in doc comments, it does look slightly odd to have so many # next to each other:

## # Wizard Spells
##
## A collection of tasks for casting magical spells.
##
## ## Author
##
## Merlin the Magnificent <merlin@camelot.gov>

Personally, I can live with that though.

[a-frantz]

I've spent a fair amount of time over the years wrangling with WDL documentation, both in writing it (as a maintainer of the St. Jude Cloud workflows repo) and as a tool implementer (as the primary author of wdldoc and it's successor Sprocket doc). Clay has been talking about this proposal for a long time and I've been hesitant to throw my support behind the idea, but I've come finally come around and support WDL adopting doc comments.

Including documentation as part of the WDL language's syntax made sense when the tooling around WDL development was primitive; it encouraged authors to actually write documentation that could be read and used by downstream users of the WDL. Bioinformaticians don't have the best track record when it comes to software best practices and lack of proper documentation has often plagued genomics tooling. Including the meta and parameter_meta sections in the WDL specification encouraged WDL authors to write proper documentation from the get-go, especially in the days before WDL linters were in widespread use and documentation best practices could be enforced more programmatically.

However in today's WDL landscape, the tooling for WDL development is rapidly maturing and we can now rely on documentation enforcement via tooling.
I agree with everything in the motivation section above; doc comments will be more ergonomic and are the right direction for the language.

🚀 🚀 🚀

[patmagee]

I really like this idea and it has definitely come up multiple times in the past with different tools even building their own form of documentation syntax to alleviate the problem. Having documentation directly on the thing you are actually documenting is important and I think this will only lead to an increase in doc readability.

Also, I am +1 for keeping the ## syntax as it is going to be more familiar then adding a new comment syntax //. It reminds me of java using // vs /** which are just slight variants of each other.

If we decide to pursue this, I think we should launch the feature with a formal specification (Ie like java) OR a set of conventions (like python). I am personally more in favour of a formal specifications because I view conventions as weak commitment to a spec....

One thing I want to propose is not necessarily deprecating the meta and parameter_meta (although I think the later can be). The reality is, people have been free to use those sections as free form entry values for eons and have come up with a bunch of use cases beyond documentation that border on functional. I have seen UI's built using the meta sections, validation described in them etc. I would be more in favour of keeping them for now and evaluating what use cases would not easily be covered by the documentation.

[Serial-ATA]

I've implemented doc comments (using the ## syntax) in stjude-rust-labs/sprocket#551. So far, there's support for structs and enums.

Internally, the comments get broken into the meta.description and meta.help fields that sprocket doc was previously using. The first paragraph of the doc comment is mapped to meta.description, and is intended to be a short summary. All remaining paragraphs are joined with newlines and mapped to meta.help.

Documents can mix doc comments and meta/parameter_meta sections, with doc comments taking precedence (example).