Skip to content

Parser does not support cwltool extensions #309

@stxue1

Description

@stxue1

Given the workflow below:

#!/usr/bin/env cwl-runner

cwlVersion: v1.0
class: CommandLineTool
$namespaces:
  cwltool: "http://commonwl.org/cwltool#"

doc: |
  Trivial MPI test that prints the process IDs of each of the parallel
  processes. Requires Python (but you have cwltool running, right?)
  and an MPI implementation.

baseCommand: python
requirements:
  cwltool:MPIRequirement:
    processes: 2
arguments: [-c, 'import os; print(os.getpid())']
inputs: []
outputs:
  pids:
    type: stdout

MPIRequirement is part of the cwltool extensions. This workflow also works with both cwltool and toil-cwl-runner given the right flags, ex:

$ export PATH=$PATH:/home/heaucques/Documents/toil/src/toil/test/cwl/mock_mpi/ # where fake_mpi_run.py exists
$ cwltool --enable-dev --enable-ext --mpi-config-file /home/heaucques/Documents/toil/src/toil/test/cwl/mock_mpi/fake_mpi.yml file:///home/heaucques/Documents/toil/src/toil/test/cwl/mpi_simple.cwl
INFO /home/heaucques/Documents/toil/venv3.12/bin/cwltool 3.1.20240508115724
INFO [job mpi_simple.cwl] /tmp/1ez0it5c$ fake_mpi_run.py \
    --num \
    2 \
    --no-fail \
    python \
    -c \
    'import os; print(os.getpid())' > /tmp/1ez0it5c/c83195d490fa84f03f0cb9b142c6f385d736bd94
INFO [job mpi_simple.cwl] completed success
{
    "pids": {
        "location": "file:///home/heaucques/Documents/toil/c83195d490fa84f03f0cb9b142c6f385d736bd94",
        "basename": "c83195d490fa84f03f0cb9b142c6f385d736bd94",
        "class": "File",
        "checksum": "sha1$010210c4a6298629ae0653165e70c393bc831cad",
        "size": 12,
        "path": "/home/heaucques/Documents/toil/c83195d490fa84f03f0cb9b142c6f385d736bd94"
    }
}INFO Final process status is success

And the toil equivalent

However, running cwl-docker-extract mpi_simple.cwl crashes at parsing as by default, these extensions are disabled.

$ cwl-docker-extract file:///home/heaucques/Documents/toil/src/toil/test/cwl/mpi_simple.cwl
Traceback (most recent call last):
  File "/home/heaucques/Documents/toil/venv3.12/bin/cwl-docker-extract", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/cwl_utils/docker_extract.py", line 139, in main
    run(arg_parser().parse_args(sys.argv[1:]))
  File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/cwl_utils/docker_extract.py", line 55, in run
    top = cwl.load_document_by_uri(args.input)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/cwl_utils/parser/__init__.py", line 183, in load_document_by_uri
    return load_document_by_string(doc, real_uri, loadingOptions, id_, load_all)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/cwl_utils/parser/__init__.py", line 211, in load_document_by_string
    return load_document_by_yaml(result, uri, loadingOptions, id_, load_all)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/cwl_utils/parser/__init__.py", line 227, in load_document_by_yaml
    result = cwl_v1_0.load_document_by_yaml(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/cwl_utils/parser/cwl_v1_0.py", line 21054, in load_document_by_yaml
    result, metadata = _document_load(
                       ^^^^^^^^^^^^^^^
  File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/cwl_utils/parser/cwl_v1_0.py", line 1038, in _document_load
    loader.load(doc, baseuri, loadingOptions, docRoot=baseuri),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/cwl_utils/parser/cwl_v1_0.py", line 806, in load
    raise ValidationException("", None, errors, "*")
schema_salad.exceptions.ValidationException: src/toil/test/cwl/mpi_simple.cwl:3:1:   Object `mpi_simple.cwl` is not valid because:
src/toil/test/cwl/mpi_simple.cwl:14:1:   the `requirements` field is not valid because:
                                           Expected a <class 'NoneType'> but got list

cwl-docker-extract does not appear to support extensions:

$ cwl-docker-extract --help
usage: cwl-docker-extract [-h] [--dir DIR] [-s] [--container-engine CONTAINER_ENGINE] [--force-download] input

Save container images specified in a CWL document (Workflow or CommandLineTool). For CWL Workflows, all steps will also be searched (recursively).

positional arguments:
  input                 Input CWL document (CWL Workflow or CWL CommandLineTool)

options:
  -h, --help            show this help message and exit
  --dir DIR             Directory in which to save images
  -s, --singularity     Use singularity to pull the image
  --container-engine CONTAINER_ENGINE
                        Specify which command to use to run OCI containers. Defaults to 'docker' (or 'singularity' if --singularity/-s is passed).
  --force-download      Force pulling a newer container.

Is there a way to enable support for these extensions to allow the parser to pass for these cases?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions