-
-
Notifications
You must be signed in to change notification settings - Fork 20
Closed
Description
Given the workflow below:
#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: CommandLineTool
$namespaces:
cwltool: "http://commonwl.org/cwltool#"
doc: |
Trivial MPI test that prints the process IDs of each of the parallel
processes. Requires Python (but you have cwltool running, right?)
and an MPI implementation.
baseCommand: python
requirements:
cwltool:MPIRequirement:
processes: 2
arguments: [-c, 'import os; print(os.getpid())']
inputs: []
outputs:
pids:
type: stdout
MPIRequirement
is part of the cwltool extensions. This workflow also works with both cwltool and toil-cwl-runner given the right flags, ex:
$ export PATH=$PATH:/home/heaucques/Documents/toil/src/toil/test/cwl/mock_mpi/ # where fake_mpi_run.py exists
$ cwltool --enable-dev --enable-ext --mpi-config-file /home/heaucques/Documents/toil/src/toil/test/cwl/mock_mpi/fake_mpi.yml file:///home/heaucques/Documents/toil/src/toil/test/cwl/mpi_simple.cwl
INFO /home/heaucques/Documents/toil/venv3.12/bin/cwltool 3.1.20240508115724
INFO [job mpi_simple.cwl] /tmp/1ez0it5c$ fake_mpi_run.py \
--num \
2 \
--no-fail \
python \
-c \
'import os; print(os.getpid())' > /tmp/1ez0it5c/c83195d490fa84f03f0cb9b142c6f385d736bd94
INFO [job mpi_simple.cwl] completed success
{
"pids": {
"location": "file:///home/heaucques/Documents/toil/c83195d490fa84f03f0cb9b142c6f385d736bd94",
"basename": "c83195d490fa84f03f0cb9b142c6f385d736bd94",
"class": "File",
"checksum": "sha1$010210c4a6298629ae0653165e70c393bc831cad",
"size": 12,
"path": "/home/heaucques/Documents/toil/c83195d490fa84f03f0cb9b142c6f385d736bd94"
}
}INFO Final process status is success
However, running cwl-docker-extract mpi_simple.cwl
crashes at parsing as by default, these extensions are disabled.
$ cwl-docker-extract file:///home/heaucques/Documents/toil/src/toil/test/cwl/mpi_simple.cwl
Traceback (most recent call last):
File "/home/heaucques/Documents/toil/venv3.12/bin/cwl-docker-extract", line 8, in <module>
sys.exit(main())
^^^^^^
File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/cwl_utils/docker_extract.py", line 139, in main
run(arg_parser().parse_args(sys.argv[1:]))
File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/cwl_utils/docker_extract.py", line 55, in run
top = cwl.load_document_by_uri(args.input)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/cwl_utils/parser/__init__.py", line 183, in load_document_by_uri
return load_document_by_string(doc, real_uri, loadingOptions, id_, load_all)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/cwl_utils/parser/__init__.py", line 211, in load_document_by_string
return load_document_by_yaml(result, uri, loadingOptions, id_, load_all)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/cwl_utils/parser/__init__.py", line 227, in load_document_by_yaml
result = cwl_v1_0.load_document_by_yaml(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/cwl_utils/parser/cwl_v1_0.py", line 21054, in load_document_by_yaml
result, metadata = _document_load(
^^^^^^^^^^^^^^^
File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/cwl_utils/parser/cwl_v1_0.py", line 1038, in _document_load
loader.load(doc, baseuri, loadingOptions, docRoot=baseuri),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/cwl_utils/parser/cwl_v1_0.py", line 806, in load
raise ValidationException("", None, errors, "*")
schema_salad.exceptions.ValidationException: src/toil/test/cwl/mpi_simple.cwl:3:1: Object `mpi_simple.cwl` is not valid because:
src/toil/test/cwl/mpi_simple.cwl:14:1: the `requirements` field is not valid because:
Expected a <class 'NoneType'> but got list
cwl-docker-extract
does not appear to support extensions:
$ cwl-docker-extract --help
usage: cwl-docker-extract [-h] [--dir DIR] [-s] [--container-engine CONTAINER_ENGINE] [--force-download] input
Save container images specified in a CWL document (Workflow or CommandLineTool). For CWL Workflows, all steps will also be searched (recursively).
positional arguments:
input Input CWL document (CWL Workflow or CWL CommandLineTool)
options:
-h, --help show this help message and exit
--dir DIR Directory in which to save images
-s, --singularity Use singularity to pull the image
--container-engine CONTAINER_ENGINE
Specify which command to use to run OCI containers. Defaults to 'docker' (or 'singularity' if --singularity/-s is passed).
--force-download Force pulling a newer container.
Is there a way to enable support for these extensions to allow the parser to pass for these cases?
Metadata
Metadata
Assignees
Labels
No labels