The path is not passed correctly from one step to the next in a workflow.

[FarahZKhan]

Hi,
I have been facing this problem while executing the tool specification I wrote and added docker requirement in it for GATK tools. Before adding docker requirement, the workflow was working fine but after adding it following issues were faced.
Following are the links to the test data used (Assuming hg19.fa and supplementary reference files might be available otherwise will share those too):

Test data:
https://www.dropbox.com/sh/01vvou1p6i4w2ai/AADmVyVMSq0XQ2FrNfyo2MOfa?dl=0
Workflow and tool specification files: https://www.dropbox.com/sh/ukvz4vh0zg57ubw/AADDdP46SHkyDOCUgwxp_lz7a?dl=0

All the test files are included in test_data directory along with the variant files. The last step (realignTargetCreator) fails to execute and generates following error:

INFO  02:49:08,676 HelpFormatter - -------------------------------------------------------------------------------- 
INFO  02:49:08,678 HelpFormatter - The Genome Analysis Toolkit (GATK) v3.4-0-gf196186, Compiled 2015/09/27 12:19:32 
INFO  02:49:08,679 HelpFormatter - Copyright (c) 2010 The Broad Institute 
INFO  02:49:08,679 HelpFormatter - For support and documentation go to http://www.broadinstitute.org/gatk 
INFO  02:49:08,683 HelpFormatter - Program Args: -T RealignerTargetCreator -R /tmp/job925415211_tools/outputFiles/hg19.fa -I /tmp/job925415211_tmpOjuEb6/markDups.bam --known /tmp/job925415211_tools/dbsnp_138.vcf --known /tmp/job925415211_Ref_datasets/Mills_and_1000G_gold_st.vcf --known /tmp/job925415211_Ref_datasets/1000G_phase1.indels.hg19.vcf -o realignTargetCreator_output.intervals 
INFO  02:49:08,698 HelpFormatter - Executing as biodocker@578307327c0a on Linux 4.0.9-boot2docker amd64; OpenJDK 64-Bit Server VM 1.7.0_79-b14. 
INFO  02:49:08,699 HelpFormatter - Date/Time: 2015/12/09 02:49:08 
INFO  02:49:08,699 HelpFormatter - -------------------------------------------------------------------------------- 
INFO  02:49:08,700 HelpFormatter - -------------------------------------------------------------------------------- 
INFO  02:49:08,798 GenomeAnalysisEngine - Strictness is SILENT 
INFO  02:49:08,932 GenomeAnalysisEngine - Downsampling Settings: Method: BY_SAMPLE, Target Coverage: 1000 
INFO  02:49:08,941 SAMDataSource$SAMReaders - Initializing SAMRecords in serial 
##### ERROR ------------------------------------------------------------------------------------------
##### ERROR A USER ERROR has occurred (version 3.4-0-gf196186): 
##### ERROR
##### ERROR This means that one or more arguments or inputs in your command are incorrect.
##### ERROR The error message below tells you what is the problem.
##### ERROR
##### ERROR If the problem is an invalid argument, please check the online documentation guide
##### ERROR (or rerun your command with --help) to view allowable command-line arguments for this tool.
##### ERROR
##### ERROR Visit our website and forum for extensive documentation and answers to 
##### ERROR commonly asked questions http://www.broadinstitute.org/gatk
##### ERROR
##### ERROR Please do NOT post this error to the GATK forum unless you have really tried to fix it yourself.
##### ERROR
##### ERROR MESSAGE: Couldn't read file /tmp/job925415211_tmpOjuEb6/markDups.bam because java.io.FileNotFoundException: /tmp/job925415211_tmpOjuEb6/markDups.bam (Is a directory)

When the workflow is executed in --debug mode, I observed that all the other tools are using the original tmp dir location of the file suring input binding if used as a source for the next step whereas realignTargetCreator behaves differently.

I am not sure where the root of error might be and after discussing it on cwl gitter chat room I am generating this issue. According to my understanding, the mapping of path is not correct but I might be wrong. realignTargetCreator is using GATK docker that might not be detecting the file presence.

To add further, the tool works perfectly fine when run alone.
Any help will be much appreciated,
Thank you.

[mr-c]

Hello @FarahZKhan

I can't get your workflow to run. Here are steps I took. Can you share the command you run?

1. Downloaded all files from both DropBox links to a new directory
2. Create virtualenv and installed latests cwl-runner

virtualenv env
source env/bin/activate
pip install cwl-runner

1. Moved a couple files into the test data directory

mkdir test-data
mv Mills_and_1000G_gold_st.vcf test_data/
mv 1000G_phase1.indels.hg19.vcf test_data/

1. Attempted to execute the workflow

cwl-runner GATKtillMarkdups.cwl GATKtillMarkdupsTest.json 

[mr-c]

@portah pointed out that the file markDups.cwl is missing

[mr-c]

According to @tetron there was a transient problem with cwltool. I refreshed and got the same missing files response that @portah reported.

Now I get the following error:

(env)mcrusoe@mrcdev:~/test-cwltool-issue$ cwl-runner GATKtillMarkdupsTest.json GATKtillMarkdups.cwl 
/home/mcrusoe/test-cwltool-issue/env/bin/cwl-runner 1.0.20151209160516
Tool definition failed validation:
While checking field `reference`
  Field `path` contains undefined reference to `file:///home/mcrusoe/test-cwltool-issue/outputFiles/hg19.fa`
While checking field `PicardJarFile`
  Field `path` contains undefined reference to `file:///home/mcrusoe/test-cwltool-issue/test_data/broadinstitute-picard-8415062/dist/picard.jar`

[mr-c]

Why is the picard.jar an input file? We'll have to address that later.

(env)mcrusoe@mrcdev:~/test-cwltool-issue$ cd test_data
(env)mcrusoe@mrcdev:~/test-cwltool-issue/test_data$ unzip ../broadinstitute-picard-8415062.zip 
(env)mcrusoe@mrcdev:~/test-cwltool-issue/test_data$ cd ..
(env)mcrusoe@mrcdev:~/test-cwltool-issue$ mkdir outputFiles
(env)mcrusoe@mrcdev:~/test-cwltool-issue$ touch outputFiles/hg19.fa
(env)mcrusoe@mrcdev:~/test-cwltool-issue$ cwl-runner GATKtillMarkdupsTest.json GATKtillMarkdups.cwl                                         
/home/mcrusoe/test-cwltool-issue/env/bin/cwl-runner 1.0.20151209160516
Tool definition failed validation:
Validation error in object file:///home/mcrusoe/test-cwltool-issue/GATKtillMarkdupsTest.json
  Could not validate as `CommandLineTool` because
    missing required field `inputs`
    missing required field `outputs`
    missing required field `class`
    missing required field `baseCommand`
    could not validate field `RevReadFile` because it is not recognized and strict is True, valid fields are: id, inputs, outputs, requireme
nts, hints, label, description, cwlVersion, class, baseCommand, arguments, stdin, stdout, successCodes, temporaryFailCodes, permanentFailCod
es
    could not validate field `outputfile_realignTarget` because it is not recognized and strict is True, valid fields are: id, inputs, outpu
ts, requirements, hints, label, description, cwlVersion, class, baseCommand, arguments, stdin, stdout, successCodes, temporaryFailCodes, per
manentFailCodes
    could not validate field `reference` because it is not recognized and strict is True, valid fields are: id, inputs, outputs, requirement
s, hints, label, description, cwlVersion, class, baseCommand, arguments, stdin, stdout, successCodes, temporaryFailCodes, permanentFailCodes
    could not validate field `PicardJarFile` because it is not recognized and strict is True, valid fields are: id, inputs, outputs, require
ments, hints, label, description, cwlVersion, class, baseCommand, arguments, stdin, stdout, successCodes, temporaryFailCodes, permanentFailC
odes
    could not validate field `metricsFile` because it is not recognized and strict is True, valid fields are: id, inputs, outputs, requireme
nts, hints, label, description, cwlVersion, class, baseCommand, arguments, stdin, stdout, successCodes, temporaryFailCodes, permanentFailCod
es
    could not validate field `outputFileName` because it is not recognized and strict is True, valid fields are: id, inputs, outputs, requir
ements, hints, label, description, cwlVersion, class, baseCommand, arguments, stdin, stdout, successCodes, temporaryFailCodes, permanentFail
Codes
    could not validate field `outputFileName_mergedSam` because it is not recognized and strict is True, valid fields are: id, inputs, outpu
ts, requirements, hints, label, description, cwlVersion, class, baseCommand, arguments, stdin, stdout, successCodes, temporaryFailCodes, per
manentFailCodes
    could not validate field `readSorted` because it is not recognized and strict is True, valid fields are: id, inputs, outputs, requiremen
ts, hints, label, description, cwlVersion, class, baseCommand, arguments, stdin, stdout, successCodes, temporaryFailCodes, permanentFailCode
s
    could not validate field `FwdReadFile` because it is not recognized and strict is True, valid fields are: id, inputs, outputs, requireme
nts, hints, label, description, cwlVersion, class, baseCommand, arguments, stdin, stdout, successCodes, temporaryFailCodes, permanentFailCod
es
    could not validate field `outputFileName_sortSam` because it is not recognized and strict is True, valid fields are: id, inputs, outputs
, requirements, hints, label, description, cwlVersion, class, baseCommand, arguments, stdin, stdout, successCodes, temporaryFailCodes, perma
nentFailCodes
    could not validate field `known` because it is not recognized and strict is True, valid fields are: id, inputs, outputs, requirements, h
ints, label, description, cwlVersion, class, baseCommand, arguments, stdin, stdout, successCodes, temporaryFailCodes, permanentFailCodes
    could not validate field `removeDuplicates` because it is not recognized and strict is True, valid fields are: id, inputs, outputs, requ
irements, hints, label, description, cwlVersion, class, baseCommand, arguments, stdin, stdout, successCodes, temporaryFailCodes, permanentFa
ilCodes
    could not validate field `createIndex` because it is not recognized and strict is True, valid fields are: id, inputs, outputs, requireme
nts, hints, label, description, cwlVersion, class, baseCommand, arguments, stdin, stdout, successCodes, temporaryFailCodes, permanentFailCod
es
    could not validate field `output_name` because it is not recognized and strict is True, valid fields are: id, inputs, outputs, requireme
nts, hints, label, description, cwlVersion, class, baseCommand, arguments, stdin, stdout, successCodes, temporaryFailCodes, permanentFailCod
es
[...]

[mr-c]

@portah caught my switching the workflow file for the job document; that brings up more missing files errors.

Correcting them:

(env)mcrusoe@mrcdev:~/test-cwltool-issue$ cp /home/mcrusoe/common-workflow-language/draft-2/draft-2/node-engine.cwl .
(env)mcrusoe@mrcdev:~/test-cwltool-issue$ cp /home/mcrusoe/workflows/tools/envvar-global.cwl .
(env)mcrusoe@mrcdev:~/test-cwltool-issue$ cwl-runner GATKtillMarkdups.cwl GATKtillMarkdupsTest.json

Leads to a huge error dump pointing out another missing file: bwa-docker.cwl

[mr-c]

Okay, I copied that from the workflows repository as well.

(env)mcrusoe@mrcdev:~/test-cwltool-issue$ cp /home/mcrusoe/workflows/tools/bwa-docker.cwl .

But I am still missing the markDups.cwl file.

[mr-c]

Substituting tools/markDuplicates.cwl from the workflows repository for markDups.cwl and copying in a few other files shows that they are not valid substitutes for each other

Parameter 'createIndex' of inputs in workflow step file:///home/mcrusoe/test-cwltool-issue/GATKtillMarkdups.cwl#markDups does not correspond to parameter in file:///home/mcrusoe/test-cwltool-issue/markDups.cwl

Here is my recommendation: Create a new branch or fork of https://github.com/common-workflow-language/workflows and add all of the new workflow and tool specification files to the appropriate directories. Make sure you can duplicate your error under your new setup and send us a pull request from your new branch or fork and mark it [WIP].

That way we can comment in-line with your changes and it will be a lot easier to assist each other.

[FarahZKhan]

@mr-c Before making the picard tools available in a workflow, I tried adding docker requirement to lessen the hassle for copying the picard folder with jar file etc. There is an error I encountered explained here:
common-workflow-library/legacy#41
Can you please look into it for a moment. It does not require many input files (unlike realigntargetCreator) and can be executed with one input file (provided).

[mr-c]

The pull request common-workflow-library/legacy#40 solves the issue with markDuplicates.cwl.

Which hg19.fa file are you using?

[FarahZKhan]

@mr-c I just solved the issues for which the thread was started.
For reference: I added docker for picard tools as well and that worked in this case as every tool in my workflow has docker requirement now (GATK docker, picard docker and bwa docker). Thank you so much for your help. I will add all the files (tool files and workflow along with the test files) tomorrow in my fork from c-w-l/workflows and pull request the changes.
Thank you again for looking into it.

[mr-c]

Glad to hear it!

On Thu, Dec 10, 2015 at 1:37 PM FarahZKhan notifications@github.com wrote:

@mr-c https://github.com/mr-c I just solved the issues for which the
thread was started.
For reference: I added docker for picard tools as well and that worked in
this case as every tool in my workflow has docker requirement now (GATK
docker, picard docker and bwa docker). Thank you so much for your help. I
will add all the files (tool files and workflow along with the test files)
tomorrow in my fork from c-w-l/workflows and pull request the changes.
Thank you again for looking into it.

—
Reply to this email directly or view it on GitHub
#28 (comment)
.