Skip to content

simplify default tests + add conda/singularity #736

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
maxulysse merged 13 commits into from
Sep 8, 2022
Merged

simplify default tests + add conda/singularity #736

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
59b60bc
simplify default tests + add conda/singularity
maxulysse Sep 7, 2022
34c7eff
update CHANGELOG
maxulysse Sep 7, 2022
d0375ce
fix conda test
maxulysse Sep 7, 2022
b162417
fix scope for fixOwnership + sort items
maxulysse Sep 7, 2022
dd3b5cb
fix indent + sort
maxulysse Sep 7, 2022
05333ee
add timeout for conda
maxulysse Sep 7, 2022
ff38247
better comments
maxulysse Sep 7, 2022
5b307f9
enhance installation
maxulysse Sep 7, 2022
e3e6cdc
exclude latest version for conda
maxulysse Sep 7, 2022
2d0e91f
Merge remote-tracking branch 'upstream/dev' into dev_complete_CI
maxulysse Sep 8, 2022
dd5faeb
remove docker.fixOwnership
maxulysse Sep 8, 2022
3856599
prettier again
maxulysse Sep 8, 2022
349e1d6
fix checksum
maxulysse Sep 8, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 21 additions & 2 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ jobs:
- "latest-everything"
test:
- "default"
profile: ["docker"]
profile: ["docker", "singularity", "conda"]
steps:
- name: Check out pipeline code
uses: actions/checkout@v2
Expand All @@ -44,10 +44,28 @@ jobs:
- name: Install dependencies
run: python -m pip install --upgrade pip pytest-workflow

- name: Set up Singularity
if: matrix.profile == 'singularity'
uses: eWaterCycle/setup-singularity@v5
with:
singularity-version: 3.7.1

- name: Set up miniconda
if: matrix.profile == 'conda'
uses: conda-incubator/setup-miniconda@v2
with:
auto-update-conda: true
channels: conda-forge,bioconda,defaults
python-version: ${{ matrix.python-version }}

- name: Conda clean
if: matrix.profile == 'conda'
run: conda clean -a

- name: Run pipeline with tests settings
uses: Wandalen/wretry.action@v1.0.11
with:
command: PROFILE=${{ matrix.profile }} pytest --tag ${{ matrix.test }} --symlink --kwdof --git-aware --color=yes
command: TMPDIR=~ PROFILE=${{ matrix.profile }} pytest --tag ${{ matrix.test }} --symlink --kwdof --git-aware --color=yes
attempt_limit: 3

- name: Output log on failure
Expand Down Expand Up @@ -91,6 +109,7 @@ jobs:
- "freebayes"
- "gatk4_spark"
- "haplotypecaller"
- "intervals"
- "manta"
- "markduplicates"
- "mpileup"
Expand Down
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- [#730](https://github.com/nf-core/sarek/pull/730) - Reduce number of tests
- [#731](https://github.com/nf-core/sarek/pull/731) - Run the somatic test as default on `-profile test_full`, the germline can be tested with `-profile test_full_somatic`
- [#733](https://github.com/nf-core/sarek/pull/733) - Add description for params.cf_chrom_len
- [#736](https://github.com/nf-core/sarek/pull/736) - More extensive CI for default test

### Fixed

Expand Down
40 changes: 21 additions & 19 deletions conf/test.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -256,26 +256,28 @@ process {
// ENABLE CI containers for testing
if (System.getenv('PROFILE')) {
if ("$PROFILE" == "conda") {
params.enable_conda = true
docker.enabled = false
singularity.enabled = false
podman.enabled = false
shifter.enabled = false
charliecloud.enabled = false
conda.createTimeout = "120 min"
params.enable_conda = true
charliecloud.enabled = false
docker.enabled = false
podman.enabled = false
shifter.enabled = false
singularity.enabled = false
} else if ("$PROFILE" == "docker") {
docker.enabled = true
docker.userEmulation = { params.use_gatk_spark ? false : true }.call()
singularity.enabled = false
podman.enabled = false
shifter.enabled = false
charliecloud.enabled = false
fixOwnership = true
Copy link
Member Author

@maxulysse maxulysse Sep 8, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decided to remove that instead of fixing it, as it was causing the strelkabp test to fail...

Copy link
Contributor

@FriederikeHanssen FriederikeHanssen Sep 8, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

only now?

Copy link
Member Author

@maxulysse maxulysse Sep 8, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes because it wasn't working, it should have been:
docker.fixOwnership = true
But when I do that, then the test fails.
So as it wasn't really working to start with, decide to remove it instead of fixing it

Copy link
Contributor

@FriederikeHanssen FriederikeHanssen Sep 8, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

docker.enabled = true
docker.fixOwnership = true
docker.userEmulation = { params.use_gatk_spark ? false : true }.call()
charliecloud.enabled = false
podman.enabled = false
shifter.enabled = false
singularity.enabled = false
} else if ("$PROFILE" == "singularity") {
singularity.enabled = true
singularity.autoMounts = true
docker.enabled = false
podman.enabled = false
shifter.enabled = false
charliecloud.enabled = false
params.singularity_pull_docker_container = false
singularity.autoMounts = true
singularity.enabled = true
charliecloud.enabled = false
docker.enabled = false
podman.enabled = false
shifter.enabled = false
}
}
32 changes: 16 additions & 16 deletions nextflow.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,63 +143,63 @@ profiles {
debug { process.beforeScript = 'echo $HOSTNAME' }
conda {
params.enable_conda = true
charliecloud.enabled = false
docker.enabled = false
singularity.enabled = false
podman.enabled = false
shifter.enabled = false
charliecloud.enabled = false
singularity.enabled = false
}
mamba {
params.enable_conda = true
conda.useMamba = true
params.enable_conda = true
charliecloud.enabled = false
docker.enabled = false
singularity.enabled = false
podman.enabled = false
shifter.enabled = false
charliecloud.enabled = false
singularity.enabled = false
}
docker {
docker.enabled = true
docker.fixOwnership = true
docker.userEmulation = { params.use_gatk_spark ? false : true }.call()
singularity.enabled = false
charliecloud.enabled = false
podman.enabled = false
shifter.enabled = false
charliecloud.enabled = false
fixOwnership = true
singularity.enabled = false
}
singularity {
singularity.enabled = true
singularity.autoMounts = true
singularity.enabled = true
charliecloud.enabled = false
docker.enabled = false
podman.enabled = false
shifter.enabled = false
charliecloud.enabled = false
}
podman {
podman.enabled = true
charliecloud.enabled = false
docker.enabled = false
singularity.enabled = false
shifter.enabled = false
charliecloud.enabled = false
singularity.enabled = false
}
shifter {
shifter.enabled = true
charliecloud.enabled = false
docker.enabled = false
singularity.enabled = false
podman.enabled = false
charliecloud.enabled = false
singularity.enabled = false
}
charliecloud {
charliecloud.enabled = true
docker.enabled = false
singularity.enabled = false
podman.enabled = false
shifter.enabled = false
singularity.enabled = false
}
gitpod {
executor.name = 'local'
executor.cpus = 16
executor.memory = 60.GB
executor.name = 'local'
}
test { includeConfig 'conf/test.config' }
test_full { includeConfig 'conf/test_full.config' }
Expand Down
63 changes: 3 additions & 60 deletions tests/test_default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
- path: results/preprocessing/recalibrated/test/test.recal.cram.crai
# binary changes md5sums on reruns.
- path: results/reports/bcftools/strelka/test/test.strelka.variants.bcftools_stats.txt
md5sum: 28d99fd4fa63e24e6805f1d12c3b71d3
# conda changes md5sums for test.
- path: results/reports/fastqc/test-test_L1
- path: results/reports/markduplicates/test/test.md.metrics
contains: ["test 8547 767 84 523391 3882 0 0 0.385081", "1.0 767 767"]
Expand All @@ -49,9 +49,9 @@
- path: results/reports/mosdepth/test/test.recal.regions.bed.gz.csi
md5sum: b1c2a861f64e20a94108a6de3b76c582
- path: results/reports/samtools/test/test.md.cram.stats
md5sum: dcf70bbcfb92e01027978f28d2035d78
# conda changes md5sums for test.
- path: results/reports/samtools/test/test.recal.cram.stats
md5sum: 5528d952f5dc74a39f28e27165bf96be
# conda changes md5sums for test.
- path: results/reports/vcftools/strelka/test/test.strelka.variants.FILTER.summary
md5sum: dd87f507da7de20d5318841af312493b
- path: results/reports/vcftools/strelka/test/test.strelka.variants.TsTv.count
Expand All @@ -67,60 +67,3 @@
- path: results/variant_calling/strelka/test/test.strelka.variants.vcf.gz.tbi
- path: results/strelka
should_exist: false
- name: Run default pipeline without intervals
command: nextflow run main.nf -profile test,no_intervals
tags:
- default
- no_intervals
- preprocessing
files:
- path: results/csv/markduplicates.csv
md5sum: 0d6120bb99e92f6810343270711ca53e
- path: results/csv/markduplicates_no_table.csv
md5sum: 2a2d3d4842befd4def39156463859ee3
- path: results/csv/recalibrated.csv
md5sum: 42628ec994c16f565e5407b40a9c1ac3
- path: results/multiqc
- path: results/no_intervals.bed
md5sum: f3dac01ea66b95fe477446fde2d31489
- path: results/no_intervals.bed.gz
md5sum: f3dac01ea66b95fe477446fde2d31489
- path: results/no_intervals.bed.gz.tbi
md5sum: f3dac01ea66b95fe477446fde2d31489
- path: results/preprocessing/markduplicates/test/test.md.cram
# binary changes md5sums on reruns.
- path: results/preprocessing/markduplicates/test/test.md.cram.crai
# binary changes md5sums on reruns.
- path: results/preprocessing/recal_table/test/test.recal.table
md5sum: 4ac774bf5f1157e77426fd82f5ac0fbe
- path: results/preprocessing/recalibrated/test/test.recal.cram
# binary changes md5sums on reruns.
- path: results/preprocessing/recalibrated/test/test.recal.cram.crai
# binary changes md5sums on reruns.
- path: results/reports/fastqc/test-test_L1
- path: results/reports/markduplicates/test/test.md.metrics
contains: ["test 8547 767 84 523391 3882 0 0 0.385081", "1.0 767 767"]
- path: results/reports/mosdepth/test/test.md.mosdepth.global.dist.txt
md5sum: 76fa71922a3f748e507c2364c531dfcb
- path: results/reports/mosdepth/test/test.md.mosdepth.region.dist.txt
md5sum: abc5df85e302b79985627888870882da
- path: results/reports/mosdepth/test/test.md.mosdepth.summary.txt
md5sum: d536456436eb275159b8c6af83213d80
- path: results/reports/mosdepth/test/test.md.regions.bed.gz
md5sum: 38fe39894abe62e38f8ac214cba64f2b
- path: results/reports/mosdepth/test/test.md.regions.bed.gz.csi
md5sum: b1c2a861f64e20a94108a6de3b76c582
- path: results/reports/mosdepth/test/test.recal.mosdepth.global.dist.txt
md5sum: 76fa71922a3f748e507c2364c531dfcb
- path: results/reports/mosdepth/test/test.recal.mosdepth.region.dist.txt
md5sum: abc5df85e302b79985627888870882da
- path: results/reports/mosdepth/test/test.recal.mosdepth.summary.txt
md5sum: d536456436eb275159b8c6af83213d80
- path: results/reports/mosdepth/test/test.recal.regions.bed.gz
md5sum: 38fe39894abe62e38f8ac214cba64f2b
- path: results/reports/mosdepth/test/test.recal.regions.bed.gz.csi
md5sum: b1c2a861f64e20a94108a6de3b76c582
- path: results/reports/samtools/test/test.md.cram.stats
md5sum: dcf70bbcfb92e01027978f28d2035d78
- path: results/reports/samtools/test/test.recal.cram.stats
md5sum: f7005dd4646700e107bfc00e36c432fe
105 changes: 105 additions & 0 deletions tests/test_intervals.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
- name: Run intervals false pipeline
command: nextflow run main.nf -profile test --intervals false
tags:
- intervals
- intervals_false
- preprocessing
files:
- path: results/csv/markduplicates.csv
md5sum: 0d6120bb99e92f6810343270711ca53e
- path: results/csv/markduplicates_no_table.csv
md5sum: 2a2d3d4842befd4def39156463859ee3
- path: results/csv/recalibrated.csv
md5sum: 42628ec994c16f565e5407b40a9c1ac3
- path: results/multiqc
- path: results/preprocessing/markduplicates/test/test.md.cram
# binary changes md5sums on reruns.
- path: results/preprocessing/markduplicates/test/test.md.cram.crai
# binary changes md5sums on reruns.
- path: results/preprocessing/recal_table/test/test.recal.table
md5sum: 4ac774bf5f1157e77426fd82f5ac0fbe
- path: results/preprocessing/recalibrated/test/test.recal.cram
# binary changes md5sums on reruns.
- path: results/preprocessing/recalibrated/test/test.recal.cram.crai
# binary changes md5sums on reruns.
- path: results/reports/fastqc/test-test_L1
- path: results/reports/markduplicates/test/test.md.metrics
contains: ["test 8547 767 84 523391 3882 0 0 0.385081", "1.0 767 767"]
- path: results/reports/mosdepth/test/test.md.mosdepth.global.dist.txt
md5sum: 76fa71922a3f748e507c2364c531dfcb
- path: results/reports/mosdepth/test/test.md.mosdepth.region.dist.txt
md5sum: abc5df85e302b79985627888870882da
- path: results/reports/mosdepth/test/test.md.mosdepth.summary.txt
md5sum: d536456436eb275159b8c6af83213d80
- path: results/reports/mosdepth/test/test.md.regions.bed.gz
md5sum: 38fe39894abe62e38f8ac214cba64f2b
- path: results/reports/mosdepth/test/test.recal.mosdepth.global.dist.txt
md5sum: 76fa71922a3f748e507c2364c531dfcb
- path: results/reports/mosdepth/test/test.recal.mosdepth.region.dist.txt
md5sum: abc5df85e302b79985627888870882da
- path: results/reports/mosdepth/test/test.recal.mosdepth.summary.txt
md5sum: d536456436eb275159b8c6af83213d80
- path: results/reports/mosdepth/test/test.recal.regions.bed.gz
md5sum: 38fe39894abe62e38f8ac214cba64f2b
- path: results/reports/samtools/test/test.md.cram.stats
md5sum: dcf70bbcfb92e01027978f28d2035d78
- path: results/reports/samtools/test/test.recal.cram.stats
md5sum: 5528d952f5dc74a39f28e27165bf96be

- name: Run default pipeline without intervals
command: nextflow run main.nf -profile test,no_intervals
tags:
- intervals
- no_intervals
- preprocessing
files:
- path: results/csv/markduplicates.csv
md5sum: 0d6120bb99e92f6810343270711ca53e
- path: results/csv/markduplicates_no_table.csv
md5sum: 2a2d3d4842befd4def39156463859ee3
- path: results/csv/recalibrated.csv
md5sum: 42628ec994c16f565e5407b40a9c1ac3
- path: results/multiqc
- path: results/no_intervals.bed
md5sum: f3dac01ea66b95fe477446fde2d31489
- path: results/no_intervals.bed.gz
md5sum: f3dac01ea66b95fe477446fde2d31489
- path: results/no_intervals.bed.gz.tbi
md5sum: f3dac01ea66b95fe477446fde2d31489
- path: results/preprocessing/markduplicates/test/test.md.cram
# binary changes md5sums on reruns.
- path: results/preprocessing/markduplicates/test/test.md.cram.crai
# binary changes md5sums on reruns.
- path: results/preprocessing/recal_table/test/test.recal.table
md5sum: 4ac774bf5f1157e77426fd82f5ac0fbe
- path: results/preprocessing/recalibrated/test/test.recal.cram
# binary changes md5sums on reruns.
- path: results/preprocessing/recalibrated/test/test.recal.cram.crai
# binary changes md5sums on reruns.
- path: results/reports/fastqc/test-test_L1
- path: results/reports/markduplicates/test/test.md.metrics
contains: ["test 8547 767 84 523391 3882 0 0 0.385081", "1.0 767 767"]
- path: results/reports/mosdepth/test/test.md.mosdepth.global.dist.txt
md5sum: 76fa71922a3f748e507c2364c531dfcb
- path: results/reports/mosdepth/test/test.md.mosdepth.region.dist.txt
md5sum: abc5df85e302b79985627888870882da
- path: results/reports/mosdepth/test/test.md.mosdepth.summary.txt
md5sum: d536456436eb275159b8c6af83213d80
- path: results/reports/mosdepth/test/test.md.regions.bed.gz
md5sum: 38fe39894abe62e38f8ac214cba64f2b
- path: results/reports/mosdepth/test/test.md.regions.bed.gz.csi
md5sum: b1c2a861f64e20a94108a6de3b76c582
- path: results/reports/mosdepth/test/test.recal.mosdepth.global.dist.txt
md5sum: 76fa71922a3f748e507c2364c531dfcb
- path: results/reports/mosdepth/test/test.recal.mosdepth.region.dist.txt
md5sum: abc5df85e302b79985627888870882da
- path: results/reports/mosdepth/test/test.recal.mosdepth.summary.txt
md5sum: d536456436eb275159b8c6af83213d80
- path: results/reports/mosdepth/test/test.recal.regions.bed.gz
md5sum: 38fe39894abe62e38f8ac214cba64f2b
- path: results/reports/mosdepth/test/test.recal.regions.bed.gz.csi
md5sum: b1c2a861f64e20a94108a6de3b76c582
- path: results/reports/samtools/test/test.md.cram.stats
md5sum: dcf70bbcfb92e01027978f28d2035d78
- path: results/reports/samtools/test/test.recal.cram.stats
md5sum: f7005dd4646700e107bfc00e36c432fe
Loading