Sarek bcftools normalization #1682

Patricie34 · 2024-10-09T17:06:34Z

PR checklist

…cfs_with_tbis and fasta.

resolve issue with spacing Co-authored-by: Maxime U Garcia <[email protected]>

…A support

Patricie34 · 2024-10-09T17:11:44Z

Hi all,

I've modified the normalization step to include all VCFs, not just the germline ones. For this, I used the pull request from JC-Delmas as a base. I am aware that this still requires a lot of work, and I would greatly appreciate any advice or feedback you can provide.

Thank you!

Patricie

conf/modules/post_variant_calling.config

maxulysse · 2024-10-10T06:01:58Z

modules/nf-core/bcftools/norm/environment.yml

Can you run nf-core modules update bcftools/norm that's an old version of the modules

github-actions · 2024-10-10T06:03:00Z

`nf-core pipelines lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit 6c39a63

+| ✅ 215 tests passed       |+
#| ❔  11 tests were ignored |#
!| ❗   4 tests had warnings |!

❗ Test warnings:

pipeline_todos - TODO string in main.nf: Optionally add in-text citation tools to this list.
pipeline_todos - TODO string in main.nf: Optionally add bibliographic entries to this list.
pipeline_todos - TODO string in main.nf: Only uncomment below if logic in toolCitationText/toolBibliographyText has been filled!
pipeline_todos - TODO string in base.config: Check the defaults for all processes

❔ Tests ignored:

files_exist - File is ignored: .github/workflows/awsfulltest.yml
files_exist - File is ignored: .github/workflows/awstest.yml
files_exist - File is ignored: conf/modules.config
files_unchanged - File ignored due to lint config: assets/nf-core-sarek_logo_light.png
files_unchanged - File ignored due to lint config: docs/images/nf-core-sarek_logo_light.png
files_unchanged - File ignored due to lint config: docs/images/nf-core-sarek_logo_dark.png
files_unchanged - File ignored due to lint config: .gitignore or .prettierignore
actions_ci - actions_ci
actions_awstest - 'awstest.yml' workflow not found: /home/runner/work/sarek/sarek/.github/workflows/awstest.yml
template_strings - template_strings
modules_config - modules_config

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: assets/nf-core-sarek_logo_light.png
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/images/nf-core-sarek_logo_light.png
files_exist - File found: docs/images/nf-core-sarek_logo_dark.png
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: conf/igenomes.config
files_exist - File found: conf/igenomes_ignored.config
files_exist - File found: modules.json
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: docs/images/nf-core-sarek_logo.png
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/NfcoreTemplate.groovy
files_exist - File not found check: lib/Utils.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: lib/WorkflowMain.groovy
files_exist - File not found check: lib/WorkflowSarek.groovy
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: pipeline_template.yml
files_exist - File not found check: Singularity
files_exist - File not found check: lib/nfcore_external_java_deps.jar
files_exist - File not found check: .travis.yml
nextflow_config - Found nf-schema plugin
nextflow_config - Config variable found: manifest.name
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: manifest.homePage
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: validation.help.enabled
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable found: validation.help.beforeText
nextflow_config - Config variable found: validation.help.afterText
nextflow_config - Config variable found: validation.help.command
nextflow_config - Config variable found: validation.summary.beforeText
nextflow_config - Config variable found: validation.summary.afterText
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config variable (correctly) not found: params.max_cpus
nextflow_config - Config variable (correctly) not found: params.max_memory
nextflow_config - Config variable (correctly) not found: params.max_time
nextflow_config - Config variable (correctly) not found: params.validationFailUnrecognisedParams
nextflow_config - Config variable (correctly) not found: params.validationLenientMode
nextflow_config - Config variable (correctly) not found: params.validationSchemaIgnoreParams
nextflow_config - Config variable (correctly) not found: params.validationShowHiddenParams
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config manifest.name began with nf-core/
nextflow_config - Config variable manifest.homePage began with https://github.com/nf-core/
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config manifest.version ends in dev: 3.5.0dev
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
nextflow_config - nextflow.config contains configuration profile test
nextflow_config - Config default value correct: params.step= mapping
nextflow_config - Config default value correct: params.split_fastq= 50000000
nextflow_config - Config default value correct: params.nucleotides_per_second= 200000
nextflow_config - Config default value correct: params.clip_r1= 0
nextflow_config - Config default value correct: params.clip_r2= 0
nextflow_config - Config default value correct: params.three_prime_clip_r1= 0
nextflow_config - Config default value correct: params.three_prime_clip_r2= 0
nextflow_config - Config default value correct: params.trim_nextseq= 0
nextflow_config - Config default value correct: params.length_required= 15
nextflow_config - Config default value correct: params.group_by_umi_strategy= Adjacency
nextflow_config - Config default value correct: params.aligner= bwa-mem
nextflow_config - Config default value correct: params.ascat_min_base_qual= 20
nextflow_config - Config default value correct: params.ascat_min_counts= 10
nextflow_config - Config default value correct: params.ascat_min_map_qual= 35
nextflow_config - Config default value correct: params.cf_coeff= 0.05
nextflow_config - Config default value correct: params.cf_contamination= 0
nextflow_config - Config default value correct: params.cf_minqual= 0
nextflow_config - Config default value correct: params.cf_mincov= 0
nextflow_config - Config default value correct: params.cf_ploidy= 2
nextflow_config - Config default value correct: params.sentieon_haplotyper_emit_mode= variant
nextflow_config - Config default value correct: params.sentieon_dnascope_emit_mode= variant
nextflow_config - Config default value correct: params.sentieon_dnascope_pcr_indel_model= CONSERVATIVE
nextflow_config - Config default value correct: params.dbnsfp_fields= rs_dbSNP,HGVSc_VEP,HGVSp_VEP,1000Gp3_EAS_AF,1000Gp3_AMR_AF,LRT_score,GERP++_RS,gnomAD_exomes_AF
nextflow_config - Config default value correct: params.vep_custom_args= --everything --filter_common --per_gene --total_length --offline --format vcf
nextflow_config - Config default value correct: params.vep_version= 111.0-0
nextflow_config - Config default value correct: params.vep_out_format= vcf
nextflow_config - Config default value correct: params.igenomes_base= s3://ngi-igenomes/igenomes/
nextflow_config - Config default value correct: params.genome= GATK.GRCh38
nextflow_config - Config default value correct: params.snpeff_cache= s3://annotation-cache/snpeff_cache/
nextflow_config - Config default value correct: params.vep_cache= s3://annotation-cache/vep_cache/
nextflow_config - Config default value correct: params.custom_config_version= master
nextflow_config - Config default value correct: params.custom_config_base= https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Config default value correct: params.test_data_base= https://raw.githubusercontent.com/nf-core/test-datasets/sarek3
nextflow_config - Config default value correct: params.seq_platform= ILLUMINA
nextflow_config - Config default value correct: params.publish_dir_mode= copy
nextflow_config - Config default value correct: params.max_multiqc_email_size= 25.MB
nextflow_config - Config default value correct: params.validate_params= true
nextflow_config - Config default value correct: params.pipelines_testdata_base_path= https://raw.githubusercontent.com/nf-core/test-datasets/
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - CODE_OF_CONDUCT.md matches the template
files_unchanged - LICENSE matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/CONTRIBUTING.md matches the template
files_unchanged - .github/ISSUE_TEMPLATE/bug_report.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/PULL_REQUEST_TEMPLATE.md matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - .github/workflows/linting.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - assets/sendmail_template.txt matches the template
files_unchanged - docs/README.md matches the template
readme - README Nextflow minimum version badge matched config. Badge: 24.04.2, Config: 24.04.2
readme - README Zenodo placeholder was replaced with DOI.
plugin_includes - No wrong validation plugin imports have been found
pipeline_name_conventions - Name adheres to nf-core convention
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: branch.yml
actions_schema_validation - Workflow validation passed: fix-linting.yml
actions_schema_validation - Workflow validation passed: release-announcements.yml
actions_schema_validation - Workflow validation passed: template_version_comment.yml
actions_schema_validation - Workflow validation passed: download_pipeline.yml
actions_schema_validation - Workflow validation passed: ncbench.yml
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: cloudtest.yml
actions_schema_validation - Workflow validation passed: pytest.yml
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
multiqc_config - assets/multiqc_config.yml found and not ignored.
multiqc_config - assets/multiqc_config.yml contains report_section_order
multiqc_config - assets/multiqc_config.yml contains export_plots
multiqc_config - assets/multiqc_config.yml contains report_comment
multiqc_config - assets/multiqc_config.yml follows the ordering scheme of the minimally required plugins.
multiqc_config - assets/multiqc_config.yml contains a matching 'report_comment'.
multiqc_config - assets/multiqc_config.yml contains 'export_plots: true'.
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'
base_config - conf/base.config found and not ignored.
base_config - UNZIP found in conf/base.config and Nextflow scripts.
base_config - FASTQC found in conf/base.config and Nextflow scripts.
base_config - FASTP found in conf/base.config and Nextflow scripts.
base_config - BWAMEM1_MEM found in conf/base.config and Nextflow scripts.
base_config - CNVKIT_BATCH found in conf/base.config and Nextflow scripts.
base_config - GATK4_MARKDUPLICATES found in conf/base.config and Nextflow scripts.
base_config - GATK4_APPLYBQSR found in conf/base.config and Nextflow scripts.
base_config - MOSDEPTH found in conf/base.config and Nextflow scripts.
base_config - STRELKA found in conf/base.config and Nextflow scripts.
base_config - SAMTOOLS_CONVERT found in conf/base.config and Nextflow scripts.
base_config - GATK4_MERGEVCFS found in conf/base.config and Nextflow scripts.
base_config - MULTIQC found in conf/base.config and Nextflow scripts.
nfcore_yml - Repository type in .nf-core.yml is valid: pipeline
nfcore_yml - nf-core version in .nf-core.yml is set to the latest version: 3.0.2

Run details

nf-core/tools version 3.0.2
Run at 2024-12-04 06:57:12

maxulysse · 2024-10-10T06:03:24Z

@nf-core-bot fix linting pretty please 🙏

maxulysse · 2024-10-10T06:03:58Z

We're missing CHANGELOG + tests + subway map

Co-authored-by: Maxime U Garcia <[email protected]>

maxulysse · 2024-10-11T10:01:00Z

@nf-core-bot fix linting pretty please 🙏

maxulysse · 2024-10-15T05:59:24Z

nextflow

You don't need to commit this file

this file is still there

conf/modules/post_variant_calling.config

subworkflows/local/vcf_normalization/main.nf

maxulysse · 2024-11-25T15:30:55Z

issues we still need to assess:

WHY do we output vcfs_tbi from the concatenate subworkflow, when we just need vcf for vcftools and we don't seem to remove them anywhere?
I think we probably need to output just vcf from there, or keep it somewhere and map it out for the downstream processes.
I have little clues why it's not failing.

We need a variant caller id from concatenate as well.

I'm guessing we might need to output vcfs = VCFS_NORM_SORT.out.vcf from the normalization subworkflows and something similar from the concatenate one.

subworkflows/local/vcf_normalization/main.nf

tests/test_normalize_vcfs.yml

FriederikeHanssen

Amazing work! Now only the fun little side quests are missing:

Updating the readme to include this in the possible steps
Updating the subway map
Add a section to the output.md docs

nf-core-bot · 2024-12-17T06:32:15Z

Warning

Newer version of the nf-core template is available.

Your pipeline is using an old version of the nf-core template: 3.0.2.
Please update your pipeline to the latest version.

For more documentation on how to update your pipeline, please see the nf-core documentation and Synchronisation documentation.

maxulysse · 2024-12-17T17:00:21Z

@Patricie34 sorry, I forgot to merge in #1760 (which is now done).

Can you sync you branch once more, and move your PR up in the CHANGELOG?

…to update the latest changes.

JC-Delmas and others added 15 commits April 25, 2024 10:55

First modification to contribute to the bcftools/norm module in Sarek

4772da1

Changes in the GERMLINE_VCFS_NORM process

451aaec

Add fasta argument to POST_VARIANTCALLING process.

d97726b

add fasta input as argument

e034ff0

remove vcfs in the GERMLINE_VCFS_NORM process, replaced by germline_v…

8469832

…cfs_with_tbis and fasta.

First modification to contribute to the bcftools/norm module in Sarek

e885888

Changes in the GERMLINE_VCFS_NORM process

9e94a05

Add fasta argument to POST_VARIANTCALLING process.

2bdba7e

add fasta input as argument

1214f10

remove vcfs in the GERMLINE_VCFS_NORM process, replaced by germline_v…

b7ba4f2

…cfs_with_tbis and fasta.

Update workflows/sarek/main.nf

34bf47b

resolve issue with spacing Co-authored-by: Maxime U Garcia <[email protected]>

Resolved merge conflict by keeping changes from branch 34bf47b

6dff9af

Refactor normalization and concatenation of VCF files

d289261

Modify and adjust two scripts to add normalization and integrate FAST…

c78af62

…A support

Added normalization for all vcfs

d646ec3

Patricie34 requested review from FriederikeHanssen and maxulysse as code owners October 9, 2024 17:06

maxulysse reviewed Oct 10, 2024

View reviewed changes

conf/modules/post_variant_calling.config Outdated Show resolved Hide resolved

maxulysse reviewed Oct 10, 2024

View reviewed changes

Patricie34 and others added 3 commits October 11, 2024 09:01

Fixed linting issues and updated schema parameters

8fb64b2

Update conf/modules/post_variant_calling.config

92094af

Co-authored-by: Maxime U Garcia <[email protected]>

edit of normalization steps

fbbfe1b

Fixed linting issues

24791dc

maxulysse reviewed Oct 15, 2024

View reviewed changes

Merge remote-tracking branch 'upstream/dev' into sarek_bcftools_norm

50f1b4b

FriederikeHanssen mentioned this pull request Nov 20, 2024

Implemented TMB score calculation #1738

Open

11 tasks

maxulysse reviewed Nov 25, 2024

View reviewed changes

conf/modules/post_variant_calling.config Outdated Show resolved Hide resolved

Update conf/modules/post_variant_calling.config

0dbcabd

maxulysse reviewed Nov 25, 2024

View reviewed changes

subworkflows/local/vcf_normalization/main.nf Outdated Show resolved Hide resolved

Update subworkflows/local/vcf_normalization/main.nf

3f99fb2

maxulysse reviewed Nov 25, 2024

View reviewed changes

subworkflows/local/vcf_normalization/main.nf Outdated Show resolved Hide resolved

Update subworkflows/local/vcf_normalization/main.nf

2ed8ad1

maxulysse reviewed Nov 25, 2024

View reviewed changes

subworkflows/local/vcf_normalization/main.nf Show resolved Hide resolved

versions corrected

817847c

maxulysse reviewed Nov 27, 2024

View reviewed changes

tests/test_normalize_vcfs.yml Outdated Show resolved Hide resolved

maxulysse and others added 8 commits November 27, 2024 12:36

Apply suggestions from code review

af736b4

vcf_concatenate_germline workflow edit

e877ed4

major changes in concatenate_vcfs

bb48905

paths in test.yml changed

4980695

linting changed

a31e7db

test_concatenate.yml edited

74fd353

test_concatenate.yml edited

ee2bcb9

test_concatenate.yml adn normalize_yml edited

4b69e35

FriederikeHanssen requested changes Nov 28, 2024

View reviewed changes

Patricie34 added 4 commits December 4, 2024 07:46

update docs, changelog and readme

b0a44d6

output.md linting

6c39a63

subway_map updated

7ae3b44

Resolved merge conflicts with upstream/dev

6f1f8ec

Subway_map updated

01b974b

Patricie34 added 3 commits December 18, 2024 06:28

Merge remote-tracking branch 'upstream/dev' into sarek_bcftools_norm …

fe5027a

…to update the latest changes.

CHANGELOG.md updated

f58bc10

CHANGELOG edited

5784bb7

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sarek bcftools normalization #1682

Sarek bcftools normalization #1682

Patricie34 commented Oct 9, 2024 •

edited

Loading

Patricie34 commented Oct 9, 2024

maxulysse Oct 10, 2024

github-actions bot commented Oct 10, 2024 •

edited

Loading

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

maxulysse commented Oct 10, 2024

maxulysse commented Oct 10, 2024

maxulysse commented Oct 11, 2024

maxulysse Oct 15, 2024

maxulysse Oct 15, 2024

maxulysse commented Nov 25, 2024

FriederikeHanssen left a comment

nf-core-bot commented Dec 17, 2024

maxulysse commented Dec 17, 2024

Sarek bcftools normalization #1682

Are you sure you want to change the base?

Sarek bcftools normalization #1682

Conversation

Patricie34 commented Oct 9, 2024 • edited Loading

PR checklist

Patricie34 commented Oct 9, 2024

maxulysse Oct 10, 2024

Choose a reason for hiding this comment

github-actions bot commented Oct 10, 2024 • edited Loading

nf-core pipelines lint overall result: Passed ✅ ⚠️

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

maxulysse commented Oct 10, 2024

maxulysse commented Oct 10, 2024

maxulysse commented Oct 11, 2024

maxulysse Oct 15, 2024

Choose a reason for hiding this comment

maxulysse Oct 15, 2024

Choose a reason for hiding this comment

maxulysse commented Nov 25, 2024

FriederikeHanssen left a comment

Choose a reason for hiding this comment

nf-core-bot commented Dec 17, 2024

maxulysse commented Dec 17, 2024

Patricie34 commented Oct 9, 2024 •

edited

Loading

github-actions bot commented Oct 10, 2024 •

edited

Loading

`nf-core pipelines lint` overall result: Passed ✅ ⚠️