Add UMI clustering #24

mirpedrol · 2023-05-05T07:26:25Z

Add UMI clustering functionality.
Close #16

PR checklist

This comment contains a description of changes (with reason).
If you've fixed a bug or added code that should be tested, add tests!
If you've added a new tool - have you followed the pipeline conventions in the contribution docs- [ ] If necessary, also make a PR on the nf-core/crisprseq branch on the nf-core/test-datasets repository.
Make sure your code lints (nf-core lint).
Ensure the test suite passes (nextflow run . -profile test,docker --outdir <OUTDIR>).
Usage Documentation in docs/usage.md is updated.
Output Documentation in docs/output.md is updated.
CHANGELOG.md is updated.
README.md is updated (including new tool citations and authors/contributors).

…lignments to ignore empty alignments

…lustering

github-actions · 2023-05-05T07:28:32Z

`nf-core lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit 2227d89

+| ✅ 155 tests passed       |+
#| ❔  17 tests were ignored |#
!| ❗   3 tests had warnings |!

❗ Test warnings:

pipeline_todos - TODO string in README.md: If applicable, make list of people who have also contributed
pipeline_todos - TODO string in WorkflowMain.groovy: Add Zenodo DOI for pipeline after first release
pipeline_todos - TODO string in methods_description_template.yml: #Update the HTML below to your prefered methods description, e.g. add publication citation for this pipeline

❔ Tests ignored:

files_exist - File is ignored: c
files_exist - File is ignored: o
files_exist - File is ignored: n
files_exist - File is ignored: f
files_exist - File is ignored: /
files_exist - File is ignored: t
files_exist - File is ignored: e
files_exist - File is ignored: s
files_exist - File is ignored: t
files_exist - File is ignored: .
files_exist - File is ignored: c
files_exist - File is ignored: o
files_exist - File is ignored: n
files_exist - File is ignored: f
files_exist - File is ignored: i
files_exist - File is ignored: g
actions_ci - actions_ci

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: assets/nf-core-crisprseq_logo_light.png
files_exist - File found: conf/modules.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/images/nf-core-crisprseq_logo_light.png
files_exist - File found: docs/images/nf-core-crisprseq_logo_dark.png
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: lib/nfcore_external_java_deps.jar
files_exist - File found: lib/NfcoreSchema.groovy
files_exist - File found: lib/NfcoreTemplate.groovy
files_exist - File found: lib/Utils.groovy
files_exist - File found: lib/WorkflowMain.groovy
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: conf/igenomes.config
files_exist - File found: .github/workflows/awstest.yml
files_exist - File found: .github/workflows/awsfulltest.yml
files_exist - File found: lib/WorkflowCrisprseq.groovy
files_exist - File found: modules.json
files_exist - File found: pyproject.toml
files_exist - File not found check: Singularity
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: docs/images/nf-core-crisprseq_logo.png
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: .travis.yml
nextflow_config - Config variable found: manifest.name
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: manifest.homePage
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: params.show_hidden_params
nextflow_config - Config variable found: params.schema_ignore_params
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config manifest.name began with nf-core/
nextflow_config - Config variable manifest.homePage began with https://github.com/nf-core/
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config manifest.version ends in dev: '1.1dev'
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - CODE_OF_CONDUCT.md matches the template
files_unchanged - LICENSE matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/CONTRIBUTING.md matches the template
files_unchanged - .github/ISSUE_TEMPLATE/bug_report.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/PULL_REQUEST_TEMPLATE.md matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - .github/workflows/linting.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - assets/sendmail_template.txt matches the template
files_unchanged - assets/nf-core-crisprseq_logo_light.png matches the template
files_unchanged - docs/images/nf-core-crisprseq_logo_light.png matches the template
files_unchanged - docs/images/nf-core-crisprseq_logo_dark.png matches the template
files_unchanged - docs/README.md matches the template
files_unchanged - lib/nfcore_external_java_deps.jar matches the template
files_unchanged - lib/NfcoreSchema.groovy matches the template
files_unchanged - lib/NfcoreTemplate.groovy matches the template
files_unchanged - .gitignore matches the template
files_unchanged - .prettierignore matches the template
files_unchanged - pyproject.toml matches the template
actions_awstest - '.github/workflows/awstest.yml' is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml does not use -profile test
readme - README Nextflow minimum version badge matched config. Badge: 22.10.1, Config: 22.10.1
readme - README Nextflow minimum version in Quick Start section matched config. README: 22.10.1, Config: 22.10.1
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (153 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: branch.yml
actions_schema_validation - Workflow validation passed: awsfulltest_screening.yml
actions_schema_validation - Workflow validation passed: awsfulltest.yml
actions_schema_validation - Workflow validation passed: fix-linting.yml
actions_schema_validation - Workflow validation passed: awstest.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
actions_schema_validation - Workflow validation passed: ci.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
multiqc_config - 'assets/multiqc_config.yml' follows the ordering scheme of the minimally required plugins.
multiqc_config - 'assets/multiqc_config.yml' contains a matching 'report_comment'.
multiqc_config - 'assets/multiqc_config.yml' contains 'export_plots: true'.
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'

Run details

nf-core/tools version 2.7.2
Run at 2023-06-28 11:43:08

… into mirpedrol-umi-clustering

…lustering

CHANGELOG.md

workflows/crisprseq_targeted.nf

nextflow_schema.json

docs/usage_targeted.md

…lustering

mirpedrol added 18 commits March 15, 2023 09:54

add extract umi step

760fa55

add test_umis.config

21cda6b

fix extract_umis

373b809

install vsearch

ab49b68

add vsearch/cluster step

1eb72b8

don't compress cluster fastas

8467cd9

add closure to obtain sequence from umi

bc43a97

try to make umi_to_sequence.py faster

181c3d5

use a groovy function for umi_to_sequence

2198371

install vsearch/sort

828dbf5

add umi_bin_size parameter

731ffb3

update vsearch/sort and add sequence obtention from umi

46535d8

add first minimap step for UMI consensus

75584e2

install new module racon

4199628

add racon

e9adce8

fix bug returning only one sequence from UMI cluster and filter UMI a…

a85fc45

…lignments to ignore empty alignments

Merge branch 'dev' of https://github.com/nf-core/crisprseq into umi-c…

e8adbd0

…lustering

update changelog

424ee03

mirpedrol and others added 11 commits May 5, 2023 10:15

add test_umi tests to CI

07e21ff

install medaka

4ff306e

install samtools/faidx

ea19682

run faidx

9bf0105

add minimap and samtools for mini_align process

ebf6fd5

remove mini_align as it's now performed by medaka_consensus

d883f15

add medaka process

de43d35

join all umi consensus in a file

353a8ba

convert fasta to fastq after umi consensus

ac9f267

remove dummi_final_umi process

be105b4

pass final umi consensus tu clustering_summary

798b4db

mirpedrol added 11 commits May 9, 2023 13:35

pass final umi consensus to mapping steps

8d10395

unzip input medaka

42302dd

make removing overrepresented sequences optional

a996906

unzip medaka input

2706b7a

fix medaka error

c0974bf

hopefully fix umis bug

565e6e8

Merge branch 'umi-clustering' of https://github.com/mirpedrol/crisprseq…

4a88719

… into mirpedrol-umi-clustering

Merge branch 'mirpedrol-umi-clustering' into dev

eeffbc6

Merge branch 'dev' of https://github.com/nf-core/crisprseq into umi-c…

e15d2ef

…lustering

fix typo

6c57e3e

update schema

6dfba88

mirpedrol marked this pull request as ready for review June 12, 2023 07:13

add analysis param in test_umis

4ed1882

mirpedrol requested review from LaurenceKuhl and msanvicente June 12, 2023 07:25

add docs for UMIs

0482c06

mirpedrol commented Jun 27, 2023

View reviewed changes

mirpedrol added 8 commits June 27, 2023 15:32

Merge branch 'dev' of https://github.com/nf-core/crisprseq into umi-c…

7893867

…lustering

Apply suggestions from code review

4ab6d0b

add --umi_clustering parameter to opt-in the umi clustering steps

ae81ac0

add pipeline optional step parameters to usage documentation

602a52b

add pipeline optional step parameters to usage documentation

09201ed

fix alias SEQTK_SEQ_MASK

bf99c28

fix: MERGING_SUMMARY runs when --overrepresented is false

fb1c14f

add param umi_clustering to test_umis

7f9bfda

LaurenceKuhl approved these changes Jun 28, 2023

View reviewed changes

Merge branch 'dev' of https://github.com/nf-core/crisprseq into umi-c…

2227d89

…lustering

mirpedrol merged commit 5553085 into nf-core:dev Jun 28, 2023

mirpedrol deleted the umi-clustering branch June 28, 2023 12:11

mirpedrol mentioned this pull request Jun 29, 2023

Add UMIs #16

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add UMI clustering #24

Add UMI clustering #24

mirpedrol commented May 5, 2023 •

edited

Loading

github-actions bot commented May 5, 2023 •

edited

Loading

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

Add UMI clustering #24

Add UMI clustering #24

Conversation

mirpedrol commented May 5, 2023 • edited Loading

PR checklist

github-actions bot commented May 5, 2023 • edited Loading

nf-core lint overall result: Passed ✅ ⚠️

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

mirpedrol commented May 5, 2023 •

edited

Loading

github-actions bot commented May 5, 2023 •

edited

Loading

`nf-core lint` overall result: Passed ✅ ⚠️