Add demultiplexing subworkflow 10x #202

nschcolnicov · 2024-07-16T21:53:38Z

This ticket addresses this issue:
nf-core/modules#5984

PR checklist

…, initialized empty test_mkfastq profile file

github-actions · 2024-07-16T21:55:12Z

`nf-core lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit a747c66

+| ✅ 183 tests passed       |+
#| ❔   3 tests were ignored |#
!| ❗   6 tests had warnings |!

❗ Test warnings:

pipeline_todos - TODO string in README.md: Describe the minimum required steps to execute the pipeline, e.g. how to prepare samplesheets.
pipeline_todos - TODO string in methods_description_template.yml: #Update the HTML below to your preferred methods description, e.g. add publication citation for this pipeline
pipeline_todos - TODO string in main.nf: Optionally add in-text citation tools to this list.
pipeline_todos - TODO string in main.nf: Optionally add bibliographic entries to this list.
pipeline_todos - TODO string in main.nf: Only uncomment below if logic in toolCitationText/toolBibliographyText has been filled!
pipeline_todos - TODO string in awsfulltest.yml: You can customise AWS full pipeline tests as required

❔ Tests ignored:

files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/bug_report.yml
files_unchanged - File ignored due to lint config: .github/workflows/linting.yml
actions_ci - actions_ci

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: assets/nf-core-demultiplex_logo_light.png
files_exist - File found: conf/modules.config
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/images/nf-core-demultiplex_logo_light.png
files_exist - File found: docs/images/nf-core-demultiplex_logo_dark.png
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: conf/igenomes.config
files_exist - File found: .github/workflows/awstest.yml
files_exist - File found: .github/workflows/awsfulltest.yml
files_exist - File found: modules.json
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: docs/images/nf-core-demultiplex_logo.png
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/NfcoreTemplate.groovy
files_exist - File not found check: lib/Utils.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: lib/WorkflowMain.groovy
files_exist - File not found check: lib/WorkflowDemultiplex.groovy
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: pipeline_template.yml
files_exist - File not found check: Singularity
files_exist - File not found check: lib/nfcore_external_java_deps.jar
files_exist - File not found check: .travis.yml
nextflow_config - Config variable found: manifest.name
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: manifest.homePage
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: params.validationShowHiddenParams
nextflow_config - Config variable found: params.validationSchemaIgnoreParams
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config manifest.name began with nf-core/
nextflow_config - Config variable manifest.homePage began with https://github.com/nf-core/
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config manifest.version ends in dev: 1.5.0dev
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
nextflow_config - nextflow.config contains configuration profile test
nextflow_config - Config default value correct: params.trim_fastq= true
nextflow_config - Config default value correct: params.custom_config_version= master
nextflow_config - Config default value correct: params.custom_config_base= https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Config default value correct: params.max_cpus= 16
nextflow_config - Config default value correct: params.max_memory= 128.GB
nextflow_config - Config default value correct: params.max_time= 240.h
nextflow_config - Config default value correct: params.publish_dir_mode= copy
nextflow_config - Config default value correct: params.max_multiqc_email_size= 25.MB
nextflow_config - Config default value correct: params.remove_adapter= true
nextflow_config - Config default value correct: params.validate_params= true
nextflow_config - Config default value correct: params.pipelines_testdata_base_path= https://raw.githubusercontent.com/nf-core/test-datasets/
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - CODE_OF_CONDUCT.md matches the template
files_unchanged - LICENSE matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/CONTRIBUTING.md matches the template
files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/PULL_REQUEST_TEMPLATE.md matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - assets/sendmail_template.txt matches the template
files_unchanged - assets/nf-core-demultiplex_logo_light.png matches the template
files_unchanged - docs/images/nf-core-demultiplex_logo_light.png matches the template
files_unchanged - docs/images/nf-core-demultiplex_logo_dark.png matches the template
files_unchanged - docs/README.md matches the template
files_unchanged - .gitignore matches the template
files_unchanged - .prettierignore matches the template
actions_awstest - '.github/workflows/awstest.yml' is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml does not use -profile test
readme - README Nextflow minimum version badge matched config. Badge: 23.04.0, Config: 23.04.0
readme - README Zenodo placeholder was replaced with DOI.
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (198 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: fix-linting.yml
actions_schema_validation - Workflow validation passed: awsfulltest.yml
actions_schema_validation - Workflow validation passed: download_pipeline.yml
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: awstest.yml
actions_schema_validation - Workflow validation passed: branch.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
actions_schema_validation - Workflow validation passed: release-announcements.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
multiqc_config - assets/multiqc_config.yml found and not ignored.
multiqc_config - assets/multiqc_config.yml contains report_section_order
multiqc_config - assets/multiqc_config.yml contains export_plots
multiqc_config - assets/multiqc_config.yml contains report_comment
multiqc_config - assets/multiqc_config.yml follows the ordering scheme of the minimally required plugins.
multiqc_config - assets/multiqc_config.yml contains a matching 'report_comment'.
multiqc_config - assets/multiqc_config.yml contains 'export_plots: true'.
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'
base_config - conf/base.config found and not ignored.
modules_config - conf/modules.config found and not ignored.
modules_config - UNTAR found in conf/modules.config and Nextflow scripts.
modules_config - BCLCONVERT found in conf/modules.config and Nextflow scripts.
modules_config - BCL2FASTQ found in conf/modules.config and Nextflow scripts.
modules_config - BASES2FASTQ found in conf/modules.config and Nextflow scripts.
modules_config - FASTP found in conf/modules.config and Nextflow scripts.
modules_config - FALCO found in conf/modules.config and Nextflow scripts.
modules_config - MD5SUM found in conf/modules.config and Nextflow scripts.
modules_config - CUSTOM_DUMPSOFTWAREVERSIONS found in conf/modules.config and Nextflow scripts.
modules_config - SGDEMUX found in conf/modules.config and Nextflow scripts.
modules_config - FQTK found in conf/modules.config and Nextflow scripts.
modules_config - CELLRANGER_MKFASTQ found in conf/modules.config and Nextflow scripts.
modules_config - MULTIQC found in conf/modules.config and Nextflow scripts.
nfcore_yml - Repository type in .nf-core.yml is valid: pipeline
nfcore_yml - nf-core version in .nf-core.yml is set to the latest version: 2.14.1

Run details

nf-core/tools version 2.14.1
Run at 2024-07-25 12:24:16

conf/test_mkfastq.config

atrigila · 2024-07-17T00:33:47Z

Can you run pre-commit run --all-files and commit the suggested changes? That will fix the linting issues.

atrigila

Good job, left some comments. It would be nice to add the mkfastq test (and others) to GitHub CI tests to have automatic checks when pushing changes, if the file size allows.

README.md

conf/test_mkfastq.config

docs/output.md

modules/nf-core/cellranger/mkfastq/main.nf

subworkflows/local/mkfastq_demultiplex/main.nf

nschcolnicov · 2024-07-17T21:04:38Z

@nf-core-bot fix linting pretty please

…om/nschcolnicov/demultiplex into add_demultiplexing_subworkflow_10x

…urces for CI

fmalmeida · 2024-07-23T20:40:10Z

tests/pipeline/mkfastq.nf.test.snap

This snapshot seems to be missing the snaps of the filepaths.

Nice catch, weird, Ill look into it.

fmalmeida · 2024-07-23T20:42:14Z

workflows/demultiplex.nf

@@ -230,7 +238,7 @@ workflow DEMULTIPLEX {
            )
        )

-        MULTIQC (
+        MULTIQC ( //TODO fix multiqc not resuming


Probably the summary maps used to collect run information might have some float variable like run name, star time, or something like this.

Would recommend first looking at the "ch_workflow_summary"

Sounds good, I will address this on a separate ticket to keep things clean.

nschcolnicov · 2024-07-25T11:30:39Z

The issue with the mkfastq CI test, seems to be unrelated to the scope of this ticket. I will remove it from the PR and address it in a separate ticket: #211

…_demultiplexing_subworkflow_10x

zxBIB Schcolnicov added 5 commits July 11, 2024 20:55

Initialized subworkflow, added nf-core module, updated modules.config…

8563f28

…, initialized empty test_mkfastq profile file

More updates to mkfastq subworfklow, module and config

ca21119

More changes

7351148

Fixed output channel for mkfastq

cc4f73f

Updated mkfastq for nftests and stub runs

53ae48f

nschcolnicov requested review from fmalmeida and atrigila July 16, 2024 21:53

nschcolnicov requested a review from a team as a code owner July 16, 2024 21:53

nschcolnicov requested review from glichtenstein and removed request for a team July 16, 2024 21:53

nschcolnicov commented Jul 16, 2024

View reviewed changes

conf/test_mkfastq.config Outdated Show resolved Hide resolved

Updated PR number, and removed TODO

519c931

nschcolnicov mentioned this pull request Jul 17, 2024

Add Support for 10X Single Cell Demux #195

Closed

atrigila requested changes Jul 17, 2024

View reviewed changes

Addressing PR comments

fd6706c

nf-core-bot and others added 8 commits July 17, 2024 21:05

[automated] Fix code linting

c4cb678

Fixed editorconfig issues

06240de

Updated main.nf.test and snap

605f9a4

Merge branch 'add_demultiplexing_subworkflow_10x' of https://github.c…

dca93a2

…om/nschcolnicov/demultiplex into add_demultiplexing_subworkflow_10x

Added CI test for mkfastq:

412efdf

Pulled from dev

1ab96c0

Installed mkfastq from modules for CI test

04bd37b

Re-added removed test profile during merge, updated mkfastq test reso…

b76b664

…urces for CI

fmalmeida reviewed Jul 23, 2024

View reviewed changes

nschcolnicov force-pushed the add_demultiplexing_subworkflow_10x branch 3 times, most recently from 20c0676 to b76b664 Compare July 23, 2024 21:57

nschcolnicov force-pushed the add_demultiplexing_subworkflow_10x branch from 3b4913e to b76b664 Compare July 23, 2024 23:48

Updated CI test and mkfastq nf-test and snap

787744e

nschcolnicov added 2 commits July 25, 2024 12:21

Merge branch 'dev' of https://github.com/nf-core/demultiplex into add…

a761876

…_demultiplexing_subworkflow_10x

Pulled from dev and removed mkfastq from CI tests

a747c66

atrigila approved these changes Jul 25, 2024

View reviewed changes

nschcolnicov merged commit cccaae2 into nf-core:dev Jul 25, 2024
9 checks passed

nschcolnicov mentioned this pull request Jul 26, 2024

Document novel 10X demux subworkflow & testing appropriately #197

Closed

apeltzer mentioned this pull request Aug 12, 2024

Release PR 1.5.0 #242

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add demultiplexing subworkflow 10x #202

Add demultiplexing subworkflow 10x #202

nschcolnicov commented Jul 16, 2024 •

edited

Loading

github-actions bot commented Jul 16, 2024 •

edited

Loading

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

atrigila commented Jul 17, 2024

atrigila left a comment

nschcolnicov commented Jul 17, 2024

fmalmeida Jul 23, 2024

nschcolnicov Jul 23, 2024

fmalmeida Jul 23, 2024

nschcolnicov Jul 23, 2024

nschcolnicov commented Jul 25, 2024

Add demultiplexing subworkflow 10x #202

Add demultiplexing subworkflow 10x #202

Conversation

nschcolnicov commented Jul 16, 2024 • edited Loading

PR checklist

github-actions bot commented Jul 16, 2024 • edited Loading

nf-core lint overall result: Passed ✅ ⚠️

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

atrigila commented Jul 17, 2024

atrigila left a comment

Choose a reason for hiding this comment

nschcolnicov commented Jul 17, 2024

fmalmeida Jul 23, 2024

Choose a reason for hiding this comment

nschcolnicov Jul 23, 2024

Choose a reason for hiding this comment

fmalmeida Jul 23, 2024

Choose a reason for hiding this comment

nschcolnicov Jul 23, 2024

Choose a reason for hiding this comment

nschcolnicov commented Jul 25, 2024

nschcolnicov commented Jul 16, 2024 •

edited

Loading

github-actions bot commented Jul 16, 2024 •

edited

Loading

`nf-core lint` overall result: Passed ✅ ⚠️