Updated to support metadata and added ArborView HTML app functionality.

[kylacochrane]

This PR generates a TSV-formatted metadata file from the input samplesheet and merges it with the PHYML phylogenetic tree. The combined data is then processed through the ArborView HTML app to produce dendrograms with contextual metadata.

PR checklist

• This comment contains a description of changes (with reason).
• If you've fixed a bug or added code that should be tested, add tests!
• If you've added a new tool - have you followed the pipeline conventions in the contribution docs
• Make sure your code lints (nf-core lint).
• Ensure the test suite passes (nextflow run . -profile test,docker --outdir <OUTDIR>).
• Usage Documentation in docs/usage.md is updated.
• Output Documentation in docs/output.md is updated.
• CHANGELOG.md is updated.
• README.md is updated (including new tool citations and authors/contributors).

[github-actions]

nf-core lint overall result: Passed ✅ ⚠️

Posted for pipeline commit 32cb79a

+| ✅ 143 tests passed       |+
#| ❔  27 tests were ignored |#
!| ❗   3 tests had warnings |!

❗ Test warnings:

• nextflow_config - Config manifest.version should end in dev: 2.1.0
• schema_lint - Schema $id should be https://raw.githubusercontent.com/phac-nml/snvphylnfc/master/nextflow_schema.json
  Found https://raw.githubusercontent.com/phac-nml/snvphylnfc/main/nextflow_schema.json
• nfcore_yml - nf-core version not set in .nf-core.yml

❔ Tests ignored:

• files_exist - File is ignored: assets/nf-core-snvphylnfc_logo_light.png
• files_exist - File is ignored: docs/images/nf-core-snvphylnfc_logo_light.png
• files_exist - File is ignored: docs/images/nf-core-snvphylnfc_logo_dark.png
• files_exist - File is ignored: .github/workflows/awstest.yml
• files_exist - File is ignored: .github/workflows/awsfulltest.yml
• files_exist - File is ignored: lib/Utils.groovy
• files_exist - File is ignored: lib/WorkflowMain.groovy
• files_exist - File is ignored: lib/NfcoreTemplate.groovy
• files_exist - File is ignored: lib/WorkflowSnvphylnfc.groovy
• nextflow_config - Config variable ignored: manifest.name
• nextflow_config - Config variable ignored: manifest.homePage
• files_unchanged - File ignored due to lint config: LICENSE or LICENSE.md or LICENCE or LICENCE.md
• files_unchanged - File ignored due to lint config: .github/CONTRIBUTING.md
• files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/bug_report.yml
• files_unchanged - File ignored due to lint config: .github/PULL_REQUEST_TEMPLATE.md
• files_unchanged - File ignored due to lint config: .github/workflows/branch.yml
• files_unchanged - File ignored due to lint config: .github/workflows/linting.yml
• files_unchanged - File ignored due to lint config: assets/email_template.html
• files_unchanged - File ignored due to lint config: assets/email_template.txt
• files_unchanged - File ignored due to lint config: assets/sendmail_template.txt
• files_unchanged - File does not exist: assets/nf-core-snvphylnfc_logo_light.png
• files_unchanged - File does not exist: docs/images/nf-core-snvphylnfc_logo_light.png
• files_unchanged - File does not exist: docs/images/nf-core-snvphylnfc_logo_dark.png
• files_unchanged - File ignored due to lint config: docs/README.md
• actions_awstest - 'awstest.yml' workflow not found: /home/runner/work/snvphylnfc/snvphylnfc/.github/workflows/awstest.yml
• actions_awsfulltest - actions_awsfulltest
• pipeline_name_conventions - pipeline_name_conventions

✅ Tests passed:

• files_exist - File found: .gitattributes
• files_exist - File found: .gitignore
• files_exist - File found: .nf-core.yml
• files_exist - File found: .editorconfig
• files_exist - File found: .prettierignore
• files_exist - File found: .prettierrc.yml
• files_exist - File found: CHANGELOG.md
• files_exist - File found: CITATIONS.md
• files_exist - File found: CODE_OF_CONDUCT.md
• files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
• files_exist - File found: nextflow_schema.json
• files_exist - File found: nextflow.config
• files_exist - File found: README.md
• files_exist - File found: .github/.dockstore.yml
• files_exist - File found: .github/CONTRIBUTING.md
• files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
• files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
• files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
• files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
• files_exist - File found: .github/workflows/branch.yml
• files_exist - File found: .github/workflows/ci.yml
• files_exist - File found: .github/workflows/linting_comment.yml
• files_exist - File found: .github/workflows/linting.yml
• files_exist - File found: assets/email_template.html
• files_exist - File found: assets/email_template.txt
• files_exist - File found: assets/sendmail_template.txt
• files_exist - File found: conf/modules.config
• files_exist - File found: conf/test.config
• files_exist - File found: conf/test_full.config
• files_exist - File found: docs/output.md
• files_exist - File found: docs/README.md
• files_exist - File found: docs/README.md
• files_exist - File found: docs/usage.md
• files_exist - File found: main.nf
• files_exist - File found: assets/multiqc_config.yml
• files_exist - File found: conf/base.config
• files_exist - File found: conf/igenomes.config
• files_exist - File found: modules.json
• files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
• files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
• files_exist - File not found check: .github/workflows/push_dockerhub.yml
• files_exist - File not found check: .markdownlint.yml
• files_exist - File not found check: .nf-core.yaml
• files_exist - File not found check: .yamllint.yml
• files_exist - File not found check: bin/markdown_to_html.r
• files_exist - File not found check: conf/aws.config
• files_exist - File not found check: docs/images/nf-core-snvphylnfc_logo.png
• files_exist - File not found check: lib/Checks.groovy
• files_exist - File not found check: lib/Completion.groovy
• files_exist - File not found check: lib/Workflow.groovy
• files_exist - File not found check: parameters.settings.json
• files_exist - File not found check: pipeline_template.yml
• files_exist - File not found check: Singularity
• files_exist - File not found check: lib/nfcore_external_java_deps.jar
• files_exist - File not found check: .travis.yml
• nextflow_config - Config variable found: manifest.nextflowVersion
• nextflow_config - Config variable found: manifest.description
• nextflow_config - Config variable found: manifest.version
• nextflow_config - Config variable found: timeline.enabled
• nextflow_config - Config variable found: trace.enabled
• nextflow_config - Config variable found: report.enabled
• nextflow_config - Config variable found: dag.enabled
• nextflow_config - Config variable found: process.cpus
• nextflow_config - Config variable found: process.memory
• nextflow_config - Config variable found: process.time
• nextflow_config - Config variable found: params.outdir
• nextflow_config - Config variable found: params.input
• nextflow_config - Config variable found: params.validationShowHiddenParams
• nextflow_config - Config variable found: params.validationSchemaIgnoreParams
• nextflow_config - Config variable found: manifest.mainScript
• nextflow_config - Config variable found: timeline.file
• nextflow_config - Config variable found: trace.file
• nextflow_config - Config variable found: report.file
• nextflow_config - Config variable found: dag.file
• nextflow_config - Config variable (correctly) not found: params.nf_required_version
• nextflow_config - Config variable (correctly) not found: params.container
• nextflow_config - Config variable (correctly) not found: params.singleEnd
• nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
• nextflow_config - Config variable (correctly) not found: params.name
• nextflow_config - Config variable (correctly) not found: params.enable_conda
• nextflow_config - Config timeline.enabled had correct value: true
• nextflow_config - Config report.enabled had correct value: true
• nextflow_config - Config trace.enabled had correct value: true
• nextflow_config - Config dag.enabled had correct value: true
• nextflow_config - Config dag.file ended with .html
• nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
• nextflow_config - nextflow.config contains configuration profile test
• nextflow_config - Config default value correct: params.metadata_1_header= metadata_1
• nextflow_config - Config default value correct: params.metadata_2_header= metadata_2
• nextflow_config - Config default value correct: params.metadata_3_header= metadata_3
• nextflow_config - Config default value correct: params.metadata_4_header= metadata_4
• nextflow_config - Config default value correct: params.metadata_5_header= metadata_5
• nextflow_config - Config default value correct: params.metadata_6_header= metadata_6
• nextflow_config - Config default value correct: params.metadata_7_header= metadata_7
• nextflow_config - Config default value correct: params.metadata_8_header= metadata_8
• nextflow_config - Config default value correct: params.min_coverage_depth= 15
• nextflow_config - Config default value correct: params.min_mapping_percent_cov= 80
• nextflow_config - Config default value correct: params.min_mean_mapping_quality= 30
• nextflow_config - Config default value correct: params.window_size= 500
• nextflow_config - Config default value correct: params.density_threshold= 2
• nextflow_config - Config default value correct: params.snv_abundance_ratio= 0.75
• nextflow_config - Config default value correct: params.min_repeat_length= 150
• nextflow_config - Config default value correct: params.min_repeat_pid= 90
• nextflow_config - Config default value correct: params.max_cpus= 4
• nextflow_config - Config default value correct: params.max_memory= 2.GB
• nextflow_config - Config default value correct: params.max_time= 1.h
• nextflow_config - Config default value correct: params.publish_dir_mode= copy
• nextflow_config - Config default value correct: params.validate_params= true
• files_unchanged - .gitattributes matches the template
• files_unchanged - .prettierrc.yml matches the template
• files_unchanged - .github/.dockstore.yml matches the template
• files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
• files_unchanged - .github/workflows/linting_comment.yml matches the template
• files_unchanged - .gitignore matches the template
• files_unchanged - .prettierignore matches the template
• actions_ci - '.github/workflows/ci.yml' is triggered on expected events
• actions_ci - '.github/workflows/ci.yml' checks minimum NF version
• readme - README Zenodo placeholder was replaced with DOI.
• pipeline_todos - No TODO strings found
• template_strings - Did not find any Jinja template strings (119 files)
• schema_lint - Schema lint passed
• schema_lint - Input mimetype lint passed: 'text/csv'
• schema_params - Schema matched params returned from nextflow config
• system_exit - No System.exit calls found
• actions_schema_validation - Workflow validation passed: branch.yml
• actions_schema_validation - Workflow validation passed: linting_comment.yml
• actions_schema_validation - Workflow validation passed: linting.yml
• actions_schema_validation - Workflow validation passed: ci.yml
• merge_markers - No merge markers found in pipeline files
• modules_json - Only installed modules found in modules.json
• multiqc_config - assets/multiqc_config.yml found and not ignored.
• multiqc_config - assets/multiqc_config.yml contains report_section_order
• multiqc_config - assets/multiqc_config.yml contains export_plots
• multiqc_config - assets/multiqc_config.yml contains report_comment
• multiqc_config - assets/multiqc_config.yml follows the ordering scheme of the minimally required plugins.
• multiqc_config - assets/multiqc_config.yml contains 'export_plots: true'.
• modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'
• base_config - conf/base.config found and not ignored.
• base_config - CUSTOM_DUMPSOFTWAREVERSIONS found in conf/base.config and Nextflow scripts.
• modules_config - conf/modules.config found and not ignored.
• modules_config - CUSTOM_DUMPSOFTWAREVERSIONS found in conf/modules.config and Nextflow scripts.
• modules_config - CONSOLIDATE_BCFS found in conf/modules.config and Nextflow scripts.
• nfcore_yml - Repository type in .nf-core.yml is valid: pipeline

Run details

• nf-core/tools version 2.14.1
• Run at 2024-07-09 14:06:38

[apetkau]

This looks amazing. Thanks so much @kylacochrane 😄

I have a few comments below. In addition:

1. Could you add a message in the CHANGELOG.md about this change? Could you also reference this PR in that message (like what is done for the mikrokondo changelog: https://github.com/phac-nml/mikrokondo/blob/main/CHANGELOG.md)?
2. I am wondering if SNVPhyl will work even if all the metadata_1 ... metadata_8 columns are left out of the samplesheet.csv? It would be nice if it did, since this update would be backwards-compatible with the previous samplesheet format. Could you test this out?

[kylacochrane]

I can confirm that the pipeline functions correctly even without metadata columns, ensuring backward compatibility of the input samplesheet. You can find the relevant test here: aa4f291

Additionally, another test was added that includes only three metadata columns, renames those columns, and provides partial metadata: aa4f291

The CHANGELOG.md was updated here: 94c4f3c

[apetkau]

Thanks so much for all your great work Kyla 😄 . I have two more small comments.

[apetkau]

Thanks so much Kyla

[emarinier]

Looks great! Thanks Kyla.