Support for grid view (and a few other things)

[muffato]

The original purpose (TOLIT-2292) was to generate additional metadata in the blobDir to support the grid view that Karen would like to embed into the genome notes.

Talking to Rich, this is essentially the last major feature that needs to be implemented for the v1.0 release, so I had a go at all the other minor changes I wanted for v1.0:

• The blobDir metadata needs to record whether the reads are paired or single (TOLIT-2525)
• Updated the documentation (TOLIT-2522)
• "Publish" all important output files (TOLIT-2521)
• Convert input BAM/CRAM files to Fasta on the fly into minimap2. This saves a lot of space and time (I/O)

PR checklist

• This comment contains a description of changes (with reason).
• If you've fixed a bug or added code that should be tested, add tests!
• If you've added a new tool - have you followed the pipeline conventions in the contribution docs
• Make sure your code lints (nf-core lint).
• Ensure the test suite passes (nextflow run . -profile test,docker --outdir <OUTDIR>).
• Check for unexpected warnings in debug mode (nextflow run . -profile debug,test,docker --outdir <OUTDIR>).
• Usage Documentation in docs/usage.md is updated.
• Output Documentation in docs/output.md is updated.
• CHANGELOG.md is updated.
• README.md is updated (including new tool citations and authors/contributors).

[github-actions]

nf-core lint overall result: Passed ✅ ⚠️

Posted for pipeline commit 1909646

+| ✅ 128 tests passed       |+
#| ❔  29 tests were ignored |#
!| ❗   1 tests had warnings |!

❗ Test warnings:

• nextflow_config - Config manifest.version should end in dev: 0.7.0

❔ Tests ignored:

• files_exist - File is ignored: CODE_OF_CONDUCT.md
• files_exist - File is ignored: assets/nf-core-blobtoolkit_logo_light.png
• files_exist - File is ignored: docs/images/nf-core-blobtoolkit_logo_light.png
• files_exist - File is ignored: docs/images/nf-core-blobtoolkit_logo_dark.png
• files_exist - File is ignored: .github/ISSUE_TEMPLATE/config.yml
• files_exist - File is ignored: .github/workflows/awstest.yml
• files_exist - File is ignored: .github/workflows/awsfulltest.yml
• files_exist - File is ignored: conf/igenomes.config
• nextflow_config - Config variable ignored: manifest.name
• nextflow_config - Config variable ignored: manifest.homePage
• files_unchanged - File ignored due to lint config: CODE_OF_CONDUCT.md
• files_unchanged - File ignored due to lint config: LICENSE or LICENSE.md or LICENCE or LICENCE.md
• files_unchanged - File ignored due to lint config: .github/CONTRIBUTING.md
• files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/bug_report.yml
• files_unchanged - File does not exist: .github/ISSUE_TEMPLATE/config.yml
• files_unchanged - File ignored due to lint config: .github/PULL_REQUEST_TEMPLATE.md
• files_unchanged - File ignored due to lint config: .github/workflows/branch.yml
• files_unchanged - File ignored due to lint config: .github/workflows/linting_comment.yml
• files_unchanged - File ignored due to lint config: .github/workflows/linting.yml
• files_unchanged - File ignored due to lint config: assets/email_template.html
• files_unchanged - File ignored due to lint config: assets/email_template.txt
• files_unchanged - File ignored due to lint config: assets/sendmail_template.txt
• files_unchanged - File ignored due to lint config: assets/nf-core-blobtoolkit_logo_light.png
• files_unchanged - File ignored due to lint config: docs/images/nf-core-blobtoolkit_logo_light.png
• files_unchanged - File ignored due to lint config: docs/images/nf-core-blobtoolkit_logo_dark.png
• files_unchanged - File ignored due to lint config: lib/NfcoreTemplate.groovy
• actions_awstest - 'awstest.yml' workflow not found: /home/runner/work/blobtoolkit/blobtoolkit/.github/workflows/awstest.yml
• template_strings - template_strings
• merge_markers - merge_markers

✅ Tests passed:

• files_exist - File found: .gitattributes
• files_exist - File found: .gitignore
• files_exist - File found: .nf-core.yml
• files_exist - File found: .editorconfig
• files_exist - File found: .prettierignore
• files_exist - File found: .prettierrc.yml
• files_exist - File found: CHANGELOG.md
• files_exist - File found: CITATIONS.md
• files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
• files_exist - File found: nextflow_schema.json
• files_exist - File found: nextflow.config
• files_exist - File found: README.md
• files_exist - File found: .github/.dockstore.yml
• files_exist - File found: .github/CONTRIBUTING.md
• files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
• files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
• files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
• files_exist - File found: .github/workflows/branch.yml
• files_exist - File found: .github/workflows/ci.yml
• files_exist - File found: .github/workflows/linting_comment.yml
• files_exist - File found: .github/workflows/linting.yml
• files_exist - File found: assets/email_template.html
• files_exist - File found: assets/email_template.txt
• files_exist - File found: assets/sendmail_template.txt
• files_exist - File found: conf/modules.config
• files_exist - File found: conf/test.config
• files_exist - File found: conf/test_full.config
• files_exist - File found: docs/output.md
• files_exist - File found: docs/README.md
• files_exist - File found: docs/README.md
• files_exist - File found: docs/usage.md
• files_exist - File found: lib/nfcore_external_java_deps.jar
• files_exist - File found: lib/NfcoreTemplate.groovy
• files_exist - File found: lib/Utils.groovy
• files_exist - File found: lib/WorkflowMain.groovy
• files_exist - File found: main.nf
• files_exist - File found: assets/multiqc_config.yml
• files_exist - File found: conf/base.config
• files_exist - File found: lib/WorkflowBlobtoolkit.groovy
• files_exist - File found: modules.json
• files_exist - File found: pyproject.toml
• files_exist - File not found check: Singularity
• files_exist - File not found check: parameters.settings.json
• files_exist - File not found check: pipeline_template.yml
• files_exist - File not found check: .nf-core.yaml
• files_exist - File not found check: bin/markdown_to_html.r
• files_exist - File not found check: conf/aws.config
• files_exist - File not found check: .github/workflows/push_dockerhub.yml
• files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
• files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
• files_exist - File not found check: docs/images/nf-core-blobtoolkit_logo.png
• files_exist - File not found check: .markdownlint.yml
• files_exist - File not found check: .yamllint.yml
• files_exist - File not found check: lib/Checks.groovy
• files_exist - File not found check: lib/Completion.groovy
• files_exist - File not found check: lib/Workflow.groovy
• files_exist - File not found check: .travis.yml
• nextflow_config - Config variable found: manifest.nextflowVersion
• nextflow_config - Config variable found: manifest.description
• nextflow_config - Config variable found: manifest.version
• nextflow_config - Config variable found: timeline.enabled
• nextflow_config - Config variable found: trace.enabled
• nextflow_config - Config variable found: report.enabled
• nextflow_config - Config variable found: dag.enabled
• nextflow_config - Config variable found: process.cpus
• nextflow_config - Config variable found: process.memory
• nextflow_config - Config variable found: process.time
• nextflow_config - Config variable found: params.outdir
• nextflow_config - Config variable found: params.input
• nextflow_config - Config variable found: params.validationShowHiddenParams
• nextflow_config - Config variable found: params.validationSchemaIgnoreParams
• nextflow_config - Config variable found: manifest.mainScript
• nextflow_config - Config variable found: timeline.file
• nextflow_config - Config variable found: trace.file
• nextflow_config - Config variable found: report.file
• nextflow_config - Config variable found: dag.file
• nextflow_config - Config variable (correctly) not found: params.nf_required_version
• nextflow_config - Config variable (correctly) not found: params.container
• nextflow_config - Config variable (correctly) not found: params.singleEnd
• nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
• nextflow_config - Config variable (correctly) not found: params.name
• nextflow_config - Config variable (correctly) not found: params.enable_conda
• nextflow_config - Config timeline.enabled had correct value: true
• nextflow_config - Config report.enabled had correct value: true
• nextflow_config - Config trace.enabled had correct value: true
• nextflow_config - Config dag.enabled had correct value: true
• nextflow_config - Config dag.file ended with .html
• nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
• nextflow_config - Config params.custom_config_version is set to master
• nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
• nextflow_config - Lines for loading custom profiles found
• nextflow_config - nextflow.config contains configuration profile test
• files_unchanged - .gitattributes matches the template
• files_unchanged - .prettierrc.yml matches the template
• files_unchanged - .github/.dockstore.yml matches the template
• files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
• files_unchanged - docs/README.md matches the template
• files_unchanged - lib/nfcore_external_java_deps.jar matches the template
• files_unchanged - .gitignore matches the template
• files_unchanged - .prettierignore matches the template
• files_unchanged - pyproject.toml matches the template
• actions_ci - '.github/workflows/ci.yml' is triggered on expected events
• actions_ci - '.github/workflows/ci.yml' checks minimum NF version
• readme - README Nextflow minimum version badge matched config. Badge: 23.04.0, Config: 23.04.0
• readme - README Zenodo placeholder was replaced with DOI.
• pipeline_todos - No TODO strings found
• pipeline_name_conventions - Name adheres to nf-core convention
• schema_lint - Schema lint passed
• schema_lint - Schema title + description lint passed
• schema_lint - Input mimetype lint passed: 'text/csv'
• schema_params - Schema matched params returned from nextflow config
• system_exit - No System.exit calls found
• actions_schema_validation - Workflow validation passed: ci.yml
• actions_schema_validation - Workflow validation passed: linting_comment.yml
• actions_schema_validation - Workflow validation passed: release-announcements.yml
• actions_schema_validation - Workflow validation passed: sanger_test_full.yml
• actions_schema_validation - Workflow validation passed: linting.yml
• actions_schema_validation - Workflow validation passed: clean-up.yml
• actions_schema_validation - Workflow validation passed: fix-linting.yml
• actions_schema_validation - Workflow validation passed: sanger_test.yml
• actions_schema_validation - Workflow validation passed: branch.yml
• modules_json - Only installed modules found in modules.json
• multiqc_config - 'assets/multiqc_config.yml' contains report_section_order
• multiqc_config - 'assets/multiqc_config.yml' contains export_plots
• multiqc_config - 'assets/multiqc_config.yml' contains report_comment
• multiqc_config - 'assets/multiqc_config.yml' follows the ordering scheme of the minimally required plugins.
• multiqc_config - 'assets/multiqc_config.yml' contains 'export_plots: true'.
• modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'

Run details

• nf-core/tools version 2.11
• Run at 2024-10-02 07:29:46

[muffato]

Edit: I've decided that I won't be calling this v1.0 just yet. I think it's imperative to sort out the resource usage first (Hi-C minimap2 alignments shouldn't be going to the basement queue) and the blastn issue.

[tkchafin]

Sent all of my changes as commits