Merge/bnbowman master upstream master (#35)

* Try prefixing analysis dirs * Update CHANGELOG.md * tried to add conditional execution to actions * Update CHANGELOG.md * Add dependency statement * only overwrite id when not set * changelog * always set c_id * Revert setting force_interactive flag for rich with --no-ansi * Don't force terminal escape codes for the progress bar * Extend kallisto module regex to recognize newer output I've noticed multiqc (v 1.12) didn't recognise some kallisto output (I'm using kb_python 0.26.4). Having digged a bit, it seems to not work with more recent kallisto output, i.e. this snippet taken from here: https://github.com/pachterlab/GRNP_2020/blob/daed9c2f204f1c3f6ee0e864c3db93b0baadfc8a/notebooks/FASTQ_processing/ProcessPBMC_NG.ipynb ``` [index] k-mer length: 31 [index] number of targets: 187,626 [index] number of k-mers: 108,619,921 tcmalloc: large alloc 3221225472 bytes == 0x556459b7e000 @ 0x7feac4ab5887 0x556458814ad2 0x55645880d061 0x5564587e1372 0x7feac3935bf7 0x5564587e60da [index] number of equivalence classes: 752,021 [quant] will process sample 1: A_R1.gz A_R2.gz [quant] will process sample 2: B_R1.gz B_R2.gz [quant] finding pseudoalignments for the reads ... done [quant] processed 170,526,037 reads, 98,632,205 reads pseudoaligned ``` Turns out multiqc only looks for `pair|file` but not sample. Replacing sample for file did do the trick, hence I suggest to add `sample` in the regex pattern. I haven't tested this, but it should work now. Here is the code which generates this output: https://github.com/pachterlab/kallisto/blob/83bde908c403ea4014b5092a243e5c7240f48dd5/src/ProcessReads.cpp#L235 This is the commit which introduced it (already in 2018, so not sure why this hasn't been caught yet) pachterlab/kallisto@62e9464 * Replace logger.hasHandlers() with logger.handlers There are cases where configuring logging results in logger.handlers being empty but logger.hasHandlers() returns True: MultiQC#1643 Since the block modified removes based on logger.handlers, the condition to enter the block should check logger.handlers rather than logger.hasHandlers() * Added description of changes for pull request * Document 'no_version_check' config option * Docs tweak * Fix kwargs for MultiQC plugins * New config option 'custom_table_header_config' * Run black * Update adapterRemoval.py Returns actual proportion of reads that were collapsed and discarded * Black format * Black format * BlackPython * Fix chart labels and titles * Fix chart labels and titles * Add columns to stats table Add columns with proportion of collapsed/discarded reads to the general stats table * Add Columns - Fix format * Changelog * Fixed bug when other fields also have a "-" instead of an integer. * Updated CHANGELOG * Fixed typos * Fixed format typo * Fixed format typo * Nanostat: Remove HTML escaping Jinja2 escape() function removed in jinja2 v3.10 I don't think that this escaping should be required. I can't see any effect in the report when I remove it anyway. * Changelog * Changing 0 to None * Skip fields with `-` * Pangolin 4.0 compatability Recently pangolin has been updated to version 4.0 and this changes the output CSV file - see: https://github.com/cov-lineages/pangolin/releases/tag/v4.0 This causes the module to fail in its current state as row['qc_status'] already exists and the current replacement triggers a key error by searching for row['status'] which no longer exists. Thanks to @alexomics for tracking down the issue. * Don't duplicate custom-content section descriptions. Fixed edge-case bug in custom content where a `description` that doesn't terminate in `.` gave duplicate section descriptions. * Changelog * Tidied the verbose log, added summaries for skipped search files to debug log * Allow sorting of table columns with text contents * update changelog * optimize linegraph category comparison * Somalier: division by zero in sex ploidy plot * Changelog * Add time zone * Update changelog * Fix typo in bcl2fastq.py * Handle too long and low complexity * update changelog * fix zero division error in sambamba markdup module * black formatting * update CHANGELOG.md to address MultiQC#1654 * bclconvert checks RunInfo xml if reads are singleend or pairedend and sets clusterlength appropriately. resolves MultiQC#1697 * Added CITATION.cff file for standardized citations * fixed formatting of url * fixed citation formatting * Run prettier * Fix module crashing due to missing field in report * Fix bug where module wouldn't run if all content was within a MultiQC config file Fixes MultiQC#1686 * nanostat: add check for quality scores * update CHANGELOG.md * update CHANGELOG.md * Custom content: Fix crash when 'info' isn't set Closes MultiQC#1688 * Added nix flake support * Update docs/installation.md Co-authored-by: Phil Ewels <[email protected]> * Fix zero division error * Update fastqc.py * Update fastqc.py * fix format * add change log * fix doc ref * Don't need Prettier _and_ markdown/yamllint CI * Just capture the ValueError * Rich-codex screenshot in the readme * Corrected 'outdir' flag Missing a dash for the flag to work. * Clean up clean_img_paths * Generate new screengrabs with rich-codex * Add samtools flagstat column '% Read Mapped' * update samtools flagstat changelog * Added try,except for divisions to avoid division by 0 errors * added the fixing of malt in the change log * report median read length for fastqc * add after filtering total reads to general stats table * GitHub Actions: Tweet about new releases * Bump to v1.13 for release * rich-codex screenshots: Manual only, skip git checks * Generate new screengrabs with rich-codex * Fix changelog date * Bump to v1.14dev * Custom content: Render report even if there's only general stats there See MultiQC#1756 * Bugfix: Make `config.data_format` work again * Bump minimum version of Jinja2 to `>=3.0.0` Closes MultiQC#1642 * Disable search progress bar if running with `--quiet` or `--no-ansi` Closes MultiQC#1638 * Attempt to cooerce line / scatter x-axes into floats so as not to lose labels See MultiQC#1242 * Use row 1 as x-axis labels if no sample name. Closes MultiQC#1242 * Malt: Move changelog up to new version * Merge changelog up * Use OrderedDict instead of 'placement' * Add code comment * Add CI testing for Python 3.10 and 3.11 * Fix typo * Quotes so it's 3.10 and not 3.1 * 3.11-dev * Remove 3.11-dev for Windows * Move merge markers GHA into lint workflow file * Shorter job name * Be more selective about when slow MultiQC test runs fire - Master only for push event - Don't run if only docs / markdown * Run isort * Remove py2 'from __future__ import print_function' * Add GitHub actions CI for isort * Changelog * Remove all py2 'from __future__ imports' * Tweak some imports * Changelog * added setuptools to flake * rm emtpy bcftools stats variant depths plot * moved changelog comment * adjusted PR num * fix duplicate heatmap for kraken * changelog * use None instead * First commit of pre-commit * Comment out all the tests that don't yet work * Update gene_body_coverage.py Using a normalized coverage to make genebody coverage plot ( similar to the method used by RSeQC). Us the formula 'norm_cov = ( cov - min(cov ) / ( max(cov) - min(cov) )' to compute normalized coverage * Update gene_body_coverage.py * Update CHANGELOG.md * Update CHANGELOG.md * Update CHANGELOG.md * Move changelog entry * Test for Python 3.11 now that the official release is out * CI: Use new version of actions/checkout to avoid Node.js depreciation warning * Remove sample and chromosome before converting to int This fixes issue-1793 * Remove filtered samples from general stats table This fixes MultiQC#1780 * Update changelog * Add additional entries for qualimap when region stats present * fastp: use passed filter reads instead of after filter total reads Signed-off-by: Josh Chorlton <[email protected]> * bclconvert now handles different r1 and r2 lengths instead of assuming they are the same * updated CHANGELOG.md * update bustools * Update CHANGELOG.md * Remove changelog entry * Move changelog to entry to correct place * Fix changelog * Kraken: Improve heatmap config * Apply suggestions from code review Co-authored-by: Phil Ewels <[email protected]> * handle singleindex data * cleanup * CHANGELOG.md bclconvert fix issue link typo and note single-index paired-end data handled * Qualimap BamQC: Refactor to parse regexes per section Also: Fix randomly aggressive Snippy module parsing bug * HsMetrics: Allow custom columns in General Stats too * Replace nested loop with list comprehension when parsing output file headers * CHANGELOG * Output headers order preserved and Sample is first value * Fix ubuntu version in GitHub CI to preserve Py3.6 testing. Python 3.6, I think your days are numbered.. * add back original avg field Signed-off-by: Josh Chorlton <[email protected]> * fixes Signed-off-by: Josh Chorlton <[email protected]> * update busco colors Signed-off-by: Josh Chorlton <[email protected]> * fix: frontmatter yaml formatting issue * Update docs to use --cl-config instead of --cl_config Closes MultiQC#1825 * Update multiqc/modules/fastqc/fastqc.py Co-authored-by: Phil Ewels <[email protected]> * Update multiqc/modules/fastqc/fastqc.py Co-authored-by: Phil Ewels <[email protected]> * Update multiqc/modules/fastqc/fastqc.py Co-authored-by: Phil Ewels <[email protected]> * suggestion Signed-off-by: Josh Chorlton <[email protected]> Co-authored-by: Erik Danielsson <[email protected]> Co-authored-by: Phil Ewels <[email protected]> Co-authored-by: Ido Tamir <[email protected]> Co-authored-by: seb-mueller <[email protected]> Co-authored-by: Jonathan Oribello <[email protected]> Co-authored-by: NiemannJ <[email protected]> Co-authored-by: fgvieira <[email protected]> Co-authored-by: mattloose <[email protected]> Co-authored-by: Josh Chorlton <[email protected]> Co-authored-by: vladsaveliev <[email protected]> Co-authored-by: Sam Chorlton <> Co-authored-by: jethror1 <[email protected]> Co-authored-by: Garth Kong <[email protected]> Co-authored-by: Andrei Seleznev <[email protected]> Co-authored-by: lew2mz <[email protected]> Co-authored-by: Phil Ewels <[email protected]> Co-authored-by: phue <[email protected]> Co-authored-by: David Lewis <[email protected]> Co-authored-by: Chang Y <[email protected]> Co-authored-by: beausoleilmo <[email protected]> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Jean Mainguy <[email protected]> Co-authored-by: aidaanva <[email protected]> Co-authored-by: SusiJo <[email protected]> Co-authored-by: Phil Ewels <[email protected]> Co-authored-by: TNalpat <[email protected]> Co-authored-by: Redmar van den Berg <[email protected]> Co-authored-by: James Fellows Yates <[email protected]> Co-authored-by: Maarten-vd-Sande <[email protected]> Co-authored-by: Adam Talbot <[email protected]> Co-authored-by: Oleh Pratsko <[email protected]> Co-authored-by: Josh Chorlton <[email protected]>
bnbowman · Dec 20, 2022 · ae557c4 · ae557c4
1 parent 28c4b73
commit ae557c4
Show file tree

Hide file tree

Showing 343 changed files with 1,833 additions and 1,101 deletions.
diff --git a/.github/RELEASE_CHECKLIST.md b/.github/RELEASE_CHECKLIST.md
@@ -33,11 +33,14 @@ This checklist is for my own reference, as I forget the steps every time.
    - Upload to the website and push change to Git
 
 8. Commit and push version updates
-9. Make a [release](https://github.com/ewels/MultiQC/releases) on GitHub - paste changelog section.
-10. Check that [PyPI listing page](https://pypi.python.org/pypi/multiqc/) looks sane
-11. Update version numbers to new dev version in `setup.py` + a new section in the changelog for the development version
-12. Commit and push version bump
-13. Make a new release on `bioconda` (assuming new modules were added):
+9. Make a new branch and run the rich-codex screenshots workflow on it manually
+10. PR / merge this branch into master
+11. Make sure that all tests are passing
+12. Make a [release](https://github.com/ewels/MultiQC/releases) on GitHub - paste changelog section.
+13. Check that [PyPI listing page](https://pypi.python.org/pypi/multiqc/) looks sane
+14. Update version numbers to new dev version in `setup.py` + a new section in the changelog for the development version
+15. Commit and push version bump
+16. Make a new release on `bioconda` (assuming new modules were added):
 
     ```bash
     # Update to latest bioconda
@@ -68,6 +71,6 @@ This checklist is for my own reference, as I forget the steps every time.
     # Submit a Pull Request and merge
     ```
 
-14. Tell UPPMAX about the new version and ask for the module system to be updated.
-15. Tweet that new version is released
-16. Continue making more awesome :metal:
+17. Tell UPPMAX about the new version and ask for the module system to be updated.
+18. Tweet that new version is released
+19. Continue making more awesome :metal:
diff --git a/.github/workflows/code_checks.py b/.github/workflows/code_checks.py
@@ -1,11 +1,12 @@
 #!/usr/bin/env python
 """ Crude code-quality checks for running in CI """
 
-from rich import print
 import glob
 import os
 import sys
 
+from rich import print
+
 BASE_DIR = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 MODULES_DIR = os.path.join(BASE_DIR, "multiqc", "modules")
 num_errors = 0

diff --git a/.github/workflows/deploy-pypi.yml b/.github/workflows/deploy-pypi.yml
@@ -8,7 +8,7 @@ jobs:
     runs-on: ubuntu-latest
 
     steps:
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v3
         name: Check out source-code repository
 
       - name: Set up Python 3.7
@@ -26,7 +26,20 @@ jobs:
 
       - name: Publish MultiQC to PyPI
         if: github.repository == 'ewels/MultiQC'
-        uses: pypa/gh-action-pypi-publish@master
+        uses: pypa/gh-action-pypi-publish@release/v1
         with:
-          user: __token__
           password: ${{ secrets.PYPI_API_TOKEN }}
+
+      - name: Tweet
+        if: github.repository == 'ewels/MultiQC'
+        uses: snow-actions/[email protected]
+        with:
+          status: |
+            A new version of @MultiQC has just been released! ${{ github.event.release.name }}
+
+            Read the changelog here: ${{ github.event.release.html_url }}
+        env:
+          CONSUMER_API_KEY: ${{ secrets.TWITTER_CONSUMER_API_KEY }}
+          CONSUMER_API_SECRET_KEY: ${{ secrets.TWITTER_CONSUMER_API_SECRET_KEY }}
+          ACCESS_TOKEN: ${{ secrets.TWITTER_ACCESS_TOKEN }}
+          ACCESS_TOKEN_SECRET: ${{ secrets.TWITTER_ACCESS_TOKEN_SECRET }}
diff --git a/.github/workflows/docker.yml b/.github/workflows/docker.yml
@@ -13,7 +13,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Check out the repo
-        uses: actions/checkout@v2
+        uses: actions/checkout@v3
 
       - name: Log in to Docker Hub
         uses: docker/login-action@v1

diff --git a/.github/workflows/lint_code.yml b/.github/workflows/lint_code.yml
@@ -1,64 +1,20 @@
 name: Lint Code Base
-on: [push, pull_request]
+on:
+  push:
+    branches: [master]
+  pull_request:
 
 jobs:
-  Markdown:
+  pre-commit:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v2
-
-      - uses: actions/setup-node@v2
-        with:
-          node-version: "12"
-
-      - name: Install markdownlint
-        run: npm install -g markdownlint-cli
-
-      - name: Run Markdownlint
-        run: markdownlint ${GITHUB_WORKSPACE}
-
-  YAML:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v2
-
-      - uses: actions/setup-node@v2
-        with:
-          node-version: "12"
-
-      - name: Install yaml-lint
-        run: npm install -g yaml-lint
-
-      - name: Run yaml-lint
-        run: yamllint $(find ${GITHUB_WORKSPACE} -type f -name "*.yml")
-
-  Prettier:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v2
-
-      - uses: actions/setup-node@v2
-        with:
-          node-version: "12"
-
-      - name: Install Prettier
-        run: npm install -g prettier
-
-      - name: Run Prettier --check
-        run: prettier --check ${GITHUB_WORKSPACE}
-
-  PythonBlack:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v2
-
-      - name: Check code lints with Black
-        uses: jpetrucciani/black-check@master
-
+      - uses: actions/checkout@v3
+      - uses: actions/setup-python@v3
+      - uses: pre-commit/[email protected]
   CodeChecks:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v3
 
       - name: Install dependencies
         run: pip install rich

diff --git a/.github/workflows/merge_markers.yml b/.github/workflows/merge_markers.yml
diff --git a/.github/workflows/multiqc_linux.yml b/.github/workflows/multiqc_linux.yml
@@ -1,52 +1,97 @@
 name: "MultiQC - Linux"
-on: [push, pull_request]
+on:
+  push:
+    branches:
+      - master
+    paths-ignore:
+      - "docs/**"
+      - "*.md"
+  pull_request:
+    paths-ignore:
+      - "docs/**"
+      - "*.md"
 
 jobs:
+  changes:
+    name: Check for changes
+    runs-on: ubuntu-latest
+    outputs:
+      files_changed: ${{ steps.filter.outputs.changes }}
+      n_files_changes: ${{ steps.filter.outputs.changes.all_files_count}}
+      modules: ${{ steps.filter.outputs.changes.modules }}
+      n_modules: ${{ steps.filter.outputs.changes.modules_count }}
+      base_module: ${{ steps.filter.outputs.changes.base_module }}
+      changelog: ${{ steps.filter.outputs.changes.changelog }}
+
+    steps:
+      - uses: actions/checkout@v3
+
+      - uses: dorny/paths-filter@v2
+        id: filter
+        with:
+          filters: |
+            modules:
+              - "multiqc/modules/**"
+            base_module:
+              - "multiqc/modules/base_module.py"
+            changelog:
+              - "CHANGELOG.md"
+            all_files:
+              - "**"
+
   run_multiqc:
     name: Linux - Python ${{ matrix.python-version }}
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-20.04
+    needs: changes
     strategy:
       matrix:
-        python-version: [3.6, 3.7, 3.8, 3.9]
+        python-version: ["3.6", "3.7", "3.8", "3.9", "3.10", "3.11"]
     timeout-minutes: 10
 
     steps:
-      # Check out MultiQC code
-      - uses: actions/checkout@v2
+      - name: Check out MultiQC code
+        uses: actions/checkout@v3
 
-      # Set up Python
       - name: Set up Python ${{ matrix.python-version }}
         uses: actions/setup-python@v2
         with:
           python-version: ${{ matrix.python-version }}
 
-      # Update pip and install beautifulsoup4 for CI tests (CSP checking)
+      # For CSP checking
       - name: Install dependencies for CI tests
-        run: |
-          python -m pip install --upgrade pip setuptools beautifulsoup4
+        run: python -m pip install --upgrade pip setuptools beautifulsoup4
 
-      # Install MultiQC
       - name: Install MultiQC
         run: pip install .
 
-      # Fetch the MultiQC test data
       - name: Download test data
-        run: |
-          git clone https://github.com/ewels/MultiQC_TestData.git test_data
+        uses: actions/checkout@v3
+        with:
+          repository: ewels/MultiQC_TestData
+          path: test_data
+
+      # Set the `-m` flag if only one module has changed
+      - name: "Get a `-m` flag for the `multiqc` if only one module has changed"
+        uses: haya14busa/action-cond@v1
+        id: single_module
+        with:
+          cond: ${{ ! needs.changes.outputs.base_module && needs.changes.outputs.n_modules == 1 && (needs.changes.outputs.n_files_changes == 1 || needs.changes.outputs.n_files_changes == 2 && needs.changes.outputs.changelog)}}
+          if_true: "-m ${{ needs.changes.outputs.modules[0] }}"
+          if_false: ""
 
       # Run all of the tests!
       - name: Special case input data
-        run: multiqc test_data/data --ignore test_data/data/modules/
+        run: multiqc ${{ steps.single_module.outputs.value }} test_data/data --ignore test_data/data/modules/
 
       - name: All modules / Custom report filename
-        run: multiqc --lint test_data/data/modules/ --filename full_report.html
+        run: multiqc ${{ steps.single_module.outputs.value }} --lint test_data/data/modules/ --filename full_report.html
 
       - name: All modules / Log filename as s_name, no cleaning
-        run: multiqc --lint test_data/data/modules/ --fullnames --fn_as_s_name
+        run: multiqc ${{ steps.single_module.outputs.value }} --lint test_data/data/modules/ --fullnames --fn_as_s_name
 
       - name: Filter out all filenames (confirm no report)
         run: |
-          multiqc test_data/data/modules/ --filename all_ignored.html --ignore-samples '*'
+          multiqc ${{ steps.single_module.outputs.value }} test_data/data/modules/ --filename all_ignored.html --ignore-samples '*'
           [[ ! -f all_ignored.html ]]
 
       - name: File-list input of dirs
@@ -58,7 +103,7 @@ jobs:
         run: multiqc --lint test_data/data/modules/ -m fastqc -f -d -dd 1 -i "Forced Report" -b "This command has lots of options" --filename custom_fn --no-data-dir
 
       - name: For overwrite / Flat plots / Tagged modules only / Exclude module / Ignore samples / Full names / zip data dir / config file
-        run: multiqc --lint test_data/data/modules/ -f --flat --tag methylation --exclude clusterflow --ignore-samples ngi --fullnames --zip-data-dir -c test/config_example.yaml
+        run: multiqc ${{ steps.single_module.outputs.value }} --lint test_data/data/modules/ -f --flat --tag methylation --exclude clusterflow --ignore-samples ngi --fullnames --zip-data-dir -c test/config_example.yaml
 
       - name: Specfic module / named output dir / dev template / JSON data / File list
         run: |
@@ -68,7 +113,7 @@ jobs:
       - name: Empty directory (confirm no report)
         run: |
           mkdir empty_dir
-          multiqc -n empty empty_dir
+          multiqc -n empty empty_dir ${{ steps.single_module.outputs.value }}
           [[ ! -f empty.html ]]
 
       - name: Test for missing CSPs