BCL Convert moduleKeyError: # of >= Q30 Bases (PF)

[ElisabettaNale]

Description of bug

I am trying to use BCL convert output file (v.3.9.3) to generate the corresponding MultiQC report. The MultiQC software expects to find the following key # of >= Q30 Bases (PF) presumably in the Demultiplex_Stats.csv. file, but it can't find it.

File that triggers the error

bcl_output.zip

MultiQC Error log

multiqc . --config /hpcnfs/techunits/genomics/work/nf_elisabetta_chromium/tumgt-genomics_bclconvert/nf/multiqc_config.yaml -o .

/// MultiQC 🔍 | v1.11

|           multiqc | Search path : /hpcnfs/techunits/genomics/work/nf_elisabetta_chromium/tumgt-genomics_bclconvert/nf/work/b2/2043677ab0d4ec1b884d20da487d5a/DUAL-TEN/Reports
|         searching | ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 9/9  
╭──────────────────────────────────── Oops! The 'bclconvert' MultiQC module broke... ─────────────────────────────────────╮
│ Please copy this log and report it at https://github.com/ewels/MultiQC/issues                                           │
│ Please attach a file that triggers the error. The last file found was: ./RunInfo.xml                                    │
│                                                                                                                         │
│ Traceback (most recent call last):                                                                                      │
│   File "/usr/local/lib/python3.8/dist-packages/multiqc/multiqc.py", line 624, in run                                    │
│     output = mod()                                                                                                      │
│   File "/usr/local/lib/python3.8/dist-packages/multiqc/modules/bclconvert/bclconvert.py", line 44, in __init__          │
│     self.parse_demux_data(demux)                                                                                        │
│   File "/usr/local/lib/python3.8/dist-packages/multiqc/modules/bclconvert/bclconvert.py", line 347, in parse_demux_data │
│     lane["basesQ30"] += int(row["# of >= Q30 Bases (PF)"])                                                              │
│ KeyError: '# of >= Q30 Bases (PF)'                                                                                      │
│                                                                                                                         │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
|           multiqc | No analysis results found. Cleaning up..
|           multiqc | MultiQC complete

[Garrett-Ng]

I also came across this exact issue. The release notes for bcl-convert v3.9.3(pdf) mention that the # of >= Q30 Bases (PF) and Mean Quality Score columns were moved from Demultiplex_Stats.csv to Quality_Stats.csv as YieldQ30 and Mean Quality Score. The exact names appear a little differently in the csv (e.g. Mean Quality Score (PF)).

[ewels]

Fixed in #1595 by @yanick and me. Please do check that the new code on master works for your samples as expected 😊

Thanks for reporting!

Phil

[Paul-rk-cruz]

|            fastqc | Found 20 reports
╭─────────────── Oops! The 'bclconvert' MultiQC module broke... ───────────────╮
│ Please copy this log and report it at                                        │
│ https://github.com/ewels/MultiQC/issues                                      │
│ Please attach a file that triggers the error. The last file found was:       │
│ ./RunInfo.xml                                                                │
│                                                                              │
│ Traceback (most recent call last):                                           │
│   File "/opt/conda/lib/python3.7/site-packages/multiqc/multiqc.py", line 624 │
│     output = mod()                                                           │
│   File "/opt/conda/lib/python3.7/site-packages/multiqc/modules/bclconvert/bc │
│     self.parse_demux_data(demux)                                             │
│   File "/opt/conda/lib/python3.7/site-packages/multiqc/modules/bclconvert/bc │
│     lane["basesQ30"] += int(row["# of >= Q30 Bases (PF)"])                   │
│ KeyError: '# of >= Q30 Bases (PF)'                                           │
│                                                                              │

[Paul-rk-cruz]

Having this error trying to run bclconvert module

Command:

/opt/conda/bin/multiqc -f $rtitle $rfilename --config $multiqc_config -m fastqc -m bclconvert .

(From Nextflow Docker)

Input:

file 'RunInfo.xml' from bclconvert_runinfo.collect()
file 'Demultiplex_Stats.csv' from bclconvert_stats.collect()
file 'Quality_Metrics.csv' from bclconvert_quality_metrics.collect()
file 'Adapter_Metrics.csv' from bclconvert_adapter_metrics.collect()
file 'Top_Unknown_Barcodes.csv' from bclconvert_top_unknown_barcodes.collect()
file 'Logs/errors.log' from bclconvert_errors_log.collect()
file 'Logs/FastqComplete' from bclconvert_fastq_complete_log.collect()
file 'Logs/Info.log' from bclconvert_info_log.collect()
file 'Logs/Warnings.log' from bclconvert_warnings_log.collect()

[Paul-rk-cruz]

Any ideas to fix?

[ewels]

Hi @Paul-rk-cruz,

Are you using the development version of MultiQC? This fix has not yet been released yet.

For future reference, please open new issues rather than commenting on closed ones. It's very easy to not see these (you can always reference closed issues). It also means that you use the bug template and provide all of the information that I need to help.

Phil

[Paul-rk-cruz]

Cool thanks for your reply. I used the most recent release and built your docker and using that

[ewels]

Right, if you use the development version (ewels/multiqc:dev) then it should work. I haven't checked in a while but I think that should be building off master every time there's a new commit.

[Paul-rk-cruz]

That would be great. Thank you. How do I download the dev version?

[Paul-rk-cruz]

Nevermind, I used the dev tag for 'docker pull ewels/multiqc:dev' and got it. Will test it out. Thanks again!

[Paul-rk-cruz]

Okay this will be difficult, as I am running multiQC from within Nextflow. Is the dev version available in conda?

[ewels]

You can install any GitHub repo into a conda environment by defining a pip install section in your environment.yml file and then getting pip to install the git repo.

But I'm planning to do a release soon, probably tomorrow, so it might be easier to just wait for that 🙂

[snashraf]

Hi Ewels,

Thanks for the tool again.

I am not getting any error while running now but PF Q30 columns are showing as 0 but I can see that we have correct values in the Quality_Stats.csv file. Please check the attachment and screenshot.
Quality_Metrics.csv

[ewels]

Hi @snashraf - please add a new issue for this, ideally filling in all fields (such as MultiQC log, that tells me the version that you're running amongst other things).