The 'picard' MultiQC module broke

[AndriesDeKoker]

Description of bug:

No MultiQC report. When rerunning with multiqc . same error and an incomplete report is generated.

MultiQC Error log:

  1) cluster/victini
[WARNING]         multiqc : MultiQC Version v1.10 now available!
[INFO   ]         multiqc : This is MultiQC v1.9
[INFO   ]         multiqc : Template    : default
[INFO   ]         multiqc : Searching   : /vscmnt/gent_kyukon_scratch/_kyukon_scratch_gent/vo/001/gvo00133/vsc40742/cfRRBS_Ruben
[ERROR  ]         multiqc : Oops! The 'picard' MultiQC module broke...
  Please copy the following traceback and report it at https://github.com/ewels/MultiQC/issues
  If possible, please include a log file that triggers the error - the last file found was:
    ./.snakemake/metadata/bG9ncy8zNjc2XzA1OF9wb29sXzFfUzU4X0wwMDFfaHNfbWV0cmljcy50eHQ=
============================================================
Module picard raised an exception: Traceback (most recent call last):
  File "/apps/gent/CO7/skylake-ib/software/MultiQC/1.9-intel-2020a-Python-3.8.2/lib/python3.8/site-packages/multiqc/multiqc.py", line 569, in run
    output = mod()
  File "/apps/gent/CO7/skylake-ib/software/MultiQC/1.9-intel-2020a-Python-3.8.2/lib/python3.8/site-packages/multiqc/modules/picard/picard.py", line 65, in __init__
    n['HsMetrics'] = HsMetrics.parse_reports(self)
  File "/apps/gent/CO7/skylake-ib/software/MultiQC/1.9-intel-2020a-Python-3.8.2/lib/python3.8/site-packages/multiqc/modules/picard/HsMetrics.py", line 182, in parse_reports
    plot = table.plot(data, _get_table_headers(data))
  File "/apps/gent/CO7/skylake-ib/software/MultiQC/1.9-intel-2020a-Python-3.8.2/lib/python3.8/site-packages/multiqc/plots/table.py", line 49, in plot
    return make_table ( dt )
  File "/apps/gent/CO7/skylake-ib/software/MultiQC/1.9-intel-2020a-Python-3.8.2/lib/python3.8/site-packages/multiqc/plots/table.py", line 139, in make_table
    val = header['modify'](val)
  File "/apps/gent/CO7/skylake-ib/software/MultiQC/1.9-intel-2020a-Python-3.8.2/lib/python3.8/site-packages/multiqc/modules/picard/HsMetrics.py", line 279, in <lambda>
    headers[h]['modify'] = lambda x: x * config.base_count_multiplier
TypeError: can't multiply sequence by non-int of type 'float'
============================================================
[INFO   ]         bismark : Found 20 alignment reports
[INFO   ]         bismark : Found 10 dedup reports
[INFO   ]         bismark : Found 20 methextract reports
[INFO   ]         bismark : Found 20 bismark bam2nuc reports
[INFO   ]        cutadapt : Found 20 reports
[INFO   ]          fastqc : Found 20 reports
[INFO   ]         multiqc : Compressing plot data
[INFO   ]         multiqc : Report      : multiqc_report.html
[INFO   ]         multiqc : Data        : multiqc_data
[INFO   ]         multiqc : MultiQC complete
[Sat Mar 13 00:00:31 2021]
Error in rule multiqc:
    jobid: 0
    output: multiqc_report.html

RuleException:
CalledProcessError in line 328 of /scratch/gent/vo/001/gvo00133/vsc40742/BismarkPipeline/Bismark_pipeline_PE.snakefile:
Command ' set -euo pipefail;  ml purge && ml MultiQC/1.9-intel-2020a-Python-3.8.2; multiqc -f . ' returned non-zero exit status 1.
  File "/scratch/gent/vo/001/gvo00133/vsc40742/BismarkPipeline/Bismark_pipeline_PE.snakefile", line 328, in __rule_multiqc
  File "/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/concurrent/futures/thread.py", line 56, in run
Removing output files of failed job multiqc since they might be corrupted:
multiqc_report.html
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message

File that triggers the error:

This file: ./.snakemake/metadata/bG9ncy8zNjc2XzA1OF9wb29sXzFfUzU4X0wwMDFfaHNfbWV0cmljcy50eHQ= :

{"version": null, "code": "gAMoQxB0AGQBfA1kAo0CAQBkAFMAcQAoWAUAAABpbnB1dHEBWAYAAABvdXRwdXRxAlgGAAAAcGFyYW1zcQNYCQAAAHdpbGRjYXJkc3EEWAcAAAB0aHJlYWRzcQVYCQAAAHJlc291cmNlc3EGWAMAAABsb2dxB1gHAAAAdmVyc2lvbnEIWAQAAABydWxlcQlYCQAAAGNvbmRhX2VudnEKWA8AAABzaW5ndWxhcml0eV9pbWdxC1gQAAAAc2luZ3VsYXJpdHlfYXJnc3EMWA8AAAB1c2Vfc2luZ3VsYXJpdHlxDVgMAAAAYmVuY2hfcmVjb3JkcQ5YBQAAAGpvYmlkcQ9YCAAAAGlzX3NoZWxscRB0cRFdcRIoTljVAAAAbWwgcHVyZ2UgJiYgbWwgcGljYXJkLzIuMjEuNi1KYXZhLTExOyBqYXZhIC1qYXIgJEVCUk9PVFBJQ0FSRC9waWNhcmQuamFyIENvbGxlY3RIc01ldHJpY3MgST17aW5wdXQuYmFtfSBPPXtvdXRwdXQudHh0fSBSPXtwYXJhbXMuZ2Vub21lcGF0aH0gVEFSR0VUX0lOVEVSVkFMUz17cGFyYW1zLnRhcmdldHBhdGh9IEJBSVRfSU5URVJWQUxTPXtwYXJhbXMudGFyZ2V0cGF0aH0gcRNoDoVxFGVYBQAAAHNoZWxscRWFcRZ0cRcu", "rule": "hsmetrics", "input": ["sorted_reads/3676_058_pool_1_S58_L001_R1_001_val_1_bismark_bt2_pe.bam", "sorted_reads/3676_058_pool_1_S58_L001_R1_001_val_1_bismark_bt2_pe.bam.bai"], "log": [], "params": ["'/user/data/gent/vo/001/gvo00133/BismarkPipeline/RRBS_regions_GRCh38.interval_list'", "'/user/data/gent/vo/001/gvo00133/resources/GRCh38-noalt/GRCh38.fa'"], "shellcmd": "ml purge && ml picard/2.21.6-Java-11; java -jar $EBROOTPICARD/picard.jar CollectHsMetrics I=sorted_reads/3676_058_pool_1_S58_L001_R1_001_val_1_bismark_bt2_pe.bam O=logs/3676_058_pool_1_S58_L001_hs_metrics.txt R=/user/data/gent/vo/001/gvo00133/resources/GRCh38-noalt/GRCh38.fa TARGET_INTERVALS=/user/data/gent/vo/001/gvo00133/BismarkPipeline/RRBS_regions_GRCh38.interval_list BAIT_INTERVALS=/user/data/gent/vo/001/gvo00133/BismarkPipeline/RRBS_regions_GRCh38.interval_list ", "incomplete": false, "starttime": 1615554838.363824, "endtime": 1615554847.1671035, "job_hash": 7284811398034519501, "conda_env": null, "singularity_img_url": null}

MultiQC run details (please complete the following):

• Command used to run MultiQC:

    shell:
          "ml purge && ml MultiQC/1.9-intel-2020a-Python-3.8.2; "
          "multiqc -f ."

• MultiQC Version: MultiQC v1.9
• Operating System: CentOS 7.7
• Python Version: Python-3.8.2
• Method of MultiQC installation: done by admin of HPC

Additional context

Run as part of a snakemake pipeline.
also rerunning multiqc . -x .snakemake gives the same error and an incomplete report.

[AndriesDeKoker]

Most probably this error was caused by passing through a ',' (comma) instead of a '.' (point) as decimal_point in a remote ssh session (I am from Belgium). I am running a test at the moment.

[ewels]

I think that the error is because MultiQC is picking up the snakemake cache files and treating them as if they were from Picard. I don't really understand why it's picking the file up though, it should be ignored due to the default MultiQC config:

https://github.com/ewels/MultiQC/blob/d840affd475454a212cbdea3df7951e961733a1f/multiqc/utils/config_defaults.yaml#L101-L103

Are you using a custom MultiQC config anywhere? eg. a multiqc_config.yml file?

Phil

[AndriesDeKoker]

I am not using a custom multiqc_config.yml file.

Moving the .snakemake-folder to a location multiQC cannot find them anymore, raises the same error on snakejob log-files. Again moving these, raises the same error on hsmetrics-files. Again moving these, no error is raised. So taking a look in the hsmetrics-files and comparing these with a collaborator who is running MultiQC without error, we noticed the ',' instead of '.' . Still testing.

I am following https://stackoverflow.com/questions/29609371/how-do-not-pass-locale-through-ssh for setting up my ssh connection not passing locale config.

[AndriesDeKoker]

The 'picard' module did not break anymore, tested 2 times.
I did:
$ touch ~/.ssh/config
$ ssh -F ~/.ssh/config your_user@your_host

[ewels]

ok, interesting! Nice that you were able to get around the locale issue. I don't understand why that should have any effect though to be honest - the snippet I posted above should ignore any path that includes .snakemake, which the file that triggered the error in the log did. I'm going to take a look into this now.

[ewels]

Ok after a little testing, I realised that it should have been */.snakemake instead of .snakemake to work. Updated, so the development build from GitHub should now ignore all hidden .snakemake folders as intended.

Thanks for reporting!

Phil