Tidying: Add missing function calls to modules

[ewels]

Two core MultiQC function calls that should be used in basically every module are self.add_data_source() (docs) and self.write_data_file() (docs).

Both of these give additional functionality and are very easy to miss (I often miss them when writing and reviewing).

I just put together a super crude little script that finds files that contain the string self.find_log_files (eg. python scripts that look like they are modules) and then looks to see if these two functions can be found within the file. As expected, it returns a bunch of filenames:

  Checking for function call 'self.add_data_source'  
Can't find 'self.add_data_source' in /picard/CollectIlluminaLaneMetrics.py
Can't find 'self.add_data_source' in /picard/ExtractIlluminaBarcodes.py
Can't find 'self.add_data_source' in /picard/CollectIlluminaBasecallingMetrics.py
Can't find 'self.add_data_source' in /picard/util.py
Can't find 'self.add_data_source' in /picard/ValidateSamFile.py
Can't find 'self.add_data_source' in /picard/CrosscheckFingerprints.py
Can't find 'self.add_data_source' in /picard/QualityYieldMetrics.py
Can't find 'self.add_data_source' in /stacks/stacks.py
Can't find 'self.add_data_source' in /pycoqc/pycoqc.py
Can't find 'self.add_data_source' in /goleft_indexcov/goleft_indexcov.py
Can't find 'self.add_data_source' in /vcftools/relatedness2.py
Can't find 'self.add_data_source' in /vcftools/tstv_by_qual.py
Can't find 'self.add_data_source' in /vcftools/tstv_summary.py
Can't find 'self.add_data_source' in /vcftools/tstv_by_count.py
Can't find 'self.add_data_source' in /interop/interop.py
Can't find 'self.add_data_source' in /rna_seqc/rna_seqc.py
Can't find 'self.add_data_source' in /pangolin/pangolin.py
Can't find 'self.add_data_source' in /flexbar/flexbar.py
Can't find 'self.add_data_source' in /fgbio/ErrorRateByReadPosition.py
Can't find 'self.add_data_source' in /pychopper/pychopper.py
Can't find 'self.add_data_source' in /clipandmerge/clipandmerge.py
Can't find 'self.add_data_source' in /longranger/longranger.py
Can't find 'self.add_data_source' in /busco/busco.py
Can't find 'self.add_data_source' in /vep/vep.py
Can't find 'self.add_data_source' in /kat/kat.py
Can't find 'self.add_data_source' in /trimmomatic/trimmomatic.py
Can't find 'self.add_data_source' in /flash/flash.py
Can't find 'self.add_data_source' in /htseq/htseq.py
Can't find 'self.add_data_source' in /sargasso/sargasso.py
Can't find 'self.add_data_source' in /sortmerna/sortmerna.py
Can't find 'self.add_data_source' in /seqyclean/seqyclean.py
Can't find 'self.add_data_source' in /peddy/peddy.py
Can't find 'self.add_data_source' in /kraken/kraken.py
Can't find 'self.add_data_source' in /adapterRemoval/adapterRemoval.py
Can't find 'self.add_data_source' in /bustools/bustools.py
Can't find 'self.add_data_source' in /qorts/qorts.py
Can't find 'self.add_data_source' in /macs2/macs2.py
Can't find 'self.add_data_source' in /mirtop/mirtop.py



  Checking for function call 'self.write_data_file'  
Can't find 'self.write_data_file' in /rseqc/read_gc.py
Can't find 'self.write_data_file' in /rseqc/junction_saturation.py
Can't find 'self.write_data_file' in /rseqc/inner_distance.py
Can't find 'self.write_data_file' in /rseqc/read_duplication.py
Can't find 'self.write_data_file' in /rseqc/gene_body_coverage.py
Can't find 'self.write_data_file' in /picard/util.py
Can't find 'self.write_data_file' in /picard/CrosscheckFingerprints.py
Can't find 'self.write_data_file' in /qualimap/QM_RNASeq.py
Can't find 'self.write_data_file' in /biscuit/biscuit.py
Can't find 'self.write_data_file' in /pycoqc/pycoqc.py
Can't find 'self.write_data_file' in /hops/hops.py
Can't find 'self.write_data_file' in /mosdepth/mosdepth.py
Can't find 'self.write_data_file' in /goleft_indexcov/goleft_indexcov.py
Can't find 'self.write_data_file' in /malt/malt.py
Can't find 'self.write_data_file' in /vcftools/relatedness2.py
Can't find 'self.write_data_file' in /vcftools/tstv_by_qual.py
Can't find 'self.write_data_file' in /vcftools/tstv_summary.py
Can't find 'self.write_data_file' in /vcftools/tstv_by_count.py
Can't find 'self.write_data_file' in /homer/tagdirectory.py
Can't find 'self.write_data_file' in /deeptools/plotEnrichment.py
Can't find 'self.write_data_file' in /deeptools/bamPEFragmentSizeDistribution.py
Can't find 'self.write_data_file' in /deeptools/plotPCA.py
Can't find 'self.write_data_file' in /deeptools/estimateReadFiltering.py
Can't find 'self.write_data_file' in /deeptools/plotFingerprint.py
Can't find 'self.write_data_file' in /deeptools/plotCoverage.py
Can't find 'self.write_data_file' in /deeptools/plotProfile.py
Can't find 'self.write_data_file' in /deeptools/bamPEFragmentSizeTable.py
Can't find 'self.write_data_file' in /deeptools/plotCorrelation.py
Can't find 'self.write_data_file' in /interop/interop.py
Can't find 'self.write_data_file' in /fgbio/groupreadsbyumi.py
Can't find 'self.write_data_file' in /ngsderive/ngsderive.py
Can't find 'self.write_data_file' in /jellyfish/jellyfish.py
Can't find 'self.write_data_file' in /vep/vep.py
Can't find 'self.write_data_file' in /preseq/preseq.py
Can't find 'self.write_data_file' in /purple/purple.py
Can't find 'self.write_data_file' in /gatk/base_recalibrator.py
Can't find 'self.write_data_file' in /clusterflow/clusterflow.py
Can't find 'self.write_data_file' in /dragen/coverage_hist.py
Can't find 'self.write_data_file' in /dragen/mapping_metrics.py
Can't find 'self.write_data_file' in /dragen/coverage_metrics.py
Can't find 'self.write_data_file' in /dragen/ploidy_estimation_metrics.py
Can't find 'self.write_data_file' in /dragen/coverage_per_contig.py
Can't find 'self.write_data_file' in /dragen/vc_metrics.py
Can't find 'self.write_data_file' in /dragen/fragment_length.py
Can't find 'self.write_data_file' in /bbmap/bbmap.py
Can't find 'self.write_data_file' in /kraken/kraken.py

It would be great to go through all of these and add in the missing functions. Most of the time it should be simple, copying and pasting with the default arguments. But some may need a little customisation, for example if a non-standard sample name is used instead of f['s_name'], or if multiple input files are used for the same module but different report sections.

Script is here: https://github.com/ewels/MultiQC/blob/master/.github/workflows/code_checks.py

Once we have tidied all of these up, we can add this script to the CI to catch this before PRs are merged.

A simple workaround for files that shouldn't use this function for a legitimate reason is to add a comment explaining why that is the case with the function name. Then the function name string will be found.