Add distinct minimizer heatmap for Kraken2

[maxibor]

Since version 2.1.0, Kraken2 integrated the distinct minimizer count ability of KrakenUniq
This new function comes with a slightly altered report (see Kraken2 docs) format.

This PR adds a heatmap of the duplication rate of the minimizers (i.e. kmers) which is very useful to detect read stacking, and false positive hits.

• When encountering a "old" report, it behaves as before
• When encountering a "new report" (with distinct minimizer count) it adds the duplication heatmap.

PR checklist:

• This comment contains a description of changes (with reason)
• CHANGELOG.md has been updated

[maxibor]

[maxibor]

I didn't see it before, but this should solve #1379

[ewels]

I could have sworn I added Black CI to MultiQC, but I can't see it now. Could you please run black on your code?

Also, please some test data to https://github.com/ewels/MultiQC_TestData/tree/master/data/modules/kraken for this.

Otherwise, LGTM from quick read of the code 👍🏻 Will review again once we have test data.

[aeu79]

@maxibor Testing with my data, nice plots! (now I need to figure them out)

BTW, the unclassified disappeared from the top 5 taxa plot. Was it intended?


And that is, of course, is changing the percentages.

[aeu79]

Another issue is that it crashes when the reports are mixed (I mean, std and with minimizers in the same folder). Sorry, I was being picky with my tests for #1379 😄

[maxibor]

@aeu79 adc7087 should fix these issues

• The unclassified reads were brought back thanks to a f.seek(0) (reading the file to check the 'log version' was consuming the first line)
• The mixed reports are now nicely handled

@ewels This is now black formatted

[maxibor]

I've added test data with this PR MultiQC/test-data#187

[aeu79]

Guauu, that was fast, congratulations! Looks great:

I reached to the same conclusion by comparing the reads and unique minimizers:

(the one in red from your plot doesn't even show up in the top 50 taxa when I order them by minimizers)

BTW, with many files I'm getting this warning (only with your version):

version:  1.10.dev0 (adc7087)
All std
[INFO   ]         multiqc : This is MultiQC v1.10.dev0 (adc7087)
[INFO   ]         multiqc : Template    : default
[INFO   ]         multiqc : Searching   : /home/agu/Desktop/testMQC
[INFO   ]          kraken : Found 104 reports
/home/agu/GitHub/multiqc/MultiQC/multiqc/plots/bargraph.py:486: UserWarning: FixedFormatter should only be used together with FixedLocator
  axes.set_xticklabels(["{:.0f}%".format(x) for x in vals])
[WARNING]          kraken : Kraken2 reports of different versions were found

[ewels]

Merge conflicts after merging #1347 - will take a look now.

[ewels]

ok, conflicts fixed and code working. I can see that the TestData sample metagenome.kreport now has 100% Unclassified reads in the General Stats table (instead of empty cells) and that it's reads are now Other instead of Unclassified for some of the levels in the Top Taxa plot. But the new heatmap looks kind of uninspiring for me 😓

• Is this just that the test data is not super useful or am I doing something wrong?
• If it's possible to generate an empty plot like this, can we check for that and skip it if not useful?
• If it is the test data, please could we add some more interesting test data into the TestData repo?

Thanks all!

Phil

[ewels]

BTW, with many files I'm getting this warning (only with your version):

version:  1.10.dev0 (adc7087)
All std
[INFO   ]         multiqc : This is MultiQC v1.10.dev0 (adc7087)
[INFO   ]         multiqc : Template    : default
[INFO   ]         multiqc : Searching   : /home/agu/Desktop/testMQC
[INFO   ]          kraken : Found 104 reports
/home/agu/GitHub/multiqc/MultiQC/multiqc/plots/bargraph.py:486: UserWarning: FixedFormatter should only be used together with FixedLocator
  axes.set_xticklabels(["{:.0f}%".format(x) for x in vals])
[WARNING]          kraken : Kraken2 reports of different versions were found

@aeu79 - this warning was unrelated to the changes in this PR. It's a warning from matplotlib due to the core MultiQC code. It's only just started appearing due to an update in their end. I suspect it only showed for you here because you were running with more samples, triggering MultiQC to generate flat image bargraphs instead of interactive.

Anyway, it should now be fixed in master in de0f01b so you should hopefully not see it again. Thanks for reporting 👍🏻

[maxibor]

• Updated test data here: update kraken 2.1.1 test data test-data#189

[ewels]

Thanks @maxibor! I brought back my check to avoid repeated noticed about mixed reports, which had gotten lost.

I'm running tests on the data in MultiQC_TestData > data/modules/kraken and after your update the report now crashes with a JavaScript error (fairly difficult to do usually!).

I'll take a look now, but generally speaking it's good to also try running on just this data as then you'll come across these same issues as me and can probably save some time 😉

[ewels]

Ok added an extra loop in c2faaeb to strip out any heatmap samples with zero values. This fixes the report for me and I now get a heatmap with two cells. I also tweaked the heatmap config as I figure it doesn't need to be square (the x- and y-categories are not the same) and I also specified that the xcats are not sample names, so that they are not affected by hiding / renaming / highlighting in the report.

I'm happy with this so will merge as soon as the tests are passing - thank you! 👍🏻 (especially for speedy response after the long delay).

Please check that you're ok with my changes and let me know ASAP if not as we can fix in a follow-up PR.

Phil

[ewels]

@aeu79 - note that your updated regex from #1383 is now in this PR also, so should hopefully address your comment: #1383 (comment)

Shout if there's anything missing still.

Phil