panaroo-qc mash error when multiple threads selected

[alegione]

Hi there,
I think I've resolved this but wanted to flag it in case there was an underlying (but solvable) issue.
When running panaroo-qc with multiple threads I got a mash error (seen in a previous issue #106).
This is being run on a local PC (not server as flagged in that issue) running WSL Ubuntu 20.
I have 16 cores so set threads to 16, not realising it would run 16 instances of mash (instead of say mash screen -p 16), it threw a killed error (see below). This is resolved setting threads to something lower (like 4). Apologies if this has already been documented elsewhere but thought it might be worth specifying in the documentation that threads 4 means 4 instances of mash rather than mash with 4 threads, and identifying memory requirements for using the standard database (no idea what they are myself sorry...)

panaroo-qc --threads 16 --graph_type 'all' --ref_db ../Database/refseq.genomes.k21s1000.msh -i *gff -o output
Max p-value: 0
Loading ../Database/refseq.genomes.k21s1000.msh...
Loading ../Database/refseq.genomes.k21s1000.msh...
Loading ../Database/refseq.genomes.k21s1000.msh...
Loading ../Database/refseq.genomes.k21s1000.msh...
Loading ../Database/refseq.genomes.k21s1000.msh...
Loading ../Database/refseq.genomes.k21s1000.msh...
Loading ../Database/refseq.genomes.k21s1000.msh...
Loading ../Database/refseq.genomes.k21s1000.msh...
Loading ../Database/refseq.genomes.k21s1000.msh...
Loading ../Database/refseq.genomes.k21s1000.msh...
Loading ../Database/refseq.genomes.k21s1000.msh...
Loading ../Database/refseq.genomes.k21s1000.msh...
Loading ../Database/refseq.genomes.k21s1000.msh...
Loading ../Database/refseq.genomes.k21s1000.msh...
Loading ../Database/refseq.genomes.k21s1000.msh...
Loading ../Database/refseq.genomes.k21s1000.msh...
Killed
Killed
Killed
Killed
Killed
Killed
Killed
Killed
Killed
Killed
Killed
joblib.externals.loky.process_executor._RemoteTraceback:
Traceback (most recent call last):
File "/home/alistair/micromamba/envs/genomics/lib/python3.7/site-packages/joblib/externals/loky/process_executor.py", line 428, in _process_worker
r = call_item()
File "/home/alistair/micromamba/envs/genomics/lib/python3.7/site-packages/joblib/externals/loky/process_executor.py", line 275, in call
return self.fn(*self.args, **self.kwargs)
File "/home/alistair/micromamba/envs/genomics/lib/python3.7/site-packages/joblib/_parallel_backends.py", line 620, in call
return self.func(*args, **kwargs)
File "/home/alistair/micromamba/envs/genomics/lib/python3.7/site-packages/joblib/parallel.py", line 289, in call
for func, args, kwargs in self.items]
File "/home/alistair/micromamba/envs/genomics/lib/python3.7/site-packages/joblib/parallel.py", line 289, in
for func, args, kwargs in self.items]
File "/home/alistair/micromamba/envs/genomics/lib/python3.7/site-packages/panaroo/generate_qc_plots.py", line 262, in run_mash_screen
subprocess.run(temp_mash_cmd, shell=True, check=True)
File "/home/alistair/micromamba/envs/genomics/lib/python3.7/subprocess.py", line 512, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'mash screen -i 0.8 -w ../Database/refseq.genomes.k21s1000.msh input.gff > output/tmprifrdup5' returned non-zero exit status 137.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/alistair/micromamba/envs/genomics/bin/panaroo-qc", line 10, in
sys.exit(main())
File "/home/alistair/micromamba/envs/genomics/lib/python3.7/site-packages/panaroo/generate_qc_plots.py", line 456, in main
no_plot=args.no_plot)
File "/home/alistair/micromamba/envs/genomics/lib/python3.7/site-packages/panaroo/generate_qc_plots.py", line 382, in generate_qc_plot
outdir=outdir)
File "/home/alistair/micromamba/envs/genomics/lib/python3.7/site-packages/panaroo/generate_qc_plots.py", line 285, in get_mash_contam
delayed(run_mash_screen)(gff, mash_ref, outdir) for gff in input_gffs)
File "/home/alistair/micromamba/envs/genomics/lib/python3.7/site-packages/joblib/parallel.py", line 1098, in call
self.retrieve()
File "/home/alistair/micromamba/envs/genomics/lib/python3.7/site-packages/joblib/parallel.py", line 975, in retrieve
self._output.extend(job.get(timeout=self.timeout))
File "/home/alistair/micromamba/envs/genomics/lib/python3.7/site-packages/joblib/_parallel_backends.py", line 567, in wrap_future_result
return future.result(timeout=timeout)
File "/home/alistair/micromamba/envs/genomics/lib/python3.7/concurrent/futures/_base.py", line 435, in result
return self.__get_result()
File "/home/alistair/micromamba/envs/genomics/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
subprocess.CalledProcessError: Command 'mash screen -i 0.8 -w ../Database/refseq.genomes.k21s1000.msh input.gff > output/tmprifrdup5' returned non-zero exit status 137.

[gtonkinhill]

Thanks for flagging this, it looks like this is caused by insufficient memory. I agree it is probably better to call Mash with a larger number of threads rather than multiple instances at a time.

I will tag this as an enhancement for now and leave this issue open so that others can find it if they run into the same problem. Your solution of reducing the thread count is probably the best strategy until I implement something more robust.