smudgeplot.py hetmers not producing kmerpairs_text.smu (.trim.ktab only)

[gemmacol]

hetmers step has completed, but not resulting in .smu file?

smudgeplot.py hetmers -L 12 -t 4 -o kmerpairs -tmp $TMP --verbose Fastk_Table

Running smudgeplot v0.4.0dev
Task: hetmers
Calling: hetmers (PloidyPlot kmer pair search) -okmerpairs -e12 -T4 -v -P/nesi/nobackup/ga02470/acanthoxyla/github/new-illumina-smudgeplots/AXG/fastk2/tmp Fastk_Table

The input table is untrimmed and not symmetric

Trimming k-mers in table with count < 12
Output table ./.trim.ktab already exists, continue? yes

Making trimmed table symmetric

Starting to count covariant pairs

Done!

This outputs a file called: .symx.ktab
But not the .smu file that I was expecting.

I thought maybe we don't have the coverage for the -L 12 option so tried also -L 6 and same result.

Any suggestions why I am not generating the .smu file?

Options used for prior step:
FastK -v -t4 -k21 -M16 -T4 -P$TMP -c AXG_novaseq_R*.fq -NFastk_Table

[KamilSJaron]

Hi Gemma, the expected log looks like this:

Running smudgeplot v0.4.0dev
Task: hetmers
Calling: hetmers (PloidyPlot kmer pair search)  -otest -e10 -T4 -v Fastk_Table

  The input table is untrimmed and not symmetric

  Trimming k-mers in table with count < 10

  Making trimmed table symmetric

  Starting to count covariant pairs

  Count complete, plotting

  About to save stuff

  Saving stuff

Done!

Yours is missing

  Count complete, plotting

  About to save stuff

  Saving stuff

Done!

I am not sure why, but I suspect it's something wrong with the k-mer database. If you make a k-mer histogram (that should take just seconds), does it look sane? Something like this should do the job...

Histex -G Fastk_Table > kmer.hist

[gemmacol]

Hi Kamil,

I tried your suggestion: Histex -G Fastk_Table > kmer.hist
Then plotted kmer.hist with genomescope.
The model didn't fit well, but as far as I can tell it looks "normal"

Do you have any further suggestions what I could try, to generate the .smu file?

So far I have these intermediate files in the dir:

Fastk_Table.hist
Fastk_Table.ktab

.Fastk_Table.ktab.2
.Fastk_Table.ktab.4
.Fastk_Table.ktab.1
.Fastk_Table.ktab.3

..symx.ktab.1
.symx.ktab

[KamilSJaron]

@gemmacol That's puzzing. With Gene we can't think of a good reason this could/should happen. Did you by any chance have anything streamed to the stderr? @thegenemyers thinks there should be something, it should not crush silently.

Would you be able to upload for us the .ktab as well as all the .Fastk_Table.ktab.? We are unable to figure out what exactly is wrong with this with the information we have...

[gemmacol]

I have re-created the issue this time with a single slurm script and here provide the two files you asked for as well as the script and the log files gathered after re-running it:

1. This is the .symx.ktab : symx.zip

2. This is FastK_Table.ktab (the hidden files were too large to upload, this is what you were asking for?) : FastK_Table.zip

3. here is the exact script that was used:
   AXG.52229199.zip

And here a screenshot of the resulting files in time-reversed order (.smu still missing):

Many thanks,
Gemma

[KamilSJaron]

I am afraid the hidden files are needed for the full k-mer table. @thegenemyers?

[gemmacol]

Here are the hidden files as a google drive link: https://drive.google.com/drive/folders/12keGxYN5jkY8t--Rksq_sZ3FrcXXnDuj?usp=sharing