Draft release notes for v4.2.0

[ctb]

git log --oneline v4.1.2..latest

4.2.0 release notes

This release adds several significant features: first, we've added a set of taxonomy command-line functionality for combining sourmash gather output with taxonomy databases, and we've also added a new "picklist" feature that enables flexible selection of subsets of databases. Finally, we've added manifests to databases to support picklists as well as faster database loading and signature selection.

As of this release, we've also formally moved development over to the sourmash-bio organization on GitHub, and we've created a new gitter support channel, sourmash-bio/community. Please join us there if you have any questions, comments, or feature requests!

Major new features:

• add tax/taxonomy submodule ([MRG] add taxonomy subcommand #1543, [MRG] minor updates to tax docs #1628, [MRG] krona forces rank classification #1630, [MRG] fix truncation of tax genome classification #1648)
• add picklists for subsetting databases and results ([MRG] add basic picklist functionality to sourmash sig extract #1587, [MRG] add picklists to selector protocol and provide initial Index support #1588, [MRG] use picklist to exclude signatures #1623, [MRG] add manifests to support fast Index.select(...) and lazy loading #1590, [MRG] Fix picklist-induced error in LCA_Database #1639)
• Add manifests to support fast Index.select(...) and lazy loading ([MRG] add manifests to support fast Index.select(...) and lazy loading #1590)

Documentation updates:

• Add new GTDB databases description to docs and start legacy databases page ([MRG] Add new db description to docs and start legacy databases page #1581)
• Change dib-lab/ URLs to new sourmash-bio/ URLs. ([MRG] update docs with sourmash-bio #1629)
• Add notice for sustainable open source study ([MRG] Add notice for sustainable open source study #1580)

Minor new features:

• alias --nucleotide, --no-nucleotide for moltype args. ([MRG] alias --nucleotide, --no-nucleotide for moltype args. #1632)
• add signature names to known/unknown hash sigs output by sourmash prefetch ([MRG] add signature names to known/unknown hash sigs output by sourmash prefetch #1646)

Bug fixes and performance improvements:

• Speed up sourmash gather with prefetch by ignoring unidentifiable hashes ([MRG] Speed up sourmash gather by ignoring unidentifiable hashes #1613)
• Check for MinHash compatibility in MinHash.intersection_and_union(...) ([MRG] check compatibility in MinHash.intersection_and_union #1627)
• Fix selection w/abund and manifest column type conversions ([MRG] Fix selection w/abund and manifest column type conversions #1645)

Refactoring and cleanup:

• Fix Rust 1.59 lints ([MRG] Fix Rust 1.59 lints #1600)
• Minor cleanup in sourmash_args & sig submodules ([MRG] minor cleanup in sourmash_args & sig submodules #1586)
• Minor cleanup in minhash module ([MRG] minor cleanup in minhash module #1585)
• Fix needless borrows as suggested by clippy ([MRG] Fix needless borrows as suggested by clippy #1636)

[ctb]

This isn't really v4.2 specific, but I wanted to put them somewhere -

database	gather bench
gtdb-rs202.genomic-reps.k31.sbt.zip	0:18.49
gtdb-rs202.genomic-reps.k31.lca.json.gz	0:47.55
gtdb-rs202.genomic.k31.lca.json.gz	1:58.08
gtdb-rs202.genomic-reps.k31.zip	3:20.74
gtdb-rs202.genomic.k31.sbt.zip	6:21.36
gtdb-rs202.genomic.k31.zip	19:51.58

[ctb]

Searching against the database, but with a picklist containing 4 identifiers:

database	search w/4 item picklist
gtdb-rs202.genomic-reps.k31.zip^1^	0:01.56
gtdb-rs202.genomic.k31.zip^1^	0:07.51
gtdb-rs202.genomic-reps.k31.sbt.zip^2^	0:09.88
gtdb-rs202.genomic.k31.sbt.zip^2^	0:35.42
gtdb-rs202.genomic-reps.k31.lca.json.gz^3^	0:46.01
gtdb-rs202.genomic.k31.lca.json.gz^3^	1:51.11

Footnotes:
^1^ Zipfile collections use manifests with picklists, so only the relevant signatures are loaded and searched; hence, there's a big speedup!
^2^ SBTs don't use manifests in search, and the picklists are applied after signatures are found. This increase in speed doesn't make sense to me!?
^3^ LCA database are loaded and parsed before picklists are applied, so there should be little speed improvement from picklists.

So, I still don't understand the SBT speedup, but everything else makes sense.

[ctb]

results with sourmash 3.5.0

database	search
gtdb-rs202.genomic-reps.k31.sbt.zip	0:17.39
gtdb-rs202.genomic.k31.sbt.zip	0:41.82
gtdb-rs202.genomic-reps.k31.lca.json.gz	0:47.90
gtdb-rs202.genomic.k31.lca.json.gz	2:17.28
gtdb-rs202.genomic.k31.zip	(not supported)
gtdb-rs202.genomic-reps.k31.zip	(not supported)

OK so gotta look into the results for gtdb-rs202.genomic.k31.sbt.zip...

[ctb]

sourmash 3.5.0 full results

== This is sourmash version 3.5.0. ==
== Please cite Brown and Irber (2016), doi:10.21105/joss.00027. ==

select query k=31 automatically.
loaded query: d52ec062... (k=31, DNA)
                                                                              
loaded 1 databases.
        Command being timed: "sourmash gather bench/out.sig gtdb-rs202.genomic.k
31.sbt.zip"
        User time (seconds): 11.78
        System time (seconds): 11.31
        Percent of CPU this job got: 55%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:41.82
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 15639052
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 4566312
        Voluntary context switches: 3660
        Involuntary context switches: 301833
        Swaps: 0
        File system inputs: 31119168
        File system outputs: 129056
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

vs

sourmash latest results

== This is sourmash version 4.1.3.dev4+gb787b756. ==
== Please cite Brown and Irber (2016), doi:10.21105/joss.00027. ==

select query k=31 automatically.
loaded query: d52ec062... (k=31, DNA)
                                                                              
loaded 1 databases.
Starting prefetch sweep across databases.
Found 35 signatures via prefetch; now doing gather.
found less than 50.0 kbp in common. => exiting
        Command being timed: "sourmash gather bench/out.sig gtdb-rs202.genomic.k
31.sbt.zip"
        User time (seconds): 52.83
        System time (seconds): 12.60
        Percent of CPU this job got: 17%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 6:21.36
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 1783692
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 1118712
        Voluntary context switches: 45963
        Involuntary context switches: 326881
        Swaps: 0
        File system inputs: 5618696
        File system outputs: 129056
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

User time is much shorter for sourmash 3.5.0...

Hmm, let me try with --no-prefetch...

[ctb]

is released on pypi!