Draft release notes for v4.1.0

[ctb]

git log --oneline v4.0.0..latest:

4.1.0 release notes

This release provides several convenient features for users, including zipfile collections on input and output and a new prefetch command. sourmash gather has also received a considerable speed/memory upgrade (twice as fast, 80-90% lower memory). You should upgrade! As a reminder, v4.x has several incompatibilities with v3.x, and if you are upgrading from v3.x you should consult our migration guide.

Major new features:

• Support zipped collections of signatures ([MRG] implement a simple ZipFileLinearIndex class #1349)
• Refactor gather functionality for speed & modularity ([MRG] refactor gather functionality for speed & modularity; provide prefetch functionality. #1370, [MRG] Refactor the gather_databases function to use MinHash API over Python set API. #1512, [MRG] Force unload data from SBT searches by default (and fix ZipStorage deallocation along the way) #1513)
• Provide new command, prefetch. ([MRG] refactor gather functionality for speed & modularity; provide prefetch functionality. #1370)
• Add flexible & iterative support for outputting signatures in variety of collection formats - directories, zipfiles, etc. ([MRG] add flexible & iterative support for outputting signatures in variety of formats #1493)
• Add max_containment to API and --max-containment to command line ([MRG] add max_containment to MinHash class. #1346)
• Add --from-file option to sourmash sketch commands ([MRG] add --from-file option to sourmash sketch commands #1362)

Bug fixes that break backwards compatibility:

• Require scaled signatures for containment ([MRG] require scaled signatures for containment #1381)
• Fix CSV output for sourmash lca classify when .name is empty ([MRG] Fix CSV output for sourmash lca classify when .name is empty #1401)
• Really old SBTs (pre-v2.0) no longer load (v1 and v2 SBTs) (changed in [MRG] Rework the find functionality for Index classes #1392)

Other bug fixes:

• Add proper newline output for csv module ([MRG] add proper newline output for csv module #1319) - important for Windows!

Other new features:

• --best-only searches now work for both similarity AND containment (fixed in [MRG] Rework the find functionality for Index classes #1392)
• sourmash categorize now takes all database types
• add --name to sourmash sig merge ([MRG] added --name to merge #1480)
• decline to load really large files for LCA databases if they're not valid JSON ([MRG] check for JSON in first byte of LCA database file #1495)

Major refactoring and new internal functionality:

• Add a MultiIndex class that wraps multiple Index classes ([MRG] Add a MultiIndex class that wraps multiple Index classes. #1374)
• Refactor and dramatically simplify database loading and compatibility checking ([MRG] refactor & clean up database loading around MultiIndex class #1406, [MRG] expand signature selection and compatibility checking in database loading code #1420)
• Rework the find functionality for Index classes ([MRG] Rework the find functionality for Index classes #1392, [MRG] Adjust Index.find search protocol to support selective collection of matches #1477).
• Improved intersection and union calculations ([MRG] Improving intersection and union calculations #1475)

Documentation enhancements:

• Update the sourmash __init__.py docstring, provide __all__ for imports ([MRG] update the sourmash __init__.py docstring #1364)
• Add '-h/--help' usage instructions to 'sourmash sketch' CLI ([MRG] Add '-h/--help' usage instructions to 'sourmash sketch' CLI #1400)
• Add ORCID to contribution checklist (Add ORCID to checklist #1405)
• Add information about updating the developer environment to the developer docs (Adding information about updating developer environment to the developer docs #1432)
• Docs: Partial fix for doc build issues with notebooks ([MRG] Partial fix for doc build issues with notebooks #1516)

Refactoring and cleanup:

• Refactor the database loading code in sourmash_args ([MRG] Refactor the database loading code in sourmash_args #1373, [MRG] fix _location SNAFU in SBT; add tests #1380)
• Pin needletail version to keep MSRV at 1.37 ([MRG] pin needletail version to keep MSRV at 1.37 #1393)
• Rename load_file_list_of_signatures to load_pathlist_from_file (Rename load_file_list_of_signatures to load_pathlist_from_file #1423)
• Update call to notify in src/sourmash/search.py with f-strings (replace notify format usage with f-strings instead (#1409) (#1418)  #1422)
• Bump MSRV to 1.42 (and other dep fixes) ([MRG] Bump MSRV to 1.42 (and other dep fixes) #1461)
• CI/Rust: update and fix cbindgen config ([MRG] Update and fix cbindgen config and CI #1473)
• Refactor MinHash.downsample ([MRG] refactor MinHash.downsample #1458)
• Make MinHash.downsample(...) require keyword arguments & fix newly revealed buggy test. ([MRG] Make MinHash.downsample(...) require keyword arguments & fix newly revealed buggy test. #1448)
• Add a check for LCA database error text intests/test_lca.py (Add a check for LCA database error text intests/test_lca.py #1445)
• pin docutils version to last working ([MRG] pin docutils to v0.16 to address doc building bug #1444)
• add codecov configuration to fix paths (replace notify format usage with f-strings instead (#1409) (#1418)  #1422, [MRG] add codecov configuration to fix paths #1449)
• provide new test fixtures for cleaner testing ([MRG] Add run and runtmp pytest fixtures. #1487)
• Fix small papercuts: SyntaxWarning and coverage reports ([MRG] Fix small papercuts: SyntaxWarning and coverage reports #1488)
• Clean up clippy lints from 1.52 ([MRG] Clean up clippy lints from 1.52 #1505)
• Bump docutils from 0.16 to 0.17.1 (Bump docutils from 0.16 to 0.17.1 #1499)
• Update myst-parser requirement from ~=0.13.7 to >=0.13.7,<0.15.0 (Update myst-parser requirement from ~=0.13.7 to >=0.13.7,<0.15.0 #1520)
• replace utils.TempDirectory with runtmp in some tests ([MRG] replace utils.TempDirectory with run tmp #1502)

[ctb]

released!

[ctb]

blog post http://ivory.idyll.org/blog/2021-sourmash-v4.1.0-released.html