chore(main): release 0.2.19

[github-actions]

🤖 I have created a release beep boop

0.2.19 (2024-09-12)

Bug Fixes

• validate that either a ndjson or tsv file was given as input (7bf2ef7)

---

This PR was generated with Release Please. See documentation.

[fengelniederhammer]

See #569 (comment), I adapted the changelog manually. I only referenced the issue though, since PR and issue are already linked.

[github-actions]

This is a preview of the changelog of the next release. If this branch is not up-to-date with the current main branch, the changelog may not be accurate. Rebase your branch on the main branch to get the most accurate changelog.

Note that this might contain changes that are on main, but not yet released.

Changelog:

0.3.0 (2024-09-12)

⚠ BREAKING CHANGES

• old database_config files might be invalid if they contained insertion columns. Also, we are more prohibitive for ndjson input files, which now MUST contain nucleotide/amino acid insertions for all respective sequences. The insertions action and filter do no longer require a column field.
• return data as NDJSON instead of JSON

Features

• add more tests, make less flaky and viable with large dataset (7772ae3)
• add number of partitions to /info (1e780b5)
• add serializationVersion to SILO output for smoother transitioning to new formats (d3badb6)
• added amino acid insertion search, added many test cases and fixed various bugs (d1e4b2b)
• allow null for sequenceName in insertion contains queries (6dbe251)
• allow reading fasta files with missing segments and genes #220 (8ea9893)
• allow reading segments and genes that are null from ndjson file #220 (d0a3a7e)
• also get Runtime Config options from environment variables (33bdd65)
• also log to stdout (54b8a47)
• also return mutation destructed that does not need to be reparsed (a93abbf)
• Alternative templating of symbol classes (6b61985)
• Better test coverage for SymbolEquals filter (42c685c)
• boolean columns, resolves Boolean columns #384: const declaration (685db9f)
• boolean columns: actions/tuple: update assignTupleField() (b020109)
• boolean columns: add and use JsonValueType (2ad1268)
• boolean columns: add bool to JsonValueType, update tuple (3c990e2)
• boolean columns: add expression_type "BooleanEquals" (b82ec69)
• boolean columns: add filter_expressions/bool_equals (ff2a138)
• boolean columns: add optional_bool (a2e47f8)
• boolean columns: add storage/column/bool_column (5eb3430)
• boolean columns: column_group: update ColumnPartitionGroup (d7adecf)
• boolean columns: column_group.h: add {ColumnPartitionGroup,ColumnGroup}::bool_columns fields (af44f1a)
• boolean columns: database (c555a68)
• boolean columns: database_config: add "bool" case to DatabaseConfigReader::readConfig() (12fc7b8)
• boolean columns: database_config: add "boolean" case to de/serialisation (e6d5363)
• boolean columns: database_config: add BOOL to ValueType (0b47b82)
• boolean columns: database_config: update DatabaseMetadata::getColumnType() (30febdd)
• boolean columns: database_partition (0aef8f0)
• boolean columns: optional_bool: add == (f0aa3e8)
• boolean columns: selection (e52f2dc)
• build metadata in parallel to sequences. Do not create unaligned sequence tables in preprocessing, rather hive-partition them directly to disk. Better (debug-)logging (c1cdfeb)
• bump serialization version to 2 (e897203)
• bump serialization version to 3 (7584214)
• change base image to ubuntu (64dee0d)
• clearer Operator::negate and Expression::toString, logical Equivalents for debug printing/logging for the Leaf Operators IndexScan and BitmapSelection (026b639)
• consistent behavior of configs when starting SILO with both --preprocessing and --api (847ec7e)
• declutter README.md from linting option, which is now disabled by default and enforced in the CI for the Linter (9220435)
• details no longer shows insertions (#354) (473cd98)
• display database info after loading new database state (0249416)
• display preprocessing duration in logs in human-readable format (not in microseconds) #296 (a2499af)
• do not abort on assertment failures (4e5f598)
• do not enforce building with clang-tidy by default. Linter will still be enforced (7134e45)
• faster builds by copying @corneliusroemer image caching for our dependency images, which rarely change (#374) (7867bc7)
• generalize mutations action to have consistent behavior for different symbols (9834aea)
• generalizing symbol and mutation filters. Clear handling of ambiguous symbols (aa9ad4d)
• have no default sequence by default, implement default amino acid sequence from config (2edf924), closes #454
• have structured and destructured insertion in insertions response (0a7e46a)
• implement basic request id to trace requests #303 (4defb59)
• improve loadDB speeds (2b7cd7d)
• improve validation error message of some actions on orderByFields (a0da5b5)
• insertion action targets all insertion columns by default (6b70241)
• insertions contains action now targets all columns if the column name is missing (32a6951)
• insertions no longer in metadata or databaseConfig, instead expected for all aligned sequences #372 (1f3680c)
• introduce new storage type for Sequence Positions, where the most numerous symbol is deleted (6e15204)
• introduce storage of unaligned sequences from either ndjson file or fasta file and make them queryable via the Fasta action (44df849)
• load table lazily. Unaligned Sequences do not need to load the table (c2a8439)
• logging improvements (4c12a88)
• make threads and max queued http connections available through optional parameter (3ecde68)
• migration to duckdb 0.10.1 (c1426ef)
• new operator for matching strings against regular expressions (ade25e8)
• optimize bitmaps before finishing partition (5b06d58)
• panic: add ASSERT, UNREACHABLE, UNIMPLEMENTED, safer env var (867d08f)
• reintroduce randomize for all query actions (166045c)
• reserve space in columns when bulk inserting rows (e3c9620)
• return data as NDJSON instead of JSON (c236ba4), closes #126
• return only aliased pango lineages (abf0844)
• set the default sequence when there is only a single sequence in the reference genomes (0ae4022), closes #454
• statically disable deleted symbol optimisations because of performance penalty (4e522f4)
• stick to the default of having config value keys in camel case (a1cae40)
• streaming: fasta_aligned: process in sub-partition batches (e87d927), closes #112
• streaming: fasta: process as a batch per partition (66bfdff), closes #112
• streaming: fasta: process in sub-partition batches (12f873f), closes #112
• streaming: fasta: remove limit on total row count (df185cc), closes #112
• streaming: implement materialization cut-off (fd1a77c), closes #112
• support SAM files as sequence input and allow partial sequence input with an offset (5be9a9f)
• template class for sequence store (cef4d48)
• throw an error when there is not initialized database loaded yet #295 (b17f72a)
• throw error in preprocessing when default sequence is not in reference genomes (287bb8b), closes #454
• tidying up CLI and file configuration for runtime config. Added option for specifying the port (c3a88a0)
• update duckdb version (577c1b2)
• use 'pragma once' as include guards instead of 'ifndef...' (bc49aa5)

Bug Fixes

• adapt randomize query results to target architecture. x86 and ARM have possibly different std::hash results (#355) (600000d)
• add bash dependency which is required by conan build of pkgconf and is not installed on alpine by default (1b3f51c)
• add missing file for test (262bedc)
• add missing sequenceName field to mutation action "orderBy" (06c8c86)
• add workaround so insertions are read correctly (8a2bfa8)
• allow all sequence-names by escaping them properly in all SQL statements (901fc7e)
• allow insertions that start at 0 (#449) (f427137)
• allow sql keywords for metadata field names #259 (6fbeee5)
• also consider 'missing' symbols in the mutation action. Bugfix where Position invariant was broken because of 'missing' symbols (fab72a6)
• alternative non-exhaustive three mer index (467086f)
• always build dependency image for amd64 platform (4f73cb0)
• be able to start without genes, without nucleotide sequences or with neither (e878ed5)
• bug when filtering for indexedStringColumns which are not present in some partitions (62d4e08)
• change random ordering for gcc hashing (78055ff)
• change test to reflect new optimisations (cb63010)
• do not order keys and sequences differently, leading to wrong unaligned sequence results when dateToSortBy is set (b0f8352)
• do not return incorrect results when encountering a null value in the unaligned sequence action (a69d117)
• don't abort when reading table in chunks (26b1558)
• don't abort when unaligned sequence is missing - we should return null (#542) (2c649cb), closes #541
• duplicate key in sample.ndjson (c9b5232)
• empty input without partitioning (5fa3c92)
• empty ndjson input files more robust (#473) (9d4232b)
• endToEndResults (c807a33)
• erroneous file created during unit tests should not leak (1b764af)
• error when the Mutations action looked for sequences but the filter was empty (91e5d52)
• fix the RIGHT JOIN in preprocessing of unaligned sequences to contain the correct primary key (a25d5d0)
• floatEquals and floatBetween with null values (47b436e)
• hide nucleotide sequence for default sequence (584715c)
• increase duckdb memory limit to 80GB (20131bb)
• insertion column, remove reference (d940c52)
• insertions being added at wrong index for large files (#472) (e056ed9)
• linking error on linux (ad9076c)
• Make C++ flags in CMake compatible for MacOS (38cae69)
• manually overwrite already existing unaligned sequence files in the temp folder (f37fd88)
• metadata info test accessing getMetadataFields output no longer directly but over the address of a const (0dcdee1)
• missing include (0773999)
• more efficient ndjson emptiness check (#481) (344ec7b)
• new linter errors (ea6934c)
• nodiscard (silence warnings) (69421c1), closes #390
• non default unaligned nucleotide sequence prefix (93b4829)
• nucleotide symbol equals with dot (6ad623e)
• order of randomized tests changed (d29b2f2)
• pango lineage filter with null values (b7238a8)
• put compressors into sql function to avoid static variables (c1a11c8)
• query_handler: check for I/O errors when writing to the HTTP stream (de2b475), closes #112
• quoting {} in "x.{}" SQL struct accesses, as a string starting with a number leads to parser errors (#409) (f3ba6db)
• random (but deterministic for a version) result can depend on internal state, which was changed with duckdb update (9d9351b)
• recursive file reading for nodejs<20 (9ac0ffe)
• reduce function calls and parallelism when inserting rows into storage backend (#509) (93ba858)
• remove caching for linter. Docker image to large on github actions. (8eaeee6)
• remove incorrect compile flags (b92ee4e)
• return instead of continue (03a500a)
• revert duckdb migration due to it being unable to build the new version on the GitHub runner (2a62c54)
• revert test numbers to pre-optimization (99b600a)
• single partition build fix (a8af1c9)
• specify namespace fmt in calls to format_to (#353) (62ffa3b)
• specifying apk versions (2c2354e)
• start with empty files without throwing an error (d407b92)
• streaming: report an error when sorting requested while streaming (313d6bb)
• streaming: report an error when the unimplemented offset or limit matters (3be9c06), closes #511
• test with deterministic results, remove 2 unused variables (96a424b)
• trigger release due to refactor commit-5e3041e5 (#494) (#496) (ecd4ba0)
• typo in the error message if a column is not found (67029a4)
• update cmake version on ubuntu (227a2dd)
• Upper and lower bound should be inclusive in DateBetween filter (53c6c05)
• validate that either a ndjson or tsv file was given as input (7bf2ef7)
• Wrong compare function used in multi-threaded case, which displayed wrong tuples in the details endpoint when a limit was used (650bf36)

[github-actions]

🤖 Created releases:

• v0.2.19
  🌻