Progress tracker

[johnkerl]

Context

End-to-end functioning of the API for the May-June timeframe.

Higher-level goals

• SOMA-read and schema-introspection APIs
  • Need to be able to introspect schema on obs, var, X, uns, and obsm:
    • Add a few more SOMA accessors #94
    • 'for matrix in soma.obsm: ...' et al #97
    • SOMA-component dataframe/schema accessors #104
    • Allow shape accessors on X et al. #125
  • Need to be able to slice/read the same
    • Implement SOMA-level dimension-slicing #102
    • Attribute-filter logic for obs and var #103
    • see also issue Collection-slice prototype #95
  • apis/python/examples/ directory, moving toward vignette-style deliverable
    • Webdocs experiment #136
  • Refactoring of README content:
    • Split ingest material to README-ingestion.mdKerl/readme divide #122
• More on SOMA-read and schema-introspection APIs
  • Possible work integrating metadata hints/labels from the I/O routines: e.g., embedding discovery, X scale, etc.
  • Provenance hints on various SOMA slots ("this came from a numpy.ndarray" etc.) and readback-as-such support
• Resolve AnnData ingestion issues:
  • uns value with zero length value raises exception in from_anndata #71
    • Store uns scalars as group metadata, not 1D arrays #115
  • Chunked reads for CSC and dense (in addition to CSR)
    • Chunked writes #37
    • Implement chunked-X writes for CSC and dense #120
    • Note, global-order-write feature to avoid need for consolidation is still open / pending TileDB-Py
• SOMA API: support for incremental queries when used to map/reduce X/raw.X (à la TileDB.query(return_incomplete=True)), with performance essentially the same as from native TileDB
• I/O namespace: move IO API into the tiledbsc.io namespace
  • API: move to/from_ functions into "io" namespace #83
• I/O: resolve any logical data organizational changes from spec edits
  • Python/R raw status
  • Impl: programmable X layer: X["data"] not X.data, and ability to add others
    • soma.X.data -> soma.X["data"] #142
• Multimodal data: gap analysis with respect to current SOMA/SOCO API
  • This is addressed in the next-generation data model https://github.com/single-cell-data/SOMA/blob/spec-revision/brainstorming.md
• SOMA-collection steps:
  • Populate a SOMA collection:
    • Initial SOMACollection class #118
    • Webdocs experiment #136 (examples)
    • Preliminary SOMA-collection slice/batch queries #173
    • Uniformization examples #161
  • Batch-process, e.g. mean of raw/X/data grouping by obs['cell_type_ontology_term_id']
    • Preliminary SOMA-collection slice/batch queries #173
  • Random sampling
  • Slice-creation: filter on values in specified obs/var, create AnnData object, store at .h5ad
    • Preliminary SOMA-collection slice/batch queries #173
  • Add/delete SOMAs to/from the collection
  • Synchronize API-spec edits and make sure demo material uses only the documented SOMA API
• Documentation
  • Have Python-API online help clearly navigable: what methods are available for which SOMA slots
  • Have Python-API docs accessible on the web
    • https://tiledb-singlecell-docs.s3.amazonaws.com/docs/overview.html
      [X] Storage support:
  • Local disk
  • s3://... supported by tiledbsc-r and tiledbsc-py
    • Examples on Allow S3 ingest #148
    • Timings look good when queried from m6i.4xlarge in the same S3 region as the storage, as well as from TileDB Cloud notebooks
  • tiledb://... supported by tiledbsc-r and tiledbsc-py
    • tiledb-cloud testing results #189
• Proposal/demo of SOCO-level query acceleration on large-scale collections
  • Loop over collections: Initial SOMACollection class #118
  • Possible addition of a sparse-array index layer
  • Possible consolidating-write approach
• More SOMA-collection steps:
  • WIP: Larger datasets and query timings
  • Collection-rebuild timings
  • Larger-scale map-reduce timings
  • Larger-scale random sampling

Performance/scale:

• X obs_id filter RLE vs zstd
• Global-order writes in TileDB-Py Chunked writes #37
  • Needs exposure of global-order-write flag to Python API -- TileDB Shortcut task 17495
  • Prep-sort work within tiledbsc-py -- be sure to column-sort data going in to the chunk writes
  • Re-ingest datasets
• Column-major storage: TileDB-Py #3143 -- performance-tuning level
• Apply dictionary encoding Dictionary encoding for second matrix dimensions #36

TileDB-API support

• TileDB attributes can be UTF-8, but, are not queryable if they are -- so we must store them as ASCII. In this package, make sure queries read back with decode ("MT-CO3" not b"MT-CO3"):
  • Store to-be-queried obs/var columns as ASCII (workaround) #101
  • Read back obs/var IDs as suitably decoded strings #117
  • Extend ASCII-queryability workaround #138
  • Correct ASCII-to-Unicode readback for attribute_filter #141
• Exposure of global-order-write flag to Python API -- TileDB Shortcut task 17495 -- see also "Performance/scale" above -- without this, column-level queries maybe be slow which we need to solve for GA -- update 2022-06-06: query performance is not negatively impacted on local disk or S3 by the existence of unconsolidated chunks
• Query syntax:
  • and -- in place
  • or -- like obs.cell_type == "blood" or obs.cell_type == "lung" -- as of 0.0.5
  • in -- like obs.cell_type in ["blood", "lung"] -- WIP -- TileDB Shortcut story 19015
• Empty-string writes
  • Pending TileDB-Py #1089
  • But, that can be avoided in this package by writing scalar uns values as group metadata rather than 1D arrays -- which is in turn blocked by:
    • Group-metadata discovery
      • TileDB Shortcut story 17879 and TileDB-Py PR 1123
      • Update to TileDB-Py 0.15.2 #144
• "name" in G: TileDB Shortcut story 18057 -- non-blocking; will simplify some code in this package
  • TileDB-Py PR 1125
• Reversible typecasts Reversible typecasts (parent task) #106
  • TILEDB_BOOL TileDB Shortcut story 17783 -- released in tiledbsc-py 0.0.6
  • In this package, save array metadata like "was CSR", etc., along with language/version of the code that wrote the data (tiledbsc-r / tiledbsc-py)
• tiledb.from_numpy (for uns) overwrite/append flag: TileDB Shortcut story 17765 -- important for S3/TileDB-Cloud ops
• tiledb.from_numpy (for uns): 1D arrays are not uploadable.
  • Work around issue uploading 1D uns to tiledb URIs #201 workaround
  • TileDB Shortcut story 19039 filed

[johnkerl]

This tracked issues for the May/June timeframe. I'll open a new one for current cross-company issue-tracking.

Update: #302