SOMA-collection-level slice/batch queries [needs revision] #157

johnkerl · 2022-06-09T19:40:13Z

Overview

[cartographer](https://github.com/single-cell-data/TileDB-SingleCell/blob/kerl/soma- slice/apis/python/examples/cartographer.py), due originally to @bkmartinjr , is a portable/reusable example of how to take a group of .h5ad files and create a SOMACollection having uniform obs schema split out to Uniformization examples #161
~~splitter is a little tool that was used to create the four small .h5ad files for unit-test/example-documentation purposes~~ split out to Doc bits for splitting out little test files #167
examples/soco-slice-query.md is a walk-through example of using this code.
test_soco_slice_query.py similar, as a unit-test case.
examples/soco-batch-query.md is another walk-through example of using this code.
The "meat" of the PR is in:
- Mods to soma.py
- New soma_slice.py

Walkthrough

soco-slice-query.md shows you how to actually execute a slice query. This demonstrates the power of TileDB SOMAs to deliver targeted data for analysis at low latency. Notice that this examples writes the sliced output in both SOMA and .h5ad format, so you can go on to do all manner of analysis using for example Scanpy.
soco-batch-query.md shows you how to do a longer-running, cross-cutting batch query -- in this example, computing the mean of X/data, grouping by obs['cell_type_ontology_term_id'].

Notes

This replaces SOMA-collection sketching [WIP] #80.
The SOMASlice object currently has a (well-isolated) runtime dependency on AnnData for its concat -- namely, it converts a list of SOMASlice to a list of AnnData, leverages AnnData's concat, then converts the concatenated AnnData object back to SOMASlice.

Update

#173 is the re-do

apis/python/examples/soma-slice-query.md

ebezzi · 2022-06-10T13:08:44Z

apis/python/examples/soma-slice-query.md

+]:
+    soma_path = os.path.join(soco_path, name)
+    soma = tiledbsc.SOMA(soma_path)
+    tiledbsc.io.from_h5ad(soma, h5ad)


I would argue that this is a non-pythonic way to call a function with such a name. Generally, I would expect that a from_* function returns the created object, instead of mutating an argument. I understand that the object is disk-based so it makes sense to pre-create it and pass it as an argument, but then I'd probably rename the function to something that makes the side effect more explicit (like write_from_h5ad). @bkmartinjr your opinion?

@ebezzi originally it was soma.from_h5ad(h5ad) but there was a desire to have a separate tiledbsc.io namespace (#83) in order to more clearly separate I/O (peripheral) from SOMA-API processing (central).

I'm not sure if it is non-pythonic (I lack expertise to make that determination). But I do find it awkward and likely error-prone as the two step process (create the soma, write the soma) allows for cases that will fail (eg, the soma is modified before from_h5ad is called).

I personally prefer a single function that both creates the soma and ingests the H5AD, eg, soma.io.from_h5ad(soma_path: str, h5ad_path: str, ...) -> soma. Ie, this function would call the SOMA constructor.

good to know @bkmartinjr -- thanks for the feedback! I'm fine with this; only wish I'd known sooner :)

Well...I am unclear who is establishing the API and the review process for it. So I'm just giving you my opinions :-)

@bkmartinjr @ebezzi I'll go ahead & make this change but on a separate PR -- this soma.io business is old & predates this PR, and this change can be neatly factored out

Maybe it's worth posting this as a separate issue for discussion/feedback? The R API currently works this way too (ie, ingestion and SOMA instantiation are separate steps) and altering that behavior would require a lot of small updates to docs/tests/etc.

aaronwolen

Still working through it but I posted a couple of comments for now.

apis/python/src/tiledbsc/annotation_dataframe.py

apis/python/src/tiledbsc/soma.py

…SingleCell into kerl/soma-slice

johnkerl · 2022-06-20T15:25:33Z

After rebases this PR is completely out of control (> 200 commits including mostly merge commits).

Closing, and will resubmit.

johnkerl · 2022-06-20T17:00:41Z

re-opening long enough to squash

johnkerl · 2022-06-20T17:06:38Z

failed squash :(

johnkerl · 2022-06-20T18:32:40Z

#173 is the re-do

* Re-do of #157 * remove now-duplicate file * bring an unaffected example file up to date with main * add var.feature_name to examples/collection_counts.py * mkmd.sh doc-gen * Allow obs_ids/var_ids and obs_query_string/var_query_string in soco/soma attribute_filter * mkmd.sh doc-gen * attribute_filter -> query * manual testing

…build-python311-0-1_h50996c Rebuild for python311

johnkerl requested review from aaronwolen, bkmartinjr and Shelnutt2 June 9, 2022 19:47

johnkerl marked this pull request as ready for review June 9, 2022 19:47

johnkerl mentioned this pull request Jun 9, 2022

SOMA-collection sketching [WIP] #80

Closed

johnkerl force-pushed the kerl/soma-slice branch 2 times, most recently from fb77162 to fc07e3c Compare June 9, 2022 19:51

johnkerl changed the title ~~SOMA-level dimension slicing [RFC]~~ SOMA-collection-level dimension slicing [RFC] Jun 9, 2022

johnkerl force-pushed the kerl/soma-slice branch from fc07e3c to 43cbdad Compare June 9, 2022 20:21

johnkerl mentioned this pull request Jun 9, 2022

Progress tracker #113

Closed

61 tasks

johnkerl force-pushed the kerl/soma-slice branch 2 times, most recently from 91a78c9 to 9ebc9a4 Compare June 9, 2022 22:00

ebezzi reviewed Jun 10, 2022

View reviewed changes

aaronwolen reviewed Jun 10, 2022

View reviewed changes

apis/python/src/tiledbsc/annotation_dataframe.py Outdated Show resolved Hide resolved

apis/python/src/tiledbsc/annotation_dataframe.py Outdated Show resolved Hide resolved

apis/python/src/tiledbsc/soma.py Outdated Show resolved Hide resolved

johnkerl commented Jun 10, 2022

View reviewed changes

apis/python/src/tiledbsc/soma.py Show resolved Hide resolved

aaronwolen self-assigned this Jun 10, 2022

johnkerl force-pushed the kerl/soma-slice branch 14 times, most recently from 771d3f5 to 5470ff2 Compare June 11, 2022 03:46

johnkerl added 13 commits June 20, 2022 11:19

move cartographer script from tools to examples

955e999

minor neatens

9568b91

python-doc autogen from mkmd.sh

216ba17

doc-neatens

78ebe26

omit normalization examples

12b572c

split out #168 to make this PR smaller

277d288

split out #172 to make this PR smaller

e1f1a3a

split out #170 to make this PR smaller

3e7fd8b

split out #169 to make this PR smaller

4284fb7

split out #167 to make this PR smaller

65ff1d6

mkmk.sh updates for last few commits

b6e7f03

fix unit test

39af5a3

Merge branch 'kerl/soma-slice' of github.com:single-cell-data/TileDB-…

a5b6907

…SingleCell into kerl/soma-slice

johnkerl closed this Jun 20, 2022

johnkerl reopened this Jun 20, 2022

johnkerl closed this Jun 20, 2022

johnkerl added a commit that referenced this pull request Jun 20, 2022

Re-do of #157

422279d

johnkerl mentioned this pull request Jun 20, 2022

Preliminary SOMA-collection slice/batch queries #173

Merged

johnkerl added a commit that referenced this pull request Jun 20, 2022

Re-do of #157

5b88891

johnkerl added a commit that referenced this pull request Jun 20, 2022

Re-do of #157

2f890a6

johnkerl added a commit that referenced this pull request Jun 21, 2022

Re-do of #157

84acfe2

johnkerl added a commit that referenced this pull request Jun 21, 2022

Re-do of #157

3e05ce7

johnkerl added a commit that referenced this pull request Jun 21, 2022

Re-do of #157

22d61d7

johnkerl deleted the kerl/soma-slice branch June 23, 2022 13:57

nguyenv pushed a commit to nguyenv/TileDB-SOMA that referenced this pull request Jan 10, 2024

Merge pull request single-cell-data#157 from regro-cf-autotick-bot/re…

d61a04b

…build-python311-0-1_h50996c Rebuild for python311

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SOMA-collection-level slice/batch queries [needs revision] #157

SOMA-collection-level slice/batch queries [needs revision] #157

johnkerl commented Jun 9, 2022 •

edited

Loading

ebezzi Jun 10, 2022

johnkerl Jun 10, 2022

bkmartinjr Jun 13, 2022

johnkerl Jun 13, 2022

bkmartinjr Jun 13, 2022

johnkerl Jun 13, 2022

aaronwolen Jun 13, 2022

aaronwolen left a comment

johnkerl commented Jun 20, 2022

johnkerl commented Jun 20, 2022

johnkerl commented Jun 20, 2022

johnkerl commented Jun 20, 2022

SOMA-collection-level slice/batch queries [needs revision] #157

SOMA-collection-level slice/batch queries [needs revision] #157

Conversation

johnkerl commented Jun 9, 2022 • edited Loading

Overview

Walkthrough

Notes

Update

ebezzi Jun 10, 2022

Choose a reason for hiding this comment

johnkerl Jun 10, 2022

Choose a reason for hiding this comment

bkmartinjr Jun 13, 2022

Choose a reason for hiding this comment

johnkerl Jun 13, 2022

Choose a reason for hiding this comment

bkmartinjr Jun 13, 2022

Choose a reason for hiding this comment

johnkerl Jun 13, 2022

Choose a reason for hiding this comment

aaronwolen Jun 13, 2022

Choose a reason for hiding this comment

aaronwolen left a comment

Choose a reason for hiding this comment

johnkerl commented Jun 20, 2022

johnkerl commented Jun 20, 2022

johnkerl commented Jun 20, 2022

johnkerl commented Jun 20, 2022

johnkerl commented Jun 9, 2022 •

edited

Loading