[python] Support `Enumeration` in Python API #1511

nguyenv · 2023-06-28T19:06:07Z

Issue and/or context: #866

Changes:

Reads are done in C++. Writes are done through TileDB-Py (TileDB-Inc/TileDB-Py#1790).

Notes for Reviewer:

Depends on #1519.

shortcut-integration · 2023-06-28T19:06:09Z

This pull request has been linked to Shortcut Story #30316: Enumerated data types AKA categoricals AKA factors.

johnkerl · 2023-06-29T20:22:17Z

[sc-30316]

nguyenv · 2023-07-07T13:29:29Z

This PR is now at a point where it can be evaluated for feedback.

johnkerl · 2023-09-06T14:30:25Z

@nguyenv can you please resolve conflicts on this PR to bring it up to date? I'm happy to help if you need.

nguyenv · 2023-09-06T14:33:32Z

@nguyenv can you please resolve conflicts on this PR to bring it up to date? I'm happy to help if you need.

Yes I've been looking at this since yesterday . I'm seeing failures in the most recently added unit tests that I haven't resolved yet.

===================================================================================================== short test summary info ======================================================================================================
FAILED apis/python/tests/test_basic_anndata_io.py::test_obs_with_categorical_int_nan_enumeration - tiledb.cc.TileDBError: URI /home/vivian/TileDB-SOMA/apis/python/testdata/categorical_int_nan.h5ad is not a valid file
FAILED apis/python/tests/test_registration_mappings.py::test_multiples_without_experiment[True-permutation0] - IndexError: Chunk index out of range.
FAILED apis/python/tests/test_registration_mappings.py::test_multiples_without_experiment[True-permutation1] - IndexError: Chunk index out of range.
FAILED apis/python/tests/test_registration_mappings.py::test_multiples_without_experiment[True-permutation2] - IndexError: Chunk index out of range.
=============================================================================== 14 failed, 1358 passed, 3 xfailed, 194 warnings in 653.45s (0:10:53) ===============================================================================
make: *** [Makefile:35: test] Error 1

johnkerl · 2023-09-06T14:41:00Z

@nguyenv those are tests I created -- I'll help take a look

codecov-commenter · 2023-09-14T19:28:59Z

Codecov Report

Patch has no changes to coverable lines.

❗ Current head eeffaea differs from pull request most recent head ae1474d. Consider uploading reports for the commit ae1474d to get more accurate results

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

📢 Thoughts on this report? Let us know!.

* Addition of `SOMAArray::get_enum` and `SOMAArray::get_enum_label_on_attr` * Attach an enumeration/dictionary to the `ColumnBuffer` if applicable; this is used when converting from `ArrayBuffers` to Arrow Tables in the Python and R APIs

* Prior to TileDB-Inc/TileDB#4272, the SOMA unit tests were erroneously writing a byte vector for string dimensions which maps to `TILEDB_BLOB` rather than `TILEDB_STRING_ASCII`

…ions

…ss (#1650) * temp * robustness * extract method for obsm/varm outgest * complete rebase to main * more unit-test cases * remove R debugs * robustness * complete rebase to main * [python] Leverage bounding-box feature for obsm/varm outgest robustness * test data for holey obsm * unit-test cases * on-line help improvements

* [python] Improve schema-printer [RFC] * neaten * code-neaten inspired by dirk's 1675

johnkerl · 2023-09-15T13:24:43Z

This has the same interop fail as #1519 which I am debugging separately: #1679. Otherwise looks good.

nguyenv changed the title ~~[c++/python][wip] Support Enumerations in Python (writes), C++ (reads)~~ [c++/python][wip] Support Enumerations in Python API Jun 28, 2023

nguyenv changed the title ~~[c++/python][wip] Support Enumerations in Python API~~ [c++/python][wip] Support Enumeration in Python API Jun 28, 2023

johnkerl mentioned this pull request Jul 5, 2023

[tracking] Add support for enumerated types aka categoricals aka factors #866

Closed

pablo-gar mentioned this pull request Jul 5, 2023

add enumerated/categorical support chanzuckerberg/cellxgene-census#604

Closed

nguyenv mentioned this pull request Jul 5, 2023

[c++] Support Enumeration in C++ codebase #1519

Merged

nguyenv changed the title ~~[c++/python][wip] Support Enumeration in Python API~~ [python] Support Enumeration in Python API Jul 19, 2023

nguyenv force-pushed the viviannguyen/sc-30316/enumerated-data-types-aka-categoricals-aka branch from 1ad3e2c to 2fbc777 Compare August 17, 2023 19:14

johnkerl mentioned this pull request Aug 18, 2023

[python] Expand unit testing for enumerated types #1624

Merged

nguyenv force-pushed the viviannguyen/sc-30316/enumerated-data-types-aka-categoricals-aka branch from c2c81de to a84e76c Compare August 24, 2023 02:44

johnkerl mentioned this pull request Sep 7, 2023

[c++] Depend on core 2.17.0-rc0 #1662

Closed

nguyenv force-pushed the viviannguyen/sc-30316/enumerated-data-types-aka-categoricals-aka branch from 6e22be8 to 1058f9d Compare September 7, 2023 19:52

johnkerl mentioned this pull request Sep 11, 2023

[c++] Don't use Python to generate C++ test inputs #1668

Merged

nguyenv force-pushed the viviannguyen/sc-30316/enumerated-data-types-aka-categoricals-aka branch from ba4559c to 0687b16 Compare September 11, 2023 18:00

This was referenced Sep 14, 2023

[c++] Depend on core 2.17.0 #1677

Merged

[r] Add enumeration support #1559

Merged

nguyenv marked this pull request as ready for review September 14, 2023 20:59

nguyenv added 7 commits September 14, 2023 16:51

[c++] Support Enumeration in C++ Codebase

02e9a58

* Addition of `SOMAArray::get_enum` and `SOMAArray::get_enum_label_on_attr` * Attach an enumeration/dictionary to the `ColumnBuffer` if applicable; this is used when converting from `ArrayBuffers` to Arrow Tables in the Python and R APIs

Add get_attr_to_enum_mapping Function

4f66cde

Add Unit Tests for Enumeration in C++

17bcb52

to_varlen_buffers Returns std::string

d068d6c

* Prior to TileDB-Inc/TileDB#4272, the SOMA unit tests were erroneously writing a byte vector for string dimensions which maps to `TILEDB_BLOB` rather than `TILEDB_STRING_ASCII`

WIP support enumeration for schema

5d568e5

[python][wip][nomerge] Support Enumerations in Python (writes)

69baafe

Run pre-commit hook

1be5a2d

nguyenv and others added 23 commits September 14, 2023 17:06

Add Unit Tests for Enumeration in C++

d58c471

to_varlen_buffers Returns std::string

3611085

* Prior to TileDB-Inc/TileDB#4272, the SOMA unit tests were erroneously writing a byte vector for string dimensions which maps to `TILEDB_BLOB` rather than `TILEDB_STRING_ASCII`

WIP support enumeration for schema

a415c69

[python][wip][nomerge] Support Enumerations in Python (writes)

953eb12

Run pre-commit hook

02241e8

[python] Expand unit-testing for enumerated types

a82c583

used pre-prepared input for categorical-int-nan data

f84eb75

[python] Support Enumerations On Nullable Attributes and Query Condit…

cd7473f

…ions

WIP

f9be965

Do Not Index Chunk For Empty Arrow Column

8525aaf

fix TypeError: Object of type bool_ is not JSON serializable

408ceb9

Use Dict For Typing

32d3fff

Update Typing

74f2708

Use typed ndarray

a290af6

More Typing Corrections

9eeb862

Recomment tiledb-py dep

5a9e432

Use tiledb-py dep; typing

495f977

[python] Update default-filter-list handling in unit tests (#1676)

e603f4d

[python] Improve schema-printer for enumerated types (#1673)

2035462

* [python] Improve schema-printer [RFC] * neaten * code-neaten inspired by dirk's 1675

2.17.0

554ba0e

fix merge

eebbe2c

merge

f697900

nguyenv force-pushed the viviannguyen/sc-30316/enumerated-data-types-aka-categoricals-aka branch from 14f4387 to f697900 Compare September 14, 2023 22:36

nguyenv and others added 2 commits September 14, 2023 18:39

pre-commit

ae1474d

merge

e52a2d8

johnkerl approved these changes Sep 15, 2023

View reviewed changes

johnkerl merged commit bd13a68 into main Sep 15, 2023

johnkerl deleted the viviannguyen/sc-30316/enumerated-data-types-aka-categoricals-aka branch September 15, 2023 13:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[python] Support `Enumeration` in Python API #1511

[python] Support `Enumeration` in Python API #1511

nguyenv commented Jun 28, 2023 •

edited

Loading

shortcut-integration bot commented Jun 28, 2023

johnkerl commented Jun 29, 2023

nguyenv commented Jul 7, 2023

johnkerl commented Sep 6, 2023

nguyenv commented Sep 6, 2023

johnkerl commented Sep 6, 2023

codecov-commenter commented Sep 14, 2023 •

edited

Loading

johnkerl commented Sep 15, 2023 •

edited

Loading

[python] Support Enumeration in Python API #1511

[python] Support Enumeration in Python API #1511

Conversation

nguyenv commented Jun 28, 2023 • edited Loading

shortcut-integration bot commented Jun 28, 2023

johnkerl commented Jun 29, 2023

nguyenv commented Jul 7, 2023

johnkerl commented Sep 6, 2023

nguyenv commented Sep 6, 2023

johnkerl commented Sep 6, 2023

codecov-commenter commented Sep 14, 2023 • edited Loading

Codecov Report

johnkerl commented Sep 15, 2023 • edited Loading

[python] Support `Enumeration` in Python API #1511

[python] Support `Enumeration` in Python API #1511

nguyenv commented Jun 28, 2023 •

edited

Loading

codecov-commenter commented Sep 14, 2023 •

edited

Loading

johnkerl commented Sep 15, 2023 •

edited

Loading