[r] Add enumeration support

[eddelbuettel]

Issue and/or context:

As described in #1558 and #866, adding enumeration support is desirable once we have TileDB Embedded 2.17 available

Changes:

This PR supports reading of columns with enumerations (aka dictionaries aka factor variable) directly via Arrow. Preliminary write support is also available (but still goes through the tiledb R package for writes).

Notes for Reviewer:

This PR is now work-in-progress and not ready for a merge while we await TileDB 2.17. The branch and PR are ready but should only be merged once prequisites are been merged. It likely needs #1519 (C++ side) and #1663 (CI support).

CI is turned off as the TileDB default build is still without support for enumerations.

[shortcut-integration]

This pull request has been linked to Shortcut Story #32235: [r] Support enumerations in SOMA.

[eddelbuettel]

A quick snapshot of what works as of today / this branch when 2.17 is available.

Code

library(tiledbsoma)
library(tiledb)

pp <- palmerpenguins::penguins
uri <- tempfile(pattern="tiledb")

fromDataFrame(pp, uri)

## read one: soma_array_reader
p <- soma_array_reader(uri)
print(tibble::as_tibble(arrow::RecordBatch$import_from_c(p[[1]], p[[2]])))

## read twp: iterated reader for sdf
p <- SOMADataFrameOpen(uri)$read()$concat()
print(tibble::as_tibble(p))

Output

# A tibble: 344 × 9
   `__tiledb_rows` species island    bill_length_mm bill_depth_mm flipper_length_mm body_mass_g sex     year
             <int> <fct>   <fct>              <dbl>         <dbl>             <int>       <int> <fct>  <int>
 1               1 Adelie  Torgersen           39.1          18.7               181        3750 male    2007
 2               2 Adelie  Torgersen           39.5          17.4               186        3800 female  2007
 3               3 Adelie  Torgersen           40.3          18                 195        3250 female  2007
 4               4 Adelie  Torgersen           NA            NA                  NA          NA NA      2007
 5               5 Adelie  Torgersen           36.7          19.3               193        3450 female  2007
 6               6 Adelie  Torgersen           39.3          20.6               190        3650 male    2007
 7               7 Adelie  Torgersen           38.9          17.8               181        3625 female  2007
 8               8 Adelie  Torgersen           39.2          19.6               195        4675 male    2007
 9               9 Adelie  Torgersen           34.1          18.1               193        3475 NA      2007
10              10 Adelie  Torgersen           42            20.2               190        4250 NA      2007
# ℹ 334 more rows
# A tibble: 344 × 9
   `__tiledb_rows` species island    bill_length_mm bill_depth_mm flipper_length_mm body_mass_g sex     year
             <int> <fct>   <fct>              <dbl>         <dbl>             <int>       <int> <fct>  <int>
 1               1 Adelie  Torgersen           39.1          18.7               181        3750 male    2007
 2               2 Adelie  Torgersen           39.5          17.4               186        3800 female  2007
 3               3 Adelie  Torgersen           40.3          18                 195        3250 female  2007
 4               4 Adelie  Torgersen           NA            NA                  NA          NA NA      2007
 5               5 Adelie  Torgersen           36.7          19.3               193        3450 female  2007
 6               6 Adelie  Torgersen           39.3          20.6               190        3650 male    2007
 7               7 Adelie  Torgersen           38.9          17.8               181        3625 female  2007
 8               8 Adelie  Torgersen           39.2          19.6               195        4675 male    2007
 9               9 Adelie  Torgersen           34.1          18.1               193        3475 NA      2007
10              10 Adelie  Torgersen           42            20.2               190        4250 NA      2007
# ℹ 334 more rows

[johnkerl]

@eddelbuettel can you please resolve conflicts?

[eddelbuettel]

Rebased again (which was a little more involved than usual), passes 1930 tests here.

(I used to have all CI turned off waiting for the merge of TileDB Core, that regressed so some read X now.)

[codecov-commenter]

Codecov Report

Patch has no changes to coverable lines.

❗ Current head 513b478 differs from pull request most recent head 10a94cd. Consider uploading reports for the commit 10a94cd to get more accurate results

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

📢 Thoughts on this report? Let us know!.

[johnkerl]

🚢

Pending CI of course! 😎

[eddelbuettel]

Pending CI of course! 😎

I should probaby update the libtiledbsoma side from the rc to the release too. Or will you take of that 'soon' (pending the unit test update) ?

(And I do expect this to go ❌ on python interop as the last ones did....) And so it happens. So I thing this needs some work somewhere else first, then a rebase and then we can talk

[johnkerl]

@eddelbuettel I tried #1677 as a candidate to go underneath #1519 #1511 #1559. There's a CI issue I'm puzzling over. Let's chat in Slack.

[eddelbuettel]

@johnkerl I am lost. Might be a candidate for outreach to the author of these contributed tests too.

[johnkerl]

The failing CI step is R/Python interop which is here
https://github.com/single-cell-data/TileDB-SOMA/blob/main/.github/workflows/r-python-interop-testing.yml
and it does quite simply python -m pip -v install -e apis/python, and apis/python/setup.py has tiledb (tiledb-py) dependency still 0.22.2 (you can see core 2.16.3 in the show_package_versions() output in the failed CI job). This is where we are at until TileDB-Py 0.23.0 is released. So, let's wait for that.

[eddelbuettel]

Ok, did a first rebase (which was much smother than a local one I had recently) so fingers crossed!

[eddelbuettel]

Ok if we all hold our breath for 20 minutes I am sure this run will turn all green....

[johnkerl]

Ok if we all hold our breath for 20 minutes I am sure this run will turn all green....

@eddelbuettel That's the plan! :)

[eddelbuettel]

Cnttlkrtksmhldngmbrthshrd

[eddelbuettel]

(That was "Can't type or talk as I am holding my breath" approximately free of vowels ... 😉 )

[eddelbuettel]

Yay yay yay

Not so fast. One more.