[python] Use true ASCII attributes in dataframes

[johnkerl]

What

This is a forward-port of #273 from the main-old branch to the main branch.

Why

• Write string dims/attrs as "ascii" per se, not bytes and not str per @Shelnutt2
• Please see also the clear presentation at Queryability of dataframe attribute columns #99

Lack of backward compatibility

• Since old SOMAs written before [python] Update ASCII storage for dataframes #273 didn't have all "ascii", on main-old (see [python] Update ASCII storage for dataframes #273) we conditionally retained the decode-on-readback logic which was essential since otherwise b"foo" != "foo"
• But on this main (greenfield) branch we do not keep backward compatibility -- any pre-existing datasets with non-ASCII attributes will need to be regenerated

Prerequisites

• This depends on Permit true-ASCII attributes in non-from-pandas dataframes TileDB-Inc/TileDB-Py#1337 -- now merged and in TileDB-Py 0.17.5 which is here as of [python] Depend on TileDB-Py 0.17.5 #394
• This PR is stacked atop [python] Conform to spec by reading as pyarrow.Table not pyarrow.RecordBatch #355 -- now merged

PR context

This is the second in a group of three related PRs:

• [python] Conform to spec by reading as pyarrow.Table not pyarrow.RecordBatch #355: Conform to https://github.com/single-cell-data/SOMA/blob/main/abstract_specification.md by having read returnpyarrow.Table not pyarrow.RecordBatch (as in an outdated version of that spec)
• This PR: Foreward-port [python] Update ASCII storage for dataframes #273 from main-old which will truly have ASCII columns, obviating the need for our util_arrow.ascii_to_unicode_pyarrow_readback
• [python/c++] Connect libtiledbsoma to tiledbsoma readers for dataframes [WIP] #360: Drop in C++ acceleration-library support for read methods, which will go in cleanly now
  • The C++ code returns pyarrow.Table and with the first PR our unit tests will be ready to go
  • When the C++ code reads Unicode cells it returns them as pyarrow.LargeBinaryArray (needing decode) but when we are properly writing ASCII cells via the Python write path then the C++ code will read ASCII cells and return them as strings (no longer needing decoding)

These three changesets could be done in a single PR, but that would be unmerciful to the reviewers.

Problem

Issue I'm seeing on this PR which confuses me:

Script:

#!/usr/bin/env python

import os, shutil
import pyarrow as pa
import tiledbsoma as t
import tiledb

uri = "foo"
if os.path.exists(uri):
    shutil.rmtree(uri)

schema = pa.schema(
    [
        ("soma_rowid", pa.uint64()),
        ("A", pa.int64()),
        ("B", pa.float64()),
        ("C", pa.string()),
    ]
)

sdf = t.SOMADataFrame(uri=uri)
sdf.create(schema=schema.remove(schema.get_field_index("soma_rowid")))

data = {
    "soma_rowid": [0, 1, 2, 3],
    "A": [10, 11, 12, 13],
    "B": [100.1, 200.2, 300.3, 400.4],
    "C": ["this", "is", "a", "test"],
}
n_data = len(data["soma_rowid"])
rb = pa.Table.from_pydict(data)
sdf.write(rb)

TileDB schema:

ArraySchema(
  domain=Domain(*[
    Dim(name='soma_rowid', domain=(0, 18446744073709551614), tile=2048, dtype='uint64', filters=FilterList([ZstdFilter(level=3), ])),
  ]),
  attrs=[
    Attr(name='A', dtype='int64', var=False, nullable=False, filters=FilterList([ZstdFilter(level=-1), ])),
    Attr(name='B', dtype='float64', var=False, nullable=False, filters=FilterList([ZstdFilter(level=-1), ])),
    Attr(name='C', dtype='ascii', var=True, nullable=False, filters=FilterList([ZstdFilter(level=-1), ])), <----- AS DESIRED
  ],
  cell_order='row-major',
  tile_order='row-major',
  capacity=100000,
  sparse=True,
  allows_duplicates=False,
)

Reading back from C++ or from Python:

>>> import tiledbsoma.libtiledbsoma as clib
>>> clib.SOMAReader('foo')
<tiledbsoma.libtiledbsoma.SOMAReader object at 0x7fe69b851070>
>>> sr=clib.SOMAReader('foo')
>>> sr.
sr.read_next(        sr.results_complete( sr.set_dim_points(   sr.set_dim_ranges(   sr.submit(
>>> sr.submit()
>>> o=sr.read_next()
>>> o
pyarrow.Table
soma_rowid: uint64
A: int64
B: double
C: large_string <------- this is why I am seeing unit-test failures -- same reason
----
soma_rowid: [[0,1,2,3]]
A: [[10,11,12,13]]
B: [[100.1,200.2,300.3,400.4]]
C: [["this","is","a","test"]]

In summary:

• TileDB arrays are created with "ascii" attributes
• When I read them back as Arrow using C++ or Python, I now get large_string not string in the Arrow readback schema

cc @gspowley and @nguyenv

@nguyenv I'll get you a simpler reprex tomorrow

Update 2022-10-04: simpler repro https://gist.github.com/johnkerl/278bb246c3aa05cc4dfb1694503744c6

[johnkerl]

fyi, now that python unit-test cases are passing, there are C++ unit-test failures which i am currently debugging

[johnkerl]

fyi, now that python unit-test cases are passing, there are C++ unit-test failures which i am currently debugging

fixed on latest commit:

9bdd02f

[johnkerl]

Note: tests will fail with

E   TypeError: dtype is not compatible with var-length attribute

until TileDB-Inc/TileDB-Py#1337 wends its way into a TileDB-Py release we can update to -- cc @nguyenv and @ihnorton