Nullable string columns

[ivirshup]

Split off from #504

It would be nice to have support for nullable string arrays. It would be good to have a consistent in-memory representation for these so we can reason about performance. However, this does not currently exist in our dependency stack. I currently think this feature will be dependent on upstream developments in pandas StringArray type.

This is less urgent than nullable integers and booleans since we already have nullable categorical arrays, and currently aggressively cast strings to categorical for performance reasons.

[ivirshup]

Pandas now has multiple ways of doing strings, custom and via arrow.

I think we could still handle both these cases by just storing a mask. This would just be a little inefficient, but we can always update.

Maybe we could even handle arrow bit masks if those seem to be the path forward (docs for bit masks, docs for np.packbits)

[pcm32]

This is also relevant for the output of sc.tl.filter_rank_genes_groups in scanpy, which makes some of the genes in the newly created uns part nan:

adata.uns['rank_genes_groups_filtered']['names'][0]
(nan, nan, 'NKG7', nan, nan, 'PPBP')

I'm adding it here since this seems to be related to the issue of h5py > 3.0 not being happy with casting non strings to strings:

TypeError: Can't implicitly convert non-string objects to strings

Above error raised while writing key 'names' of <class 'h5py._hl.group.Group'> to /

Let me know if you prefer me to open a new issue on Scanpy.

[ivirshup]

This issue wouldn't apply for rank genes groups because that object is a record array, while this issue addresses dataframe columns specifically.

[pcm32]

No problem, should I open a new one here or on scanpy?

[ivirshup]

@pcm32, sorry for the late response here. Came at a busy time of year.

I believe there will already be issues open on scanpy for this.

[ivirshup]

About implementation for nullable string support:

This is somewhat complicated by pandas having multiple backends for nullable string arrays (pyarrow and pd.StringDtype).

We probably want to go with an on disk representation of arrays similar to the arrow in memory representation, but it seems configurable in pandas whether we get the pyarrow representation or the pandas rep. We also don't want to add a hard dependency on pyarrow.

I'm also not sure how we can go from the pandas representation to something writable. We can easily get the masks (.isna()), but I don't know what to handle the "data" containing pd.NA values. I think we probably just make a copy of the array and fill replace the NA entries with some other string, but idk that there's a great choice here.

[ivirshup]

This is starting to come up more frequently, and will likely be even more of an issue with the next release of pandas (which is coming soon).

To head that off, I think I'm going to add this feature for 0.9

[nroak]

Thank you for this update, is there an estimated timeline for this issue to be patched? I'm facing it exactly where you explained- due to nan in ranked gene lists.

[flying-sheep]

OK, to be clear: This issue means support for pandas.core.arrays.string_.StringArray as described in #963, right?

[MishalAshraf]

had a similar issue when trying to concatenate objects. one dataset has a boolean obs column, the other does not. when they are combined it becomes True/False/NaN. I agree that it's up to the user to determine and explicitly define how they want to handle this.

For me, this is 1 step in a longer pipeline for data exploration. It would be nice to have a lazy save here where I concatenate multiple datasets without explicitly resolving which columns will be informative later on.

[flying-sheep]

milestone was missed, bumping to 0.10.0

[jacksonloper]

So basically the current best-practice workaround is to cast to Categorical, is that right?

[flying-sheep]

Pandas seems to get a dedicated string array type (the vote is pretty much in favor), but it’s not based on the nullable string array in numpy 2 🤦

• PDEP-14: Dedicated string data type for pandas 3.0 pandas-dev/pandas#58551
• DISC: Supporting numpy StringDType in Pandas pandas-dev/pandas#58503
• ENH: improve StringDType error in from buffer results numpy/numpy#26893
• Supporting UTF-8 data type zarr-developers/zarr-specs#83
  • Draft ZEP 0007: Strings zarr-developers/zeps#47
  • ArrowRecordBatchCodec and vlen string support zarr-developers/zarr-python#2031

nanoarrow can’t write currently

but we can just use our use_nullable_integer code to store nullable string using boolean masks.

[ilan-gold]

Potential solution (also going forward): use settings!

[flying-sheep]

To expand on it: We want opt-in writing so people don’t accidentally write files that can’t be read by most people.

So we’d add a setting that enables writing pd.StringDtype series, and if one of them is encountered without the setting being active, we throw an error that explains how to enable it and why it’s disabled by default.