[Feature request] ExperimentAxisQuery.to_anndata() (and R analogs) should drop unused categories on axis data frames

[pablo-gar]

Is your feature request related to a problem? Please describe.
Some analytical pipelines, specially those that relate to visualizations rely on the categories of pandas.Categorical. In the case of large SOMAExperiments, many times a query will result on unused categories for potentially many columns of obs or var, thus the user needs to always iterate on all columns and perform a cat.remove_unused_categories() operation.

See for example this reproducible example

import cellxgene_census
import scanpy as sc
census = cellxgene_census.open_soma(census_version="2024-05-20")

human = census["census_data"]["homo_sapiens"]
query = human.axis_query(
    measurement_name = "RNA",
    obs_query = tiledbsoma.AxisQuery(
        value_filter = "tissue == 'tongue' and is_primary_data == True"
    )
)

adata = query.to_anndata(column_names={"obs": ["tissue"]}, X_name = "normalized")
sc.pp.pca(adata)
sc.pp.neighbors(adata)
sc.tl.umap(adata)
sc.pl.umap(adata, color="tissue")

Only one "tissue" was selected but all hundreds of tissues are drawn in the umap

Describe the solution you'd like
ExperimentAxisQuery.to_anndata() returns an anndata with unused categories already removed in the axis data frames

[johnkerl]

Needs Python & R both. We can subtask.

[eddelbuettel]

For R it is a pretty common task and a base R function

> example(droplevels)

drplvl> aq <- transform(airquality, Month = factor(Month, labels = month.abb[5:9]))

drplvl> aq <- subset(aq, Month != "Jul")

drplvl> table(           aq $Month)

May Jun Jul Aug Sep 
 31  30   0  31  30 

drplvl> table(droplevels(aq)$Month)

May Jun Aug Sep 
 31  30  31  30 
> 

[mojaveazure]

I get why this is asked for, but I would not want to do it by default in the R API. R explicitly keeps unused factor levels, and while this is frustrating, it's a practice I adhere to even in Seurat. We make this easier in Seurat by exposing drop arguments to drop unused factor levels when needed (set to FALSE by default to adhere to R) and building methods to droplevels() to automagically prune unused factor levels. However, we require explicit user input to drop unused levels in order to fit in with the R way of doing things

[eddelbuettel]

It's a topic that has generated heated online discussions in other places. @pablo-gar raises a good point with respect to the plot and its 'inflated' legend -- but as @mojaveazure noted there is also a consensus that retaining factor levels as default is preferable. One can compare this to enums in C/C++ where I find the analogy of the picky compiler warning asking us to supply all levels of an enum when we write a switch appropriate: doing otherwise may lead to gnarly silent bugs. This is a tricky question. My preference would be to do what Seurat does and offer an option to drop if requested, and I lean towards a default of 'off'.

[johnkerl]

Let's make this opt-in for Python and R both (default: keep all levels; drop levels only when requested) -- @pablo-gar will this meet your needs?

[johnkerl]

Needs R PR for Seurat/SCE export

[eddelbuettel]

While I am assigned may I suggest this is pivoted over to @mojaveazure instead?

[ivirshup]

I agree with opt in, we're going to move anndata over to this.

We initially went with opt out because categoricals were new in pandas and a lot of code threw errors when there were unused categories. It's generally much better now, and there are cases where you want to retain the categories.