feat: allow registering all in-memory table types via create_table

[jcrist]

Summarizing the discussions below, here's my (@gforsyth) rough outline:

• Add fallback behavior to all backends such that if create_table or insert gets a non-ibis-expression as obj, convert it to a memtable and then continue.
• For Polars, Datafusion, pandas, and dask, use the native methods to read in these objects
• For DuckDB, use native methods to read in these objects, then create a table from the object so that create_table is actually creating a table.
• For Polars, Datafusion, DuckDB, pandas, and dask, additionally support passing in-memory objects to create_view (for all but DuckDB, the distinction between views and tables carries no meaning)

---

Is your feature request related to a problem?

Originally we had con.register to handle data ingestion of anything a backend could support. This method works fine and is defined for many backends, but is a bit "magical" in that it determines the input format based on the type and value of the input. Later on we split part of this functionality out into standard read_csv/read_parquet/read_* methods, which register dispatches to.

The duckdb backend also has a read_in_memory method for reading from an in-memory object. Unlike files (where we dispatch based on things like the file extension), in memory data always has an explicit python type, so having a method like read_in_memory to handle any in-memory data sources seems like a nice interface. It would be good to standardize this interface across all the backends that could support it. For the most part this would be extracting out the existing functionality from the .register methods and moving it to a new read_in_memory method.

Describe the solution you'd like

For any backend that could support it to define a method like:

def read_in_memory(self, source: pd.DataFrame | pa.Table | pa.RecordBatchReader, table_name: str | None = None) -> ir.Table:
    ...

What version of ibis are you running?

dev

What backend(s) are you using, if any?

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[chloeh13q]

I might be missing some context/motivation, but I was wondering: would this not function effectively the same as create_table() with an in-memory obj?

[gforsyth]

We discussed this today briefly and I think @chloeh13q is right -- this ends up being (mostly) a specialized version of passing an in-memory object to create_table.

Currently, if the obj isn't an Ibis expression, we convert it to a memtable and then create the table from that.

I think that's a reasonable default/fallback behavior for most backends and allows for users to insert pyarrow tables, polars dataframes and pandas dataframes into all backends. This is also currently implemented.

For DuckDB, Datafusion, and Polars, there are extra options available.

For Datafusion and Polars, there are native methods to read in pyarrow tables, pyarrow recordbatchreaders, polars dataframes, pandas dataframes, etc. Datafusion creates all of these objects as "Tables", Polars does not distinguish between tables and views, so they are also created as what Ibis refers to as "tables".

Adding extra handling for the fast-path for querying these objects will allow us to deprecate register without also introducing a new read_in_memory to these backends. We can also deprecate some of the existing format-specific methods in those backends (e.g. polars read_pandas).

DuckDB also has fast-paths for querying in-memory objects -- if we use the DuckDB connection register method, this will create a view, though, not a table. This seems like a potential source of confusion for users -- I'd expect to have a persistent table if I pass something to create_table.

Maybe we should also support passing in-memory objects to create_view and have that be the fast-path for DuckDB?

[jcrist]

Agreed, makes sense to me!

Maybe we should also support passing in-memory objects to create_view and have that be the fast-path for DuckDB?

I think that makes sense. To clarify, creating a view from an in-memory dataset would only work for backends that would implement it as a true view (meaning querying the in-memory data directly) - effectively limiting con.create_view("test", df) to things like duckdb/pandas/polars? FWIW I think this is more of an optimization thing and not a thing most users would need to do.

[gforsyth]

effectively limiting con.create_view("test", df) to things like duckdb/pandas/polars?

Ehh, there's a weird bifurcation.

DuckDB is the only one of the in-memory friendly systems that has it's own persistent format, so for DuckDB, there's a very real distinction between a view of an arrow table vs. a table of an arrow table.

For pandas, dask, polars, and datafusion, there is effectively no difference between views and tables, since everything is in memory and nothing persists outside of a given session.

I think it may not actually matter all that much -- we'll allow arrow tables to be passed to both create_view and create_table for these 5 backends, and on 4 of them, there will be no difference in how that manifests, and with DuckDB, there's a performance edge-case but not a big one.