feat(all): enable passing in-memory data to create_table #9251
Conversation
|
|
||
|
|
||
| @lazy_singledispatch | ||
| def _read_in_memory( |
There was a problem hiding this comment.
Ideally this could be ibis.memtable()
There was a problem hiding this comment.
Yeah, and that would unify the implementations across the backends, too. I'll open an follow-up to make use of the lazy single-dispatching for memtable insertion for the in-process backends.
|
Question: would folks rather I add |
Signed-off-by: Gil Forsyth <[email protected]>
supports pandas, polars, and pyarrow tablelikes
This means you can actually select a database with `list_tables`
I don't know that we can unregister `_clean_up_tmp_table` for specific tables, so Exasol might throw some atexit errors (which are ignored) at shutdown, because it's attempting to drop tables that have already been dropped (also not sure why Exasol complains about this with `force=True`). Still, I think it's better to not pollute the table-space with copies of memtables for every table we create.
Co-authored-by: Phillip Cloud <[email protected]>
This branch initially started with my adding `read_in_memory` everywhere before we settled on making this functionality part of `create_table` instead. This hasn't landed in a release, so I'm removing it.
gforsyth
force-pushed
the
ibis-create-table-in-memory
branch
from
a5d34a3 to
1bfbdb8
Compare
gforsyth
force-pushed
the
ibis-create-table-in-memory
branch
from
1bfbdb8 to
4e45c10
Compare
|
@gforsyth Can we add polars to one or two backends instead of all of them? |
And we also have it installed already in the postgres torch build and DuckDB.
gforsyth
force-pushed
the
ibis-create-table-in-memory
branch
from
1c5abe5 to
9a3a698
Compare
Yep -- added it explicitly to MySQL, MSSQL, and Oracle. And we already have it available on the DuckDB jobs, and the Postgres torch job (and obviously on the polars jobs) -- seems like reasonably good coverage? |
This PR adds/codifies support for passing in-memory data to
create_table.The default behavior for most backends is to first create a
memtablewithwhatever
objis passed tocreate_table, then we create a table based on thatmemtable-- because of this, semantics aroundtemptables andcatalog.databaselocations are handled correctly.After the new table (that the user has provided a name for) is created, we
drop the intermediate
memtableso we don't add two tables for every in-memoryobject passed to
create_table.Currently most backends fail when passed
RecordBatchReaders, or a singleRecordBatch, or apyarrow.Dataset-- if we add support for these tomemtable, all of those backends would start working, so I've marked thosexfails as
notimplfor now.A few backends don't work this way:
polarsreads in the table directly using their fast-path local-memory reading stuff.datafusionuses a fast-path read, then creates a table from the table that iscreated by the fast-path -- this is because the
datafusiondataframe API hasno way to specify things like
overwrite, or table location, but the CTAS fromalready present tables is very quick (and possibly zero-copy?) so no issue
there.
duckdbhas a refactoredread_in_memory(which we should deprecate), but itisn't entirely hooked up inside of
create_tableyet, so some paths may go viamemtablecreation, butmemtablecreation on DuckDB is especially fast, soI'm all for fixing this up eventually.
pysparkworks with the intermediatememtable-- there are possiblyfast-paths available, but they aren't currently implemented.
pandasanddaskhave a custom_convert_objectpathTODO:
[ ] FlinkFlink can't create tables from in-memory data?read_in_memoryfrom datafusion and polarsResolves #6593
xref #8863
Signed-off-by: Gil Forsyth [email protected]