feat: pyarrow datasets in future versions

[szst11]

Is your feature request related to a problem?

I got the message

FutureWarning: `Backend.register` is deprecated as of v9.1; use the explicit `read_*` method for the filetype you are trying to read, e.g., read_parquet, read_csv, etc.

and

FutureWarning: `Backend.read_in_memory` is deprecated as of v9.1, removed in v10.0; Pass in-memory data to `create_table` instead.

when I try to register a pyarrow Dataset

What is the motivation behind your request?

The usage of a pyarrow Dataset enables the usage of many parquet files, which are indexed only once at creation of the dataset.
For now the .register() works for that.

If I read the parquet files directly with DuckDB they are indexed on every real request.

The other option create_table is also not optimal, as it loads the dataset into the database, but I would keep the data within the parquet file and keep the pushdown filtering.

Describe the solution you'd like

I'd like to be able to still access a pyarrow dataset without copying the data into memory.

What version of ibis are you running?

<10

What backend(s) are you using, if any?

DuckDB

Code of Conduct

• I agree to follow this project's Code of Conduct

[gforsyth]

Hey @szst11 -- I'm working on that in #10206 -- you'll be able to pass the dataset to ibis.memtable and then use that in DuckDB without materializing the data and with pushdowns working.