Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(memtable): add pyarrow.dataset support #10206

Merged
merged 9 commits into from
Sep 26, 2024

Conversation

gforsyth
Copy link
Member

Description of changes

This adds support for using memtable with pyarrow.dataset.Dataset
objects. Mostly this looks the same as the PyArrowTableProxy.

Some deviations:

  • An additional to_pyarrow_lazy method for grabbing the underlying
    object without materializing it, for backends that can perform
    pushdowns (currently only enabled with DuckDB).
  • Datasets are hashable objects, so we override the PseudoHashable
    hash implementation (this seemed like less trouble than special-casing
    this one memtable everywhere else we look for TableProxy objects)
  • I've added a cached_property so we can avoid repeatedly
    materializing the data -- this does not apply to the lazy view.

Issues closed

Resolves #10082

@github-actions github-actions bot added tests Issues or PRs related to tests duckdb The DuckDB backend labels Sep 24, 2024
@gforsyth
Copy link
Member Author

I'm going to xfail polars for now -- I think we can strip out all of the custom in-memory reading stuff and that will make this work, but that seems like a separate PR

ibis/formats/pyarrow.py Outdated Show resolved Hide resolved
ibis/backends/tests/test_client.py Show resolved Hide resolved
ibis/formats/pyarrow.py Outdated Show resolved Hide resolved
ibis/formats/pyarrow.py Outdated Show resolved Hide resolved
@gforsyth gforsyth linked an issue Sep 25, 2024 that may be closed by this pull request
1 task
ibis/backends/duckdb/__init__.py Outdated Show resolved Hide resolved
ibis/formats/pyarrow.py Outdated Show resolved Hide resolved
gforsyth and others added 7 commits September 26, 2024 05:42
This adds support for using `memtable` with `pyarrow.dataset.Dataset`
objects. Mostly this looks the same as the `PyArrowTableProxy`.

Some deviations:
- An additional `to_pyarrow_lazy` method for grabbing the underlying
object without materializing it, for backends that can perform
pushdowns (currently only enabled with DuckDB).
- Datasets are hashable objects, so we override the PseudoHashable
`hash` implementation (this seemed like less trouble than special-casing
this one memtable everywhere else we look for `TableProxy` objects)
- I've added a `cached_property` so we can avoid repeatedly
materializing the data -- this does _not_ apply to the `lazy` view.
@cpcloud cpcloud added this to the 10.0 milestone Sep 26, 2024
@cpcloud cpcloud enabled auto-merge (squash) September 26, 2024 09:49
@cpcloud cpcloud merged commit 428d1a3 into ibis-project:main Sep 26, 2024
77 checks passed
@gforsyth gforsyth deleted the dataset_memtable branch November 29, 2024 22:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
duckdb The DuckDB backend tests Issues or PRs related to tests
Projects
None yet
3 participants