Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support querying polars lazyframes #1770

Open
universalmind303 opened this issue Sep 15, 2023 · 1 comment
Open

support querying polars lazyframes #1770

universalmind303 opened this issue Sep 15, 2023 · 1 comment
Labels
feat 🎇 New feature or request

Comments

@universalmind303
Copy link
Contributor

Description

currently I can query polars eager dataframes

import glaredb
import polars as pl

df = pl.DataFrame(
    {
        "A": [1, 2, 3, 4, 5],
        "fruits": ["banana", "banana", "apple", "apple", "banana"],
    }
)
con = glaredb.connect()
df = con.sql("select * from df where fruits = 'banana'").to_polars()
print(df)
con.close()

but I can't query lazyframes

lf = df.lazy()
df = con.sql("select * from lf where fruits = 'banana'").to_polars()
print(df)
con.close()
@universalmind303 universalmind303 added the feat 🎇 New feature or request label Sep 15, 2023
@universalmind303
Copy link
Contributor Author

universalmind303 commented Sep 15, 2023

I was thinking we could use substrait to convert the LP to a datafusion LP, but this does require implementing to/from substrait in polars as well. pola-rs/polars#7404

We could also just parse the logicalplan directly to datafusion LP, but that seems like something that could greatly benefit others as well, so I think it'd make sense to find some way make that shareable (like going through substrait).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feat 🎇 New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant