Querying DuckDB using Polars' logical plan

[agupta726]

Description

I want to build my application logic using the Polars dataframe API - projections, join, filtering etc and then use DuckDB to actually execute the query represented by the Polars dataframe. I suppose one way to think about this would be that I want to use Polars dataframe API only for plan generation (perhaps a Substrait plan?) but not as an engine. I'd want to pass this plan to DuckDB to do the actual execution.
Is there any way to do this? Or can this be accepted as a feature request?

[cmdlineluser]

I think this may have been previously requested:

• Allow duckdb as an engine backend via its "relational API" #15581

Which was deemed not possible due to the reasons explained in: #7404 (comment)

[alexander-beedie]

Curious why you'd want to run the Polars API on the DuckDB engine? I can't see that it's something we would be that interested in developing/supporting 🤔

(Depending on what is required, could be a job for a future iteration of Narwhals, @MarcoGorelli?)

[agupta726]

Curious why you'd want to run the Polars API on the DuckDB engine? I can't see that it's something we would be that interested in developing/supporting 🤔

(Depending on what is required, could be a job for a future iteration of Narwhals, @MarcoGorelli?)

Honestly I'm looking for a Spark like dataframe API for but C++. I have a need to generate complex SQL and I believe using a dataframe API is the right way to do it.

Any recommendations are appreciated.

[cmdlineluser]

Have you seen DuckDB's spark api?

• https://duckdb.org/docs/api/python/spark_api

[agupta726]

Have you seen DuckDB's spark api?

• https://duckdb.org/docs/api/python/spark_api

Thanks @cmdlineluser. I did see this already. I'm looking for a C++ based dataframe API though.

[ritchie46]

I'm looking for a C++ based dataframe API though.

How is Polars going to help then? It is written in Rust.

[MarcoGorelli]

Honestly I'm looking for a Spark like dataframe API for but C++

closing then as out-of-scope

(Depending on what is required, could be a job for a future iteration of Narwhals, @MarcoGorelli?)

watch this space 😉