Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

AggregateUDF expression API design #10545

Closed
jayzhan211 opened this issue May 16, 2024 · 2 comments · Fixed by #10560
Closed

AggregateUDF expression API design #10545

jayzhan211 opened this issue May 16, 2024 · 2 comments · Fixed by #10560
Assignees
@jayzhan211

Comments

@jayzhan211
Copy link
Contributor

File an issue to track the design of UDAF API

Perhaps something like

// form `FIRST_VALUE(a ORDER BY b)`
let agg_expr = AggregateUDF::call(first_value())
  .args(col("a"))
  .order_by(vec![col("b"))
  .build()?;

Originally posted by @alamb in #10492 (comment)
@alamb
Copy link
Contributor

alamb commented May 16, 2024

related to #6747 (for window APIs)

@jayzhan211 jayzhan211 self-assigned this May 17, 2024
@jayzhan211 jayzhan211 mentioned this issue May 17, 2024
Merged
@alamb
Copy link
Contributor

alamb commented Jun 8, 2024

#10560 is looking sweet to me

Michael-J-Ward added a commit to Michael-J-Ward/datafusion-python that referenced this issue Jun 13, 2024
@Michael-J-Ward
update signature for covar_samp
232ab81 
AggregateUDF expressions now have a builder API design, which removes arguments like filter and order_by

Ref apache/datafusion#10545
Ref apache/datafusion#10492
andygrove pushed a commit to apache/datafusion-python that referenced this issue Jun 14, 2024
@Michael-J-Ward
Upgrade datafusion 39 (#728)
b5446ef 
* deps: update datafusion to 39.0.0, pyo3 to 0.21, and object_store to 0.10.1

`datafusion-common` also depends on `pyo3`, so they need to be upgraded together.

* feat: remove GetIndexField

datafusion replaced Expr::GetIndexField with a FieldAccessor trait.

Ref apache/datafusion#10568
Ref apache/datafusion#10769

* feat: update ScalarFunction

The field `func_name` was changed to `func` as part of removing `ScalarFunctionDefinition` upstream.

Ref apache/datafusion#10325

* feat: incorporate upstream array_slice fixes

Fixes #670

* update ExectionPlan::children impl for DatasetExec

Ref apache/datafusion#10543

* update value_interval_daytime

Ref apache/arrow-rs#5769

* update regexp_replace and regexp_match

Fixes #677

* add gil-refs feature to pyo3

This silences pyo3's deprecation warnings for its new Bounds api.

It's the 1st step of the migration, and should be removed before merge.

Ref https://pyo3.rs/v0.21.0/migration#from-020-to-021

* fix signature for octet_length

Ref apache/datafusion#10726

* update signature for covar_samp

AggregateUDF expressions now have a builder API design, which removes arguments like filter and order_by

Ref apache/datafusion#10545
Ref apache/datafusion#10492

* convert covar_pop to expr_fn api

Ref: https://github.com/apache/datafusion/pull/10418/files

* convert median to expr_fn api

Ref apache/datafusion#10644

* convert variance sample to UDF

Ref apache/datafusion#10667

* convert first_value and last_value to UDFs

Ref apache/datafusion#10648

* checkpointing with a few todos to fix remaining compile errors

* impl PyExpr::python_value for IntervalDayTime and IntervalMonthDayNano

* convert sum aggregate function to UDF

* remove unnecessary clone on double reference

* apply cargo fmt

* remove duplicate allow-dead-code annotation

* update tpch examples for new pyarrow interval

Fixes #665

* marked q11 tpch example as expected fail

Ref #730

* add default stride of None back to array_slice
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jayzhan211 jayzhan211

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Introduce expr builder for aggregate function jayzhan211/datafusion
2 participants
@alamb @jayzhan211