Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Upgrade Datafusion to v37.1.0 #669

Merged
merged 9 commits into from
May 8, 2024
Merged

Upgrade Datafusion to v37.1.0 #669

merged 9 commits into from
May 8, 2024

Commits on May 8, 2024

  1. deps: upgrade datafusion to 37.1.0

    @Michael-J-Ward
    Michael-J-Ward committed May 8, 2024
    Configuration menu
    Copy the full SHA
    f93b78f View commit details
    Browse the repository at this point in the history

  2. feat: re-implement SessionContext::tables 

    The method was removed upstream but is used in many tests for `datafusion-python`.

Ref: apache/datafusion#9627
    @Michael-J-Ward
    Michael-J-Ward committed May 8, 2024
    Configuration menu
    Copy the full SHA
    5ec79ac View commit details
    Browse the repository at this point in the history

  3. feat: upgrade dataframe write_parquet and write_json 

    The options to write_parquet changed.

write_json has a new argument that I defaulted to None. We can expose that config later.

Ref: apache/datafusion#9382
    @Michael-J-Ward
    Michael-J-Ward committed May 8, 2024
    Configuration menu
    Copy the full SHA
    7786a71 View commit details
    Browse the repository at this point in the history

  4. feat: impl new ExecutionPlanProperties for DatasetExec 

    Ref: apache/datafusion#9346
    @Michael-J-Ward
    Michael-J-Ward committed May 8, 2024
    Configuration menu
    Copy the full SHA
    5e74b74 View commit details
    Browse the repository at this point in the history

  5. feat: add upstream variant and method params 

    - `WindowFunction` and `AggregateFunction` have `null_treatment` options.
- `ScalarValue` and `DataType` have new variants
- `SchemaProvider::table` now returns a `Result`
    @Michael-J-Ward
    Michael-J-Ward committed May 8, 2024
    Configuration menu
    Copy the full SHA
    2dda4ff View commit details
    Browse the repository at this point in the history

  6. lint: allow(deprecated) for make_scalar_function

    @Michael-J-Ward
    Michael-J-Ward committed May 8, 2024
    Configuration menu
    Copy the full SHA
    8414157 View commit details
    Browse the repository at this point in the history

  7. feat: migrate functions.rs 

    `datafusion` completed an Epic that ported many of the `BuiltInFunctions` enum to `SclarUDF`.

I created new macros to simplify the port, and used these macros to refactor a few existing functions.

Ref: apache/datafusion#9285
    @Michael-J-Ward
    Michael-J-Ward committed May 8, 2024
    Configuration menu
    Copy the full SHA
    7ad526c View commit details
    Browse the repository at this point in the history

  8. fixme: commented out last failing test 

    This is a bug upstream in datafusion

FAILED datafusion/tests/test_functions.py::test_array_functions - pyo3_runtime.PanicException: range end index 9 out of range for slice of length 8
    @Michael-J-Ward
    Michael-J-Ward committed May 8, 2024
    Configuration menu
    Copy the full SHA
    eeccbfd View commit details
    Browse the repository at this point in the history

  9. chore: update Cargo.toml package info

    @Michael-J-Ward
    Michael-J-Ward committed May 8, 2024
    Configuration menu
    Copy the full SHA
    ed8c703 View commit details
    Browse the repository at this point in the history