Upgrade Datafusion to v37.1.0 #669
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The method was removed upstream but is used in many tests for `datafusion-python`. Ref: apache/datafusion#9627
The options to write_parquet changed. write_json has a new argument that I defaulted to None. We can expose that config later. Ref: apache/datafusion#9382
- `WindowFunction` and `AggregateFunction` have `null_treatment` options. - `ScalarValue` and `DataType` have new variants - `SchemaProvider::table` now returns a `Result`
`datafusion` completed an Epic that ported many of the `BuiltInFunctions` enum to `SclarUDF`. I created new macros to simplify the port, and used these macros to refactor a few existing functions. Ref: apache/datafusion#9285
|
One follow-on that I'd like to do is parametrize I didn't want to alter the test-cases unnecessarily while doing the upgrade. |
This is a bug upstream in datafusion FAILED datafusion/tests/test_functions.py::test_array_functions - pyo3_runtime.PanicException: range end index 9 out of range for slice of length 8
This was referenced May 8, 2024
andygrove
approved these changes
May 8, 2024
There was a problem hiding this comment.
LGTM. Thank you for taking this on @Michael-J-Ward!
The conda failures are happening in main too, so will ignore those for now.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
NOTE: While each commit is a self-contained "logical" change, the project needs all the commits to compile.
Which issue does this PR close?
Closes #663. (hopefully)
This is a cleaned up version of #662.
Importantly, the final failing test is not commented out, so there is 1 failing test.
The error and test case are highlighted in this commit from previous PR: ecff357
What changes are included in this PR?
Are there any user-facing changes?
Much of
src/functions.rshave been updated per the upstream epic: apache/datafusion#9285