Support DataFusion 15

[andrei-ionescu]

Description

Add support for DataFusion 15.

Upgrades dependencies:

• DataFusion from 14 to 15
• Arrow from 26 to 28

Related Issue(s)

• closes Support Apache Arrow DataFusion 15 #1020

Documentation

[roeap]

Hi @andrei-ionescu thanks for the contribution, in this case unfortunately there already is a PR open for this :) #1014

[rtyler]

@roeap since this pull request is smaller and more isolated from other changes, is there a problem with working to bring this in? That would allow us to evaluate the non-string partition changes more independent of the datafusion changes

[roeap]

@rtyler - certainly, whatever gets us there quickest :).

@andrei-ionescu - In order to make the tests pass, we need to update the schema validation in the operations/writer file as well. you can look at #1014 to see one option to do this.

[andrei-ionescu]

@roeap, @rtyler, I implemented the suggested feedback. Thanks for giving the me hint about #1014 PR.

All checks did pass except the "Python 3.10 PyArrow latest" step. It seems that it did timeout. Do you know the root cause of this? Is it transient?

[roeap]

Thanks @andrei-ionescu, excellent work!. The python 3.10 test is in fact flaky and not caused by this PR.

One last ask, there is a test for the new type handling in datafusion 15 in the other PR. As we do the migration here, could you move the test as well? Otherwise looks great and we should be able to merge before #971 happens.

[andrei-ionescu]

@roeap Thanks for the suggestion with the test. Added it into here.

[roeap]

@andrei-ionescu thanks for adding the test - looks great! - It seems thought that a single dev artifact slipped into the code. There is a new parquet file in a new data/ folder within in root of the repo, that i cannot place.

[andrei-ionescu]

@roeap: You're right! I may have been running a cargo command on the root of the project that created that I did not noticed it in the PR. Sorry for that! I removed it.

[roeap]

Looking good - waiting for CI to finish and we are ready to merge :)