Update logical-types to main #14202

tobixdev · 2025-01-19T17:46:02Z

Which issue does this PR close?

Updates the logical-types branch to main for #12622.

Rationale for this change

newest
Updates the logical-types branch to main for #12622.

What changes are included in this PR?

Apply Scalar type to new code and resolve issues between the logical-types branch and main (I think they diverged at some point?).

Are these changes tested?

Are there any user-facing changes?

@jayzhan211

* fix: join with sort push down * chore: insert some value * apply suggestion * recover handle_costom_pushdown change * apply suggestion * add more test * add partition

…3688) Co-authored-by: zhangli20 <[email protected]>

* Optimize performance of function Signed-off-by: Tai Le Manh <[email protected]> * Add pre-check array is null * Fix clippy warnings --------- Signed-off-by: Tai Le Manh <[email protected]>

Updates the requirements on [prost-build](https://github.com/tokio-rs/prost) to permit the latest version. - [Release notes](https://github.com/tokio-rs/prost/releases) - [Changelog](https://github.com/tokio-rs/prost/blob/master/CHANGELOG.md) - [Commits](tokio-rs/prost@v0.13.3...v0.13.4) --- updated-dependencies: - dependency-name: prost-build dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Minor: Output elapsed time for sql logic test

…rojectionMapping` and `EquivalenceGroup` (apache#13675) * refactor: replace Vec with IndexMap for expression mappings in ProjectionMapping and EquivalenceGroup * chore * chore: Fix CI * chore: comment * chore: simplify

* fix: Fix parse_sql_expr not handling alias * cargo fmt * fix parse_sql_expr example(remove alias) * add testing * add SUM udaf to TestContextProvider and modify test_sql_to_expr_with_alias for function * revert change on example `parse_sql_expr`

apache#13730) Debug trait is useful for understanding what something is and how it's configured, especially if the implementation is behind dyn trait.

…pache#13731)

…13660) * add `unnest_as_table_factor` and `UnnestRelationBuilder` * unparse unnest as table factor * fix typo * add tests for the default configs * add a static const for unnest_placeholder * fix tests * fix tests

…apache#13727) * Update apache-avro requirement from 0.16 to 0.17 --- updated-dependencies: - dependency-name: apache-avro dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> * Fix compatibility changes schema handling apache-avro 0.17 - Handle ArraySchema struct - Handle MapSchema struct - Map BigDecimal => LargeBinary - Map TimestampNanos => Timestamp(TimeUnit::Nanosecond, None) - Map LocalTimestampNanos => todo!() - Add Default to FixedSchema test * Update Cargo.lock file for apache-avro 0.17 --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Marc Droogh <[email protected]> Co-authored-by: Andrew Lamb <[email protected]>

* Minor: Add doc example to RecordBatchStreamAdapter * Update datafusion/physical-plan/src/stream.rs Co-authored-by: Berkay Şahin <[email protected]> --------- Co-authored-by: Berkay Şahin <[email protected]>

…13581) * Implement GroupsAccumulator for corr(x,y) * feedbacks * fix CI MSRV * review * avoid collect in accumulation * add back cast

* fix union serialisation order in proto * clippy * address comments

…apache#13733) * Minor: make unsupported `nanosecond` part a real (not internal) error * fmt * Improve wording to refer to date part

…nes (apache#13732) * Add tests for date_part on columns + timestamps with / without timezones * Add tests from apache#13372 * remove trailing whitespace

* Optimize performance of initcap (~2x faster) Signed-off-by: Tai Le Manh <[email protected]> * format --------- Signed-off-by: Tai Le Manh <[email protected]>

…pache#13749)

Before the change, the request to use PostgreSQL was simply ignored when `--complete` flag was present.

…pache#13739) * doc-gen: migrate window functions documentation Signed-off-by: zjregee <[email protected]> * fix: update Cargo.lock --------- Signed-off-by: zjregee <[email protected]>

…pache#13751) * Refactor JoinLeftData structure by removing unused memory reservation field in hash join implementation * Add Debug and Clone derives for HashJoinStreamState and ProcessProbeBatchState enums This commit enhances the HashJoinStreamState and ProcessProbeBatchState structures by implementing the Debug and Clone traits, allowing for easier debugging and cloning of these state representations in the hash join implementation.

* Add big decimal formatting test cases with potential trailing zeros * Rename and simplify decimal rendering functions - add `decimal` to function name - drop `precision` parameter as it is not supposed to affect the result * Update to bigdecimal 0.4.7 Utilize new `to_plain_string` function

* CI: Warn on unused crates * CI: Warn on unused crates * CI: Warn on unused crates * CI: Warn on unused crates * CI: Clean up dependencies * CI: Clean up dependencies

* plan implicit lateral if table factor is UNNEST * check for outer references in `create_relation_subquery` * add sqllogictest * fix lateral constant test to not expect a subquery node * replace sqllogictest in favor of logical plan test * update lateral join sqllogictests * add sqllogictests * fix logical plan test

* Minor: improve the Deprecation / API health policy * prettier * Update docs/source/library-user-guide/api-health.md Co-authored-by: Jonah Gao <[email protected]> * Add version guidance and make more copy/paste friendly * prettier * better * rename to guidelines --------- Co-authored-by: Jonah Gao <[email protected]>

* fix: specify roottype in fieldreference Signed-off-by: MBWhite <[email protected]> * Fix formatting Signed-off-by: MBWhite <[email protected]> * review suggestion Signed-off-by: MBWhite <[email protected]> --------- Signed-off-by: MBWhite <[email protected]>

apache#14168) * Add a hint about expected extension in error message in register_csv, register_parquet, register_json, register_avro (apache#14144) * Add tests for error * fix test * fmt * Fix issues causing GitHub checks to fail * revert datafusion-testing change --------- Co-authored-by: Sergey Zhukov <[email protected]> Co-authored-by: Andrew Lamb <[email protected]>

…#14142) * External memory limit validation for sort * add bug tracker * cleanup * Update submodule * reviews * fix CI * move feature to module level

…nction (apache#14183) Co-authored-by: Cheng-Yuan-Lai <a186235@g,ail.com>

jayzhan211 · 2025-01-20T00:55:44Z

I will just merge this, since it is a rebase to logical-types

jayzhan211 · 2025-01-20T01:02:43Z

I compare it with the main, it includes unexpected changes as well

tobixdev · 2025-01-20T06:37:53Z

I'll have a look at it today.

tobixdev · 2025-01-20T12:20:25Z

So something has gone wrong here. For example, the .devcontainer/Dockerfile is equivalent to the one on main but I think that the information that this was a merge commit got lost at some point.

So here is what I did:

Switch to a branch that tracks logical-types
git merge upstream/main
Resolve merge conflicts
Make a commit

Is this the intended workflow? I didn't want to rebase so that I do not alter the existing commits.

I think the best thing is to undo this commit. Is this easily possible as this is just a feature branch? I am now working on a different branch of mine to ensure that the merge commit is properly handled. Here is the diff between said branch and main which only contains the changes we are looking for as far as I know:

https://github.com/apache/datafusion/compare/main...tobixdev:datafusion:logical-types-fix?expand=1

I still have to wait for the checks though and some minor changes may be necessary. After the checks pass, what would be the best way to get this into the feature branch?

Thanks for helping me with this :)

jayzhan211 · 2025-01-21T11:31:33Z

We can revert it, but I think what you did is correct, not sure what happen 🤔 ?

jayzhan211 · 2025-01-21T11:34:38Z

I force push to the previous commit

tobixdev · 2025-01-21T11:46:55Z

Thanks! Should I open a new PR so that we can try it with the new branch?

jayzhan211 · 2025-01-22T00:50:09Z

Thanks! Should I open a new PR so that we can try it with the new branch?

What do you mean new branch

tobixdev · 2025-01-22T07:26:47Z

I am now working on a different branch of mine to ensure that the merge commit is properly handled. Here is the diff between said branch and main which only contains the changes we are looking for as far as I know:
https://github.com/apache/datafusion/compare/main...tobixdev:datafusion:logical-types-fix?expand=1

Basically, if I should create a new PR for this one to logical-types and we hope it works this time.

EDIT: I'll create a new one as I think its more efficient communication wise. You can still decline the PR if we want to handle it otherwise.

akurmustafa and others added 30 commits December 9, 2024 20:05

Minor: Use div_ceil

d3cfc45

Fix hash join with sort push down (apache#13560)

6d7b902

* fix: join with sort push down * chore: insert some value * apply suggestion * recover handle_costom_pushdown change * apply suggestion * add more test * add partition

Improve substr() performance by avoiding using owned string (apache#1…

b73734f

…3688) Co-authored-by: zhangli20 <[email protected]>

reinstate down_cast_any_ref (apache#13705)

6fae5a0

Optimize performance of character_length function (apache#13696)

437cbf8

* Optimize performance of function Signed-off-by: Tai Le Manh <[email protected]> * Add pre-check array is null * Fix clippy warnings --------- Signed-off-by: Tai Le Manh <[email protected]>

Minor: Output elapsed time for sql logic test (apache#13718)

62fc4db

* Minor: Output elapsed time for sql logic test

refactor: simplify the make_udf_function macro (apache#13712)

0f5634e

Improve documentation for TableProvider (apache#13724)

a52eb47

Reveal implementing type and return type in simple UDF implementations (

95405a9

apache#13730) Debug trait is useful for understanding what something is and how it's configured, especially if the implementation is behind dyn trait.

minor: Extract tests for EXTRACT AND date_part to their own file (a…

28e4c64

…pache#13731)

Support unparsing UNNEST plan to UNNEST table factor SQL (apache#…

08119e6

…13660) * add `unnest_as_table_factor` and `UnnestRelationBuilder` * unparse unnest as table factor * fix typo * add tests for the default configs * add a static const for unnest_placeholder * fix tests * fix tests

Minor: Add doc example to RecordBatchStreamAdapter (apache#13725)

44f4be2

* Minor: Add doc example to RecordBatchStreamAdapter * Update datafusion/physical-plan/src/stream.rs Co-authored-by: Berkay Şahin <[email protected]> --------- Co-authored-by: Berkay Şahin <[email protected]>

Implement GroupsAccumulator for corr(x,y) aggregate function (apache#…

36a1361

…13581) * Implement GroupsAccumulator for corr(x,y) * feedbacks * fix CI MSRV * review * avoid collect in accumulation * add back cast

fix union serialisation order in proto (apache#13709)

1c73a23

* fix union serialisation order in proto * clippy * address comments

Minor: make unsupported nanosecond part a real (not internal) error (…

98c483e

…apache#13733) * Minor: make unsupported `nanosecond` part a real (not internal) error * fmt * Improve wording to refer to date part

Add tests for date_part on columns + timestamps with / without timezo…

aeddbd9

…nes (apache#13732) * Add tests for date_part on columns + timestamps with / without timezones * Add tests from apache#13372 * remove trailing whitespace

Optimize performance of initcap function (~2x faster) (apache#13691)

320e4d6

* Optimize performance of initcap (~2x faster) Signed-off-by: Tai Le Manh <[email protected]> * format --------- Signed-off-by: Tai Le Manh <[email protected]>

Minor: Add documentation explaining that initcap oly works for ASCII (a…

b69fa67

…pache#13749)

Support sqllogictest --complete with postgres (apache#13746)

4148729

Before the change, the request to use PostgreSQL was simply ignored when `--complete` flag was present.

doc-gen: migrate window functions documentation to attribute based (a…

6ac1999

…pache#13739) * doc-gen: migrate window functions documentation Signed-off-by: zjregee <[email protected]> * fix: update Cargo.lock --------- Signed-off-by: zjregee <[email protected]>

chore: clean up dependencies (apache#13728)

e8314ab

* CI: Warn on unused crates * CI: Warn on unused crates * CI: Warn on unused crates * CI: Warn on unused crates * CI: Clean up dependencies * CI: Clean up dependencies

cj-zhukov and others added 5 commits January 18, 2025 12:35

Test: Validate memory limit for sort queries to extended test (apache…

0283077

…#14142) * External memory limit validation for sort * add bug tracker * cleanup * Update submodule * reviews * fix CI * move feature to module level

refactor: switch BooleanBufferBuilder to NullBufferBuilder in sort fu…

4e88a1b

…nction (apache#14183) Co-authored-by: Cheng-Yuan-Lai <a186235@g,ail.com>

Add section to howtos.md (apache#14171)

2ad8955

[logical-types] update to main

077ea46

jayzhan211 merged commit d094edc into apache:logical-types Jan 20, 2025
27 checks passed

tobixdev mentioned this pull request Jan 22, 2025

Update Logical Types Branch #14241

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update logical-types to main #14202

Update logical-types to main #14202

tobixdev commented Jan 19, 2025

jayzhan211 commented Jan 20, 2025

jayzhan211 commented Jan 20, 2025

tobixdev commented Jan 20, 2025

tobixdev commented Jan 20, 2025 •

edited

Loading

jayzhan211 commented Jan 21, 2025 •

edited

Loading

jayzhan211 commented Jan 21, 2025

tobixdev commented Jan 21, 2025

jayzhan211 commented Jan 22, 2025

tobixdev commented Jan 22, 2025 •

edited

Loading

Update logical-types to main #14202

Update logical-types to main #14202

Conversation

tobixdev commented Jan 19, 2025

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

jayzhan211 commented Jan 20, 2025

jayzhan211 commented Jan 20, 2025

tobixdev commented Jan 20, 2025

tobixdev commented Jan 20, 2025 • edited Loading

jayzhan211 commented Jan 21, 2025 • edited Loading

jayzhan211 commented Jan 21, 2025

tobixdev commented Jan 21, 2025

jayzhan211 commented Jan 22, 2025

tobixdev commented Jan 22, 2025 • edited Loading

tobixdev commented Jan 20, 2025 •

edited

Loading

jayzhan211 commented Jan 21, 2025 •

edited

Loading

tobixdev commented Jan 22, 2025 •

edited

Loading