Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: incorrect set preserve_partitioning in SortExec #8485

Merged
merged 2 commits into from
Dec 12, 2023

Conversation

haohuaijin
Copy link
Contributor

@haohuaijin haohuaijin commented Dec 10, 2023

Which issue does this PR close?

related to #8374

Rationale for this change

In #8374, we find ProjectionPushdown will incorrectly push projection down and rewrite order by column,
before

select t1.* from t1 join t2 on t1.a = t2.a order by t1.a, t2.b;

after

select t1.* from t1 join t2 on t1.a = t2.a order by t1.a, t1.b;

the sort column change from t1.a, t2.b to t1.a, t1.b.

However, even if we find the root case, we can still not explain why setting the order by column incorrectly results in an empty output. After doing some tests, I discovered that we did not set the preserve_partitioning in SortExec. As a result, SortExec only run one partition(instead of all partition), leading to an empty output.

What changes are included in this PR?

make SortExec's preserve_patitioning consistent before and after projection_pushdown

Are these changes tested?

yes, add .slt test

Are there any user-facing changes?

@github-actions github-actions bot added core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) labels Dec 10, 2023
Copy link
Member

@waynexia waynexia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me 👍

datafusion/sqllogictest/test_files/join.slt Show resolved Hide resolved
Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @haohuaijin and @waynexia

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core Core DataFusion crate sqllogictest SQL Logic Tests (.slt)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants