Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: not do boolean folding on NULL and/or expr #1245

Merged
merged 2 commits into from
Nov 5, 2021

Conversation

NGA-TRAN
Copy link
Contributor

@NGA-TRAN NGA-TRAN commented Nov 5, 2021

Which issue does this PR close?

Closes #1244

Rationale for this change

Do not fold Boolean(NULL) and/or expr because it depends on the value of the expr

What changes are included in this PR?

Boolean(NULL) and/or expr is key as-is (aka not rewrite)

Are there any user-facing changes?

This bug was hit by IOx's storage_api::test

@github-actions github-actions bot added the datafusion Changes in the datafusion crate label Nov 5, 2021
@NGA-TRAN
Copy link
Contributor Author

NGA-TRAN commented Nov 5, 2021

@alamb and @matthewmturner Can you help review this?

@xudong963
Copy link
Member

Thanks for your nice work 🎉@NGA-TRAN

FYI

case1
postgres=# select 11 BETWEEN 0 AND 10 and NULL as c;
 c
---
 f
(1 row)

case2
postgres=# select 1 BETWEEN 0 AND 10 and NULL as c;
 c
---

(1 row)

case3
postgres=# select 1 BETWEEN 0 AND 10 or NULL as c;
 c
---
 t
(1 row)

case4
postgres=# select 11 BETWEEN 0 AND 10 or NULL as c;
 c
---

(1 row)

So I think we can add two tests about case1 and case4. @NGA-TRAN

Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @NGA-TRAN -- I find the new structure and comments much easier to follow.

datafusion/src/optimizer/constant_folding.rs Outdated Show resolved Hide resolved
// FALSE and expr (including NULL) = FALSE
Some(false) => Expr::Literal(ScalarValue::Boolean(Some(false))),
None => match *bool_expr {
// NULL and TRUE = NULL
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW I am working on getting these constant cases to work with the ConstEvaluator (so they wouldn't be strictly necessary) but it turns out these types of expressions don't actually work in the general evaluator

I am working on it in #1163

But in any event I think this code is good 👍

@NGA-TRAN
Copy link
Contributor Author

NGA-TRAN commented Nov 5, 2021

@xudong963 Those are great queries. I have tried to add them in the test but I think we have not done constant folding for Between yet (or at least not for the queries in your examples) so they are not folded at this time. Hence, I do not commit them in because they are the same as the current between query.

I can open a new ticket for improving these cases though. What do you think @alamb ?

@NGA-TRAN
Copy link
Contributor Author

NGA-TRAN commented Nov 5, 2021

@alamb I have addressed the comments. It should be ready for merge when all checks pass

@alamb
Copy link
Contributor

alamb commented Nov 5, 2021

@xudong963 Those are great queries. I have tried to add them in the test but I think we have not done constant folding for Between yet (or at least not for the queries in your examples) so they are not folded at this time. Hence, I do not commit them in because they are the same as the current between query.

I can open a new ticket for improving these cases though. What do you think @alamb ?

I think the constant folding is handled by ConstEvaluator (in a different module). The Simplifier is supposed to be focused on items that can't be folded (like TRUE OR <expr> --> <expr>).

The fact we have some boolean constant evaluation in here is a historical artifact and due to the fact we don't have full expression evaluation support for all boolean operations (yet).

Thus I don't think a separate ticket is needed

Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @NGA-TRAN

@alamb alamb merged commit bae8562 into apache:master Nov 5, 2021
@houqp houqp added the bug Something isn't working label Nov 6, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working datafusion Changes in the datafusion crate
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Should not fold a non-constant expression when it is not clear to do so
4 participants