[SPARK-22895] [SQL] Push down the deterministic predicates that are after the first non-deterministic #20069
Conversation
|
Test build #85355 has finished for PR 20069 at commit
|
|
retest this please |
|
Test build #85356 has finished for PR 20069 at commit
|
| val (pushDown, stayUp) = { | ||
| val pushDownCondition: Expression => Boolean = | ||
| p => p.deterministic && !p.references.contains(watermark.eventTime) | ||
| if (SQLConf.get.outOfOrderPredicateEvaluationEnabled) { |
There was a problem hiding this comment.
Nit: Create a function which accepts parameter Expression => Boolean and predicates: Seq[Expression], and partitionByDeterminism is using the function parameter as e => e. deterministic.
|
Test build #85508 has finished for PR 20069 at commit
|
| @@ -768,6 +768,7 @@ object SQLConf { | |||
| .booleanConf | |||
| .createWithDefault(true) | |||
|
|
|||
|
|
|||
There was a problem hiding this comment.
nit: duplicated empty line?
|
Test build #85550 has finished for PR 20069 at commit
|
|
Thanks! Merged to master. |
| @@ -851,7 +851,7 @@ object PushDownPredicate extends Rule[LogicalPlan] with PredicateHelper { | |||
|
|
|||
| case filter @ Filter(condition, union: Union) => | |||
| // Union could change the rows, so non-deterministic predicate can't be pushed down | |||
| val (pushDown, stayUp) = splitConjunctivePredicates(condition).span(_.deterministic) | |||
| val (pushDown, stayUp) = splitConjunctivePredicates(condition).partition(_.deterministic) | |||
There was a problem hiding this comment.
What does it mean "after the first non-deterministic"? Doesn't this simply partition predicates to deterministic and non-deterministic? Have it considered "first" non-deterministic?
There was a problem hiding this comment.
@viirya
IIUC, with span, the deterministic predicates after the first non-deterministic will not be pushed down, but with partition, all deterministic predicates will be pushed down.
What changes were proposed in this pull request?
Currently, we do not guarantee an order evaluation of conjuncts in either Filter or Join operator. This is also true to the mainstream RDBMS vendors like DB2 and MS SQL Server. Thus, we should also push down the deterministic predicates that are after the first non-deterministic, if possible.
How was this patch tested?
Updated the existing test cases.