[SPARK-33302][SQL] Push down filters through Expand #30278

AngersZhuuuu · 2020-11-06T09:33:51Z

What changes were proposed in this pull request?

Push down filter through expand. For case below:

create table t1(pid int, uid int, sid int, dt date, suid int) using parquet;
create table t2(pid int, vs int, uid int, csid int) using parquet;

SELECT
       years,
       appversion,                                               
       SUM(uusers) AS users                                      
FROM   (SELECT
               Date_trunc('year', dt)          AS years,
               CASE                                              
                 WHEN h.pid = 3 THEN 'iOS'           
                 WHEN h.pid = 4 THEN 'Android'       
                 ELSE 'Other'                                    
               END                             AS viewport,      
               h.vs                            AS appversion,
               Count(DISTINCT u.uid)           AS uusers
               ,Count(DISTINCT u.suid)         AS srcusers
        FROM   t1 u                                   
               join t2 h                              
                 ON h.uid = u.uid            
        GROUP  BY 1,                                             
                  2,                                             
                  3) AS a
WHERE  viewport = 'iOS'                                          
GROUP  BY 1,                                                     
          2

Plan. before this pr:

== Physical Plan ==
*(5) HashAggregate(keys=[years#30, appversion#32], functions=[sum(uusers#33L)])
+- Exchange hashpartitioning(years#30, appversion#32, 200), true, [id=#251]
   +- *(4) HashAggregate(keys=[years#30, appversion#32], functions=[partial_sum(uusers#33L)])
      +- *(4) HashAggregate(keys=[date_trunc('year', CAST(u.`dt` AS TIMESTAMP))#45, CASE WHEN (h.`pid` = 3) THEN 'iOS' WHEN (h.`pid` = 4) THEN 'Android' ELSE 'Other' END#46, vs#12], functions=[count(if ((gid#44 = 1)) u.`uid`#47 else null)])
         +- Exchange hashpartitioning(date_trunc('year', CAST(u.`dt` AS TIMESTAMP))#45, CASE WHEN (h.`pid` = 3) THEN 'iOS' WHEN (h.`pid` = 4) THEN 'Android' ELSE 'Other' END#46, vs#12, 200), true, [id=#246]
            +- *(3) HashAggregate(keys=[date_trunc('year', CAST(u.`dt` AS TIMESTAMP))#45, CASE WHEN (h.`pid` = 3) THEN 'iOS' WHEN (h.`pid` = 4) THEN 'Android' ELSE 'Other' END#46, vs#12], functions=[partial_count(if ((gid#44 = 1)) u.`uid`#47 else null)])
               +- *(3) HashAggregate(keys=[date_trunc('year', CAST(u.`dt` AS TIMESTAMP))#45, CASE WHEN (h.`pid` = 3) THEN 'iOS' WHEN (h.`pid` = 4) THEN 'Android' ELSE 'Other' END#46, vs#12, u.`uid`#47, u.`suid`#48, gid#44], functions=[])
                  +- Exchange hashpartitioning(date_trunc('year', CAST(u.`dt` AS TIMESTAMP))#45, CASE WHEN (h.`pid` = 3) THEN 'iOS' WHEN (h.`pid` = 4) THEN 'Android' ELSE 'Other' END#46, vs#12, u.`uid`#47, u.`suid`#48, gid#44, 200), true, [id=#241]
                     +- *(2) HashAggregate(keys=[date_trunc('year', CAST(u.`dt` AS TIMESTAMP))#45, CASE WHEN (h.`pid` = 3) THEN 'iOS' WHEN (h.`pid` = 4) THEN 'Android' ELSE 'Other' END#46, vs#12, u.`uid`#47, u.`suid`#48, gid#44], functions=[])
                        +- *(2) Filter (CASE WHEN (h.`pid` = 3) THEN 'iOS' WHEN (h.`pid` = 4) THEN 'Android' ELSE 'Other' END#46 = iOS)
                           +- *(2) Expand [ArrayBuffer(date_trunc(year, cast(dt#9 as timestamp), Some(Etc/GMT+7)), CASE WHEN (pid#11 = 3) THEN iOS WHEN (pid#11 = 4) THEN Android ELSE Other END, vs#12, uid#7, null, 1), ArrayBuffer(date_trunc(year, cast(dt#9 as timestamp), Some(Etc/GMT+7)), CASE WHEN (pid#11 = 3) THEN iOS WHEN (pid#11 = 4) THEN Android ELSE Other END, vs#12, null, suid#10, 2)], [date_trunc('year', CAST(u.`dt` AS TIMESTAMP))#45, CASE WHEN (h.`pid` = 3) THEN 'iOS' WHEN (h.`pid` = 4) THEN 'Android' ELSE 'Other' END#46, vs#12, u.`uid`#47, u.`suid`#48, gid#44]
                              +- *(2) Project [uid#7, dt#9, suid#10, pid#11, vs#12]
                                 +- *(2) BroadcastHashJoin [uid#7], [uid#13], Inner, BuildRight
                                    :- *(2) Project [uid#7, dt#9, suid#10]
                                    :  +- *(2) Filter isnotnull(uid#7)
                                    :     +- *(2) ColumnarToRow
                                    :        +- FileScan parquet default.t1[uid#7,dt#9,suid#10] Batched: true, DataFilters: [isnotnull(uid#7)], Format: Parquet, Location: InMemoryFileIndex[file:/root/spark-3.0.0-bin-hadoop3.2/spark-warehouse/t1], PartitionFilters: [], PushedFilters: [IsNotNull(uid)], ReadSchema: struct<uid:int,dt:date,suid:int>
                                    +- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[2, int, true] as bigint))), [id=#233]
                                       +- *(1) Project [pid#11, vs#12, uid#13]
                                          +- *(1) Filter isnotnull(uid#13)
                                             +- *(1) ColumnarToRow
                                                +- FileScan parquet default.t2[pid#11,vs#12,uid#13] Batched: true, DataFilters: [isnotnull(uid#13)], Format: Parquet, Location: InMemoryFileIndex[file:/root/spark-3.0.0-bin-hadoop3.2/spark-warehouse/t2], PartitionFilters: [], PushedFilters: [IsNotNull(uid)], ReadSchema: struct<pid:int,vs:int,uid:int>

Plan. after. this pr. :

== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=false
+- HashAggregate(keys=[years#0, appversion#2], functions=[sum(uusers#3L)], output=[years#0, appversion#2, users#5L])
   +- Exchange hashpartitioning(years#0, appversion#2, 5), true, [id=#71]
      +- HashAggregate(keys=[years#0, appversion#2], functions=[partial_sum(uusers#3L)], output=[years#0, appversion#2, sum#22L])
         +- HashAggregate(keys=[date_trunc(year, cast(dt#9 as timestamp), Some(America/Los_Angeles))#23, CASE WHEN (pid#11 = 3) THEN iOS WHEN (pid#11 = 4) THEN Android ELSE Other END#24, vs#12], functions=[count(distinct uid#7)], output=[years#0, appversion#2, uusers#3L])
            +- Exchange hashpartitioning(date_trunc(year, cast(dt#9 as timestamp), Some(America/Los_Angeles))#23, CASE WHEN (pid#11 = 3) THEN iOS WHEN (pid#11 = 4) THEN Android ELSE Other END#24, vs#12, 5), true, [id=#67]
               +- HashAggregate(keys=[date_trunc(year, cast(dt#9 as timestamp), Some(America/Los_Angeles))#23, CASE WHEN (pid#11 = 3) THEN iOS WHEN (pid#11 = 4) THEN Android ELSE Other END#24, vs#12], functions=[partial_count(distinct uid#7)], output=[date_trunc(year, cast(dt#9 as timestamp), Some(America/Los_Angeles))#23, CASE WHEN (pid#11 = 3) THEN iOS WHEN (pid#11 = 4) THEN Android ELSE Other END#24, vs#12, count#27L])
                  +- HashAggregate(keys=[date_trunc(year, cast(dt#9 as timestamp), Some(America/Los_Angeles))#23, CASE WHEN (pid#11 = 3) THEN iOS WHEN (pid#11 = 4) THEN Android ELSE Other END#24, vs#12, uid#7], functions=[], output=[date_trunc(year, cast(dt#9 as timestamp), Some(America/Los_Angeles))#23, CASE WHEN (pid#11 = 3) THEN iOS WHEN (pid#11 = 4) THEN Android ELSE Other END#24, vs#12, uid#7])
                     +- Exchange hashpartitioning(date_trunc(year, cast(dt#9 as timestamp), Some(America/Los_Angeles))#23, CASE WHEN (pid#11 = 3) THEN iOS WHEN (pid#11 = 4) THEN Android ELSE Other END#24, vs#12, uid#7, 5), true, [id=#63]
                        +- HashAggregate(keys=[date_trunc(year, cast(dt#9 as timestamp), Some(America/Los_Angeles)) AS date_trunc(year, cast(dt#9 as timestamp), Some(America/Los_Angeles))#23, CASE WHEN (pid#11 = 3) THEN iOS WHEN (pid#11 = 4) THEN Android ELSE Other END AS CASE WHEN (pid#11 = 3) THEN iOS WHEN (pid#11 = 4) THEN Android ELSE Other END#24, vs#12, uid#7], functions=[], output=[date_trunc(year, cast(dt#9 as timestamp), Some(America/Los_Angeles))#23, CASE WHEN (pid#11 = 3) THEN iOS WHEN (pid#11 = 4) THEN Android ELSE Other END#24, vs#12, uid#7])
                           +- Project [uid#7, dt#9, pid#11, vs#12]
                              +- BroadcastHashJoin [uid#7], [uid#13], Inner, BuildRight, false
                                 :- Filter isnotnull(uid#7)
                                 :  +- FileScan parquet default.t1[uid#7,dt#9] Batched: true, DataFilters: [isnotnull(uid#7)], Format: Parquet, Location: InMemoryFileIndex[file:/private/var/folders/4l/7_c5c97s1_gb0d9_d6shygx00000gn/T/warehouse-c069d87..., PartitionFilters: [], PushedFilters: [IsNotNull(uid)], ReadSchema: struct<uid:int,dt:date>
                                 +- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[2, int, false] as bigint)),false), [id=#58]
                                    +- Filter ((CASE WHEN (pid#11 = 3) THEN iOS WHEN (pid#11 = 4) THEN Android ELSE Other END = iOS) AND isnotnull(uid#13))
                                       +- FileScan parquet default.t2[pid#11,vs#12,uid#13] Batched: true, DataFilters: [(CASE WHEN (pid#11 = 3) THEN iOS WHEN (pid#11 = 4) THEN Android ELSE Other END = iOS), isnotnull..., Format: Parquet, Location: InMemoryFileIndex[file:/private/var/folders/4l/7_c5c97s1_gb0d9_d6shygx00000gn/T/warehouse-c069d87..., PartitionFilters: [], PushedFilters: [IsNotNull(uid)], ReadSchema: struct<pid:int,vs:int,uid:int>

Why are the changes needed?

Improve performance, filter more data.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Added UT

SparkQA · 2020-11-06T11:12:59Z

Test build #130714 has finished for PR 30278 at commit c3ec848.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-11-06T11:24:07Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35324/

AngersZhuuuu · 2020-11-06T11:26:40Z

retest this please

SparkQA · 2020-11-06T11:47:06Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35324/

SparkQA · 2020-11-06T12:54:14Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35332/

SparkQA · 2020-11-06T13:23:51Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35332/

SparkQA · 2020-11-06T16:07:51Z

Test build #130723 has finished for PR 30278 at commit c3ec848.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AngersZhuuuu · 2020-11-08T13:29:13Z

gentle ping @maropu @cloud-fan

maropu · 2020-11-09T00:50:17Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala

@@ -1269,6 +1269,7 @@ object PushPredicateThroughNonJoin extends Rule[LogicalPlan] with PredicateHelpe
    case _: Sort => true
    case _: BatchEvalPython => true
    case _: ArrowEvalPython => true
+    case _: Expand => true


This change affects the PushDownLeftSemiAntiJoin rule, too. So, could you add tests for the case?

This change affects the PushDownLeftSemiAntiJoin rule, too. So, could you add tests for the case?

Double check the case, seems current master fix this case by some pr, but 3.0 is still as jira desc.

seems current master fix this case

What do you mean by "fix this case"?

seems current master fix this case

What do you mean by "fix this case"?

I have found the pr #29673
Before this pr, SQL

SELECT years, appversion, SUM(uusers) AS users FROM (SELECT Date_trunc('year', dt) AS years, CASE WHEN h.pid = 3 THEN 'iOS' WHEN h.pid = 4 THEN 'Android' ELSE 'Other' END AS viewport, h.vs AS appversion, Count(DISTINCT u.uid) AS uusers ,Count(DISTINCT u.suid) AS srcusers FROM t1 u join t2 h ON h.uid = u.uid GROUP BY 1, 2, 3) AS a WHERE viewport = 'iOS' GROUP BY 1, 2

Optimized plan is

== Optimized Logical Plan == Aggregate [years#0, appversion#2], [years#0, appversion#2, sum(uusers#3L) AS users#5L] +- Aggregate [date_trunc('year', CAST(u.`dt` AS TIMESTAMP))#24, CASE WHEN (h.`pid` = 3) THEN 'iOS' WHEN (h.`pid` = 4) THEN 'Android' ELSE 'Other' END#25, vs#17], [date_trunc('year', CAST(u.`dt` AS TIMESTAMP))#24 AS years#0, vs#17 AS appversion#2, count(if ((gid#23 = 1)) u.`uid`#26 else null) AS uusers#3L] +- Aggregate [date_trunc('year', CAST(u.`dt` AS TIMESTAMP))#24, CASE WHEN (h.`pid` = 3) THEN 'iOS' WHEN (h.`pid` = 4) THEN 'Android' ELSE 'Other' END#25, vs#17, u.`uid`#26, u.`suid`#27, gid#23], [date_trunc('year', CAST(u.`dt` AS TIMESTAMP))#24, CASE WHEN (h.`pid` = 3) THEN 'iOS' WHEN (h.`pid` = 4) THEN 'Android' ELSE 'Other' END#25, vs#17, u.`uid`#26, gid#23] +- Filter (CASE WHEN (h.`pid` = 3) THEN 'iOS' WHEN (h.`pid` = 4) THEN 'Android' ELSE 'Other' END#25 = iOS) +- Expand [ArrayBuffer(date_trunc(year, cast(dt#14 as timestamp), Some(Asia/Shanghai)), CASE WHEN (pid#16 = 3) THEN iOS WHEN (pid#16 = 4) THEN Android ELSE Other END, vs#17, uid#12, null, 1), ArrayBuffer(date_trunc(year, cast(dt#14 as timestamp), Some(Asia/Shanghai)), CASE WHEN (pid#16 = 3) THEN iOS WHEN (pid#16 = 4) THEN Android ELSE Other END, vs#17, null, suid#15, 2)], [date_trunc('year', CAST(u.`dt` AS TIMESTAMP))#24, CASE WHEN (h.`pid` = 3) THEN 'iOS' WHEN (h.`pid` = 4) THEN 'Android' ELSE 'Other' END#25, vs#17, u.`uid`#26, u.`suid`#27, gid#23] +- Project [uid#12, dt#14, suid#15, pid#16, vs#17] +- Join Inner, (uid#18 = uid#12) :- Project [uid#12, dt#14, suid#15] : +- Filter isnotnull(uid#12) : +- Relation[pid#11,uid#12,sid#13,dt#14,suid#15] parquet +- Project [pid#16, vs#17, uid#18] +- Filter isnotnull(uid#18) +- Relation[pid#16,vs#17,uid#18,csid#19] parquet

After that pr, Optimized plan is

== Optimized Logical Plan == Aggregate [years#0, appversion#2], [years#0, appversion#2, sum(uusers#3L) AS users#5L] +- Aggregate [date_trunc(year, cast(dt#14 as timestamp), Some(Asia/Shanghai)), CASE WHEN (pid#16 = 3) THEN iOS WHEN (pid#16 = 4) THEN Android ELSE Other END, vs#17], [date_trunc(year, cast(dt#14 as timestamp), Some(Asia/Shanghai)) AS years#0, vs#17 AS appversion#2, count(distinct uid#12) AS uusers#3L] +- Project [uid#12, dt#14, pid#16, vs#17] +- Join Inner, (uid#18 = uid#12) :- Project [uid#12, dt#14] : +- Filter isnotnull(uid#12) : +- Relation[pid#11,uid#12,sid#13,dt#14,suid#15] parquet +- Project [pid#16, vs#17, uid#18] +- Filter ((CASE WHEN (pid#16 = 3) THEN iOS WHEN (pid#16 = 4) THEN Android ELSE Other END = iOS) AND isnotnull(uid#18)) +- Relation[pid#16,vs#17,uid#18,csid#19] parquet

Filter((CASE WHEN (pid#16 = 3) THEN iOS WHEN (pid#16 = 4) THEN Android ELSE Other END = iOS)) is pushed down and won't generate Expand

how does it related to left semi join?

This change affects the PushDownLeftSemiAntiJoin rule, too. So, could you add tests for the case?

So can we add such a test?

With test case in LeftSemiPushdownSuite

test("Unary: LeftSemi join push down through expand") { val expand = Expand(Seq(Seq('a, 'b, "null"), Seq('a, "null", 'c)), Seq('a, 'b, 'c), testRelation) val originalQuery = expand .join(testRelation1, joinType = LeftSemi, condition = Some('b === 'd && 'b === 1)) val optimized = Optimize.execute(originalQuery.analyze) val correctAnswer = Expand(Seq(Seq('a, 'b, "null"), Seq('a, "null", 'c)), Seq('a, 'b, 'c), Filter(EqualTo('b, 1), testRelation)) .join(testRelation1, joinType = LeftSemi, condition = Some('b === 'd)) .analyze comparePlans(optimized, correctAnswer) }

originalQuery is

'Join LeftSemi, (('b = 'd) AND ('b = 1)) :- 'Expand [List('a, 'b, null), List('a, null, 'c)], ['a, 'b, 'c] : +- LocalRelation <empty>, [a#0, b#1, c#2] +- LocalRelation <empty>, [d#3]

Test result is

== FAIL: Plans do not match === !'Expand [List(a#0, b#0, null), List(a#0, null, c#0)], [a#0, b#0, c#0] 'Join LeftSemi, (b#0 = d#0) !+- 'Join LeftSemi, ((b#0 = 1) AND (b#0 = d#0)) :- Expand [List(a#0, b#0, null), List(a#0, null, c#0)], [a#0, b#0, c#0] ! :- LocalRelation <empty>, [a#0, b#0, c#0] : +- Filter (b#0 = 1) ! +- LocalRelation <empty>, [d#0] : +- LocalRelation <empty>, [a#0, b#0, c#0] ! +- LocalRelation <empty>, [d#0]

Expand will be promoted below Join, so should we ignore this case or add a parameter in canPushThrough like below

def canPushThrough(p: UnaryNode, isFilterPushDown: Boolean = false): Boolean = p match { // Note that some operators (e.g. project, aggregate, union) are being handled separately // (earlier in this rule). case _: AppendColumns => true case _: Distinct => true case _: Generate => true case _: Pivot => true case _: RepartitionByExpression => true case _: Repartition => true case _: ScriptTransformation => true case _: Sort => true case _: BatchEvalPython => true case _: ArrowEvalPython => true case _: Expand => isFilterPushDown case _ => false }

I'm sorry I didn't get it. What's the issue here? We can't pushdown left-semi join through expand?

The optimized (left-side) plan above looks correct to me...

I'm sorry I didn't get it. What's the issue here? We can't pushdown left-semi join through expand?

oh..my mistake, I misunderstood some code about PushDownLeftSemiAntiJoin

The optimized (left-side) plan above looks correct to me...

My fault, I misunderstand some code about PushDownLeftSemiAntiJoin, test case added ==

cloud-fan · 2020-11-09T08:23:28Z

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala

+  test("push down predicate through expand") {
+    val input = LocalRelation('a.int, 'b.string, 'c.double)
+    val query =
+      Aggregate(


why does this test need an Aggregate?

why does this test need an Aggregate?

Not necessary, remove it.

SparkQA · 2020-11-09T13:44:55Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35395/

SparkQA · 2020-11-09T14:06:30Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35395/

SparkQA · 2020-11-09T16:52:39Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35406/

SparkQA · 2020-11-09T17:15:51Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35406/

SparkQA · 2020-11-09T17:30:26Z

Test build #130787 has finished for PR 30278 at commit 803bf0a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-11-09T20:28:42Z

Test build #130797 has finished for PR 30278 at commit 77d6e45.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2020-11-09T23:38:06Z

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala

@@ -1208,6 +1208,30 @@ class FilterPushdownSuite extends PlanTest {
      checkAnalysis = false)
  }

+
+  test("push down predicate through expand") {
+    val input = LocalRelation('a.int, 'b.string, 'c.double)


nit: could you use testRelation instead?

nit: could you use testRelation instead?

Done

maropu · 2020-11-09T23:38:22Z

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala

@@ -1208,6 +1208,30 @@ class FilterPushdownSuite extends PlanTest {
      checkAnalysis = false)
  }

+


nit: unnecessary blank.

nit: unnecessary blank.

Done

maropu · 2020-11-09T23:42:46Z

Push down filter throw expand. For case below:

typo? throw -> through

maropu · 2020-11-09T23:42:59Z

LGTM except for the minor comments.

AngersZhuuuu · 2020-11-09T23:52:03Z

Push down filter throw expand. For case below:

typo? throw -> through

Yea, updated

SparkQA · 2020-11-10T00:41:51Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35418/

SparkQA · 2020-11-10T01:10:16Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35418/

SparkQA · 2020-11-10T04:42:21Z

Test build #130808 has finished for PR 30278 at commit a09f836.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AngersZhuuuu · 2020-11-10T04:45:40Z

retest this please

SparkQA · 2020-11-10T05:34:08Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35441/

SparkQA · 2020-11-10T05:56:23Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35441/

SparkQA · 2020-11-10T08:05:01Z

Test build #130832 has finished for PR 30278 at commit a09f836.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2020-11-10T08:12:30Z

retest this please

SparkQA · 2020-11-10T09:29:53Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35458/

SparkQA · 2020-11-10T09:51:09Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35458/

SparkQA · 2020-11-10T13:56:31Z

Test build #130850 has finished for PR 30278 at commit a09f836.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2020-11-10T14:41:00Z

thanks, merging to master!

[SPARK-33302][SQL] Failed to push down filters through Expand

c3ec848

github-actions bot added the SQL label Nov 6, 2020

maropu reviewed Nov 9, 2020

View reviewed changes

cloud-fan reviewed Nov 9, 2020

View reviewed changes

Update FilterPushdownSuite.scala

803bf0a

Update LeftSemiAntiJoinPushDownSuite.scala

77d6e45

cloud-fan approved these changes Nov 9, 2020

View reviewed changes

maropu reviewed Nov 9, 2020

View reviewed changes

Update FilterPushdownSuite.scala

a09f836

maropu approved these changes Nov 9, 2020

View reviewed changes

cloud-fan closed this in 34f5e7c Nov 10, 2020

		@@ -1208,6 +1208,30 @@ class FilterPushdownSuite extends PlanTest {
		checkAnalysis = false)
		}

[SPARK-33302][SQL] Push down filters through Expand #30278

[SPARK-33302][SQL] Push down filters through Expand #30278

Conversation

AngersZhuuuu commented Nov 6, 2020 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

SparkQA commented Nov 6, 2020

SparkQA commented Nov 6, 2020

AngersZhuuuu commented Nov 6, 2020

SparkQA commented Nov 6, 2020

SparkQA commented Nov 6, 2020

SparkQA commented Nov 6, 2020

SparkQA commented Nov 6, 2020

AngersZhuuuu commented Nov 8, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Nov 9, 2020

SparkQA commented Nov 9, 2020

SparkQA commented Nov 9, 2020

SparkQA commented Nov 9, 2020

SparkQA commented Nov 9, 2020

SparkQA commented Nov 9, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maropu commented Nov 9, 2020

maropu commented Nov 9, 2020

AngersZhuuuu commented Nov 9, 2020

SparkQA commented Nov 10, 2020

SparkQA commented Nov 10, 2020

SparkQA commented Nov 10, 2020

AngersZhuuuu commented Nov 10, 2020

SparkQA commented Nov 10, 2020

SparkQA commented Nov 10, 2020

SparkQA commented Nov 10, 2020

maropu commented Nov 10, 2020

SparkQA commented Nov 10, 2020

SparkQA commented Nov 10, 2020

SparkQA commented Nov 10, 2020

cloud-fan commented Nov 10, 2020

AngersZhuuuu commented Nov 6, 2020 •

edited

Loading