[SPARK-14830][SQL] Add RemoveRepetitionFromGroupExpressions optimizer. #12590

dongjoon-hyun · 2016-04-22T00:11:28Z

What changes were proposed in this pull request?

This PR aims to optimize GroupExpressions by removing repeating expressions. RemoveRepetitionFromGroupExpressions is added.

Before

scala> sql("select a+1 from values 1,2 T(a) group by a+1, 1+a, A+1, 1+A").explain()
== Physical Plan ==
WholeStageCodegen
:  +- TungstenAggregate(key=[(a#0 + 1)#6,(1 + a#0)#7,(A#0 + 1)#8,(1 + A#0)#9], functions=[], output=[(a + 1)#5])
:     +- INPUT
+- Exchange hashpartitioning((a#0 + 1)#6, (1 + a#0)#7, (A#0 + 1)#8, (1 + A#0)#9, 200), None
   +- WholeStageCodegen
      :  +- TungstenAggregate(key=[(a#0 + 1) AS (a#0 + 1)#6,(1 + a#0) AS (1 + a#0)#7,(A#0 + 1) AS (A#0 + 1)#8,(1 + A#0) AS (1 + A#0)#9], functions=[], output=[(a#0 + 1)#6,(1 + a#0)#7,(A#0 + 1)#8,(1 + A#0)#9])
      :     +- INPUT
      +- LocalTableScan [a#0], [[1],[2]]

After

scala> sql("select a+1 from values 1,2 T(a) group by a+1, 1+a, A+1, 1+A").explain()
== Physical Plan ==
WholeStageCodegen
:  +- TungstenAggregate(key=[(a#0 + 1)#6], functions=[], output=[(a + 1)#5])
:     +- INPUT
+- Exchange hashpartitioning((a#0 + 1)#6, 200), None
   +- WholeStageCodegen
      :  +- TungstenAggregate(key=[(a#0 + 1) AS (a#0 + 1)#6], functions=[], output=[(a#0 + 1)#6])
      :     +- INPUT
      +- LocalTableScan [a#0], [[1],[2]]

How was this patch tested?

Pass the Jenkins tests (with a new testcase)

SparkQA · 2016-04-22T01:42:54Z

Test build #56613 has finished for PR 12590 at commit 75b8f73.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2016-04-23T03:45:20Z

Rebased.

SparkQA · 2016-04-23T05:12:32Z

Test build #56772 has finished for PR 12590 at commit bda4ae6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2016-04-24T20:13:55Z

Hi, @rxin .
Could you review this PR about RemoveRepetitionFromGroupExpressions?

dongjoon-hyun · 2016-04-25T09:02:40Z

Rebased.

SparkQA · 2016-04-25T10:29:02Z

Test build #56891 has finished for PR 12590 at commit 78a42c5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2016-04-25T17:25:21Z

Hi, @rxin .
Could you review this PR please?

SparkQA · 2016-04-26T06:32:21Z

Test build #56970 has finished for PR 12590 at commit 01ce1a4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2016-04-26T18:05:10Z

Hi, @marmbrus .
Could you review this PR when you have some time?

SparkQA · 2016-04-28T21:44:02Z

Test build #57274 has finished for PR 12590 at commit e3bdc16.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-04-30T07:13:18Z

Test build #57411 has finished for PR 12590 at commit d16ebc2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

marmbrus · 2016-05-02T17:24:00Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala

+object RemoveRepetitionFromGroupExpressions extends Rule[LogicalPlan] {
+  def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case a @ Aggregate(grouping, _, _) =>
+      val newGrouping = grouping.distinct


This is going to miss cases like GROUP BY A, a. I think you want to use an ExpressionSet instead of normal .distinct

Oh, thank you for review. @marmbrus .
I'll fix that.

SparkQA · 2016-05-02T19:31:47Z

Test build #57540 has finished for PR 12590 at commit 2198f0f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2016-05-02T19:38:03Z

@marmbrus . Now, it's ready for review again.
This PR becomes to do much better than what I expected.
Thank you so much.

marmbrus · 2016-05-02T19:39:50Z

Thanks, merging to master and 2.0

## What changes were proposed in this pull request? This PR aims to optimize GroupExpressions by removing repeating expressions. `RemoveRepetitionFromGroupExpressions` is added. **Before** ```scala scala> sql("select a+1 from values 1,2 T(a) group by a+1, 1+a, A+1, 1+A").explain() == Physical Plan == WholeStageCodegen : +- TungstenAggregate(key=[(a#0 + 1)#6,(1 + a#0)#7,(A#0 + 1)#8,(1 + A#0)#9], functions=[], output=[(a + 1)#5]) : +- INPUT +- Exchange hashpartitioning((a#0 + 1)#6, (1 + a#0)#7, (A#0 + 1)#8, (1 + A#0)#9, 200), None +- WholeStageCodegen : +- TungstenAggregate(key=[(a#0 + 1) AS (a#0 + 1)#6,(1 + a#0) AS (1 + a#0)#7,(A#0 + 1) AS (A#0 + 1)#8,(1 + A#0) AS (1 + A#0)#9], functions=[], output=[(a#0 + 1)#6,(1 + a#0)#7,(A#0 + 1)#8,(1 + A#0)#9]) : +- INPUT +- LocalTableScan [a#0], [[1],[2]] ``` **After** ```scala scala> sql("select a+1 from values 1,2 T(a) group by a+1, 1+a, A+1, 1+A").explain() == Physical Plan == WholeStageCodegen : +- TungstenAggregate(key=[(a#0 + 1)#6], functions=[], output=[(a + 1)#5]) : +- INPUT +- Exchange hashpartitioning((a#0 + 1)#6, 200), None +- WholeStageCodegen : +- TungstenAggregate(key=[(a#0 + 1) AS (a#0 + 1)#6], functions=[], output=[(a#0 + 1)#6]) : +- INPUT +- LocalTableScan [a#0], [[1],[2]] ``` ## How was this patch tested? Pass the Jenkins tests (with a new testcase) Author: Dongjoon Hyun <[email protected]> Closes #12590 from dongjoon-hyun/SPARK-14830. (cherry picked from commit 6e63201) Signed-off-by: Michael Armbrust <[email protected]>

dongjoon-hyun · 2016-05-02T19:44:54Z

Thank you, @marmbrus !

marmbrus reviewed May 2, 2016
View reviewed changes

dongjoon-hyun added 3 commits May 2, 2016 11:05

[SPARK-14830][SQL] Add RemoveRepetitionFromGroupExpressions optimizer.

750fe74

update testcases.

f72fd67

Use ExpressionSet.

2198f0f

asfgit closed this in 6e63201 May 2, 2016

dongjoon-hyun deleted the SPARK-14830 branch May 12, 2016 01:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-14830][SQL] Add RemoveRepetitionFromGroupExpressions optimizer. #12590

[SPARK-14830][SQL] Add RemoveRepetitionFromGroupExpressions optimizer. #12590

dongjoon-hyun commented Apr 22, 2016 •

edited

Loading

SparkQA commented Apr 22, 2016

dongjoon-hyun commented Apr 23, 2016

SparkQA commented Apr 23, 2016

dongjoon-hyun commented Apr 24, 2016

dongjoon-hyun commented Apr 25, 2016

SparkQA commented Apr 25, 2016

dongjoon-hyun commented Apr 25, 2016

SparkQA commented Apr 26, 2016

dongjoon-hyun commented Apr 26, 2016

SparkQA commented Apr 28, 2016

SparkQA commented Apr 30, 2016

marmbrus May 2, 2016

dongjoon-hyun May 2, 2016

SparkQA commented May 2, 2016

dongjoon-hyun commented May 2, 2016

marmbrus commented May 2, 2016

dongjoon-hyun commented May 2, 2016

[SPARK-14830][SQL] Add RemoveRepetitionFromGroupExpressions optimizer. #12590

[SPARK-14830][SQL] Add RemoveRepetitionFromGroupExpressions optimizer. #12590

Conversation

dongjoon-hyun commented Apr 22, 2016 • edited Loading

What changes were proposed in this pull request?

How was this patch tested?

SparkQA commented Apr 22, 2016

dongjoon-hyun commented Apr 23, 2016

SparkQA commented Apr 23, 2016

dongjoon-hyun commented Apr 24, 2016

dongjoon-hyun commented Apr 25, 2016

SparkQA commented Apr 25, 2016

dongjoon-hyun commented Apr 25, 2016

SparkQA commented Apr 26, 2016

dongjoon-hyun commented Apr 26, 2016

SparkQA commented Apr 28, 2016

SparkQA commented Apr 30, 2016

marmbrus May 2, 2016

Choose a reason for hiding this comment

dongjoon-hyun May 2, 2016

Choose a reason for hiding this comment

SparkQA commented May 2, 2016

dongjoon-hyun commented May 2, 2016

marmbrus commented May 2, 2016

dongjoon-hyun commented May 2, 2016

dongjoon-hyun commented Apr 22, 2016 •

edited

Loading