[SPARK-20754][SQL] Support TRUNC (number) #18106

wangyum · 2017-05-25T11:01:50Z

What changes were proposed in this pull request?

Move TruncDate() from datetimeExpressions.scala to misc.scala, and add support TRUNC(number), it's similar to Oracle TRUNC(number):

> SELECT TRUNC(1234567891.1234567891, 4);
 1234567891.1234
> SELECT TRUNC(1234567891.1234567891, -4);
 1234560000
> SELECT TRUNC(1234567891.1234567891, 0);
 1234567891
> SELECT TRUNC(1234567891.1234567891);
 1234567891

The MOD and POSITION function alias will be added by follow-up PR.

How was this patch tested?

unit tests

SparkQA · 2017-05-25T12:42:49Z

Test build #77361 has finished for PR 18106 at commit a5ade70.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class Trunc(data: Expression, format: Expression = Literal(0))

SparkQA · 2017-05-25T16:43:07Z

Test build #77370 has finished for PR 18106 at commit c63856b.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-05-26T02:12:42Z

Test build #77394 has finished for PR 18106 at commit e7e6e5b.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-05-26T04:56:49Z

Test build #77399 has finished for PR 18106 at commit 7157820.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

## What changes were proposed in this pull request? apache#18106 Support TRUNC (number), We should also add function alias for `MOD `and `POSITION`. `POSITION(substr IN str) `is a synonym for `LOCATE(substr,str)`. same as MySQL: https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_position ## How was this patch tested? unit tests Author: Yuming Wang <[email protected]> Closes apache#18206 from wangyum/SPARK-20754-mod&position.

Conflicts: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/MiscExpressionsSuite.scala sql/core/src/test/resources/sql-tests/inputs/datetime.sql sql/core/src/test/resources/sql-tests/inputs/operators.sql sql/core/src/test/resources/sql-tests/results/datetime.sql.out sql/core/src/test/resources/sql-tests/results/operators.sql.out

SparkQA · 2017-06-14T17:10:32Z

Test build #78057 has finished for PR 18106 at commit 3d92a48.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class Trunc(data: Expression, format: Expression)

SparkQA · 2017-06-15T04:52:33Z

Test build #78080 has started for PR 18106 at commit b391b6a.

wangyum · 2017-06-15T08:48:16Z

Jenkins, retest this please

SparkQA · 2017-06-15T11:07:07Z

Test build #78094 has finished for PR 18106 at commit b391b6a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2017-06-15T11:30:40Z

cc @gatorsmile

## What changes were proposed in this pull request? apache#18106 Support TRUNC (number), We should also add function alias for `MOD `and `POSITION`. `POSITION(substr IN str) `is a synonym for `LOCATE(substr,str)`. same as MySQL: https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_position ## How was this patch tested? unit tests Author: Yuming Wang <[email protected]> Closes apache#18206 from wangyum/SPARK-20754-mod&position.

SparkQA · 2017-06-20T06:27:38Z

Test build #78273 has started for PR 18106 at commit 5456e61.

wangyum · 2017-06-20T21:59:29Z

Retest this please.

SparkQA · 2017-06-20T22:02:38Z

Test build #78325 has started for PR 18106 at commit 5456e61.

shaneknapp · 2017-06-21T00:38:20Z

test this please

SparkQA · 2017-06-21T00:42:39Z

Test build #78344 has started for PR 18106 at commit 5456e61.

shaneknapp · 2017-06-21T04:21:22Z

test this please

viirya · 2017-06-21T05:15:42Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala

+
+  override def inputTypes: Seq[AbstractDataType] =
+    Seq(TypeCollection(DateType, DoubleType, DecimalType),
+      TypeCollection(StringType, IntegerType))


I think this might lead to wrong input types combinations such as (DoubleType, StringType) and (DateType, IntegerType)?

If we are going to have only trunc for truncating number and datetime. We should prevent wrong input types.

viirya · 2017-06-21T05:18:43Z

Is there duplicated codes between trunc(number) and trunc(date)? If no, seems to me we don't necessarily let one expression to have two different features.

viirya · 2017-06-21T05:27:12Z

Although then we can't use just one trunc function, it seems ok for me because not all databases use trunc to truncate both number and datetime.

SparkQA · 2017-06-27T12:47:01Z

Test build #78693 has finished for PR 18106 at commit f8b1f44.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

felixcheung · 2017-06-27T16:36:55Z

python/pyspark/sql/functions.py

@@ -1028,20 +1028,29 @@ def to_timestamp(col, format=None):


 @since(1.5)
-def trunc(date, format):
+def trunc(data, truncParam):


@ueshin @holdenk re: changing param name in python.

I believe this definitely breaks backward compatibility for keyword-argument usage in Python.

felixcheung · 2017-06-27T16:37:20Z

I'll add @gatorsmile since this is SQL.

wangyum · 2017-06-28T09:44:20Z

Jenkins, retest this please

SparkQA · 2017-06-28T11:23:31Z

Test build #78786 has finished for PR 18106 at commit f8b1f44.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2017-07-31T03:42:08Z

retest this please

SparkQA · 2017-07-31T05:49:14Z

Test build #80066 has finished for PR 18106 at commit f8b1f44.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2017-08-01T03:32:04Z

ping @wangyum

wangyum · 2017-08-01T10:28:04Z

I'll fix it

SparkQA · 2017-08-02T07:04:51Z

Test build #80150 has finished for PR 18106 at commit 3d40c36.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2017-08-02T07:07:31Z

Jenkins, retest this please

HyukjinKwon · 2017-08-02T07:16:50Z

python/pyspark/sql/functions.py

@@ -1028,20 +1028,29 @@ def to_timestamp(col, format=None):


 @since(1.5)
-def trunc(date, format):
+def trunc(data, truncParam):


@wangyum, would you mind revert this renaming? This breaks the compatibility if user script calls this by

trunc(..., format= ...) trunc(date=..., format= ...)

We can work around this with kwargs if it's important to change the name.

Yes but it brings complexity for both args and kwargs e.g., when both set, method signature in doc and etc. I wonder if it is that important.

SparkQA · 2017-08-02T09:49:48Z

Test build #80152 has finished for PR 18106 at commit 3d40c36.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-08-02T15:20:38Z

Test build #80159 has finished for PR 18106 at commit 931f07d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2017-10-27T23:47:48Z

This is close to merge. Could you resolve the conflicts? Then, I will review it. Thanks!

felixcheung · 2017-10-28T04:47:42Z

R has trunc in master/2.3 as well

# Conflicts: # sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala # sql/core/src/test/resources/sql-tests/inputs/operators.sql # sql/core/src/test/resources/sql-tests/results/operators.sql.out

-trunc

# Conflicts: # sql/core/src/test/resources/sql-tests/inputs/datetime.sql # sql/core/src/test/resources/sql-tests/results/datetime.sql.out

SparkQA · 2017-10-28T09:27:55Z

Test build #83155 has finished for PR 18106 at commit b59a2df.

This patch fails PySpark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

SparkQA · 2017-10-28T10:19:07Z

Test build #83158 has finished for PR 18106 at commit 679ff98.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class ClearCacheCommand() extends RunnableCommand

dongjoon-hyun · 2018-09-13T17:18:53Z

@wangyum .
https://issues.apache.org/jira/browse/SPARK-20754 is resolved by you. Could you close this PR?

wangyum · 2018-09-14T03:08:17Z

@dongjoon-hyun Actually TRUNC (number) not resolved. I will fix it soon.
https://issues.apache.org/jira/browse/SPARK-23906

dongjoon-hyun · 2018-09-14T03:12:38Z

+100, @wangyum . Thanks. :)

Support TRUNC (number)

a5ade70

Fix failed tests

c63856b

wangyum added 2 commits May 26, 2017 08:09

Add test for python api

224c867

Update comment.

e7e6e5b

Fix PySpark unit test issue

7157820

wangyum mentioned this pull request Jun 5, 2017

[SPARK-20754][SQL][FOLLOWUP] Add Function Alias For MOD/POSITION. #18206

Closed

wangyum added 2 commits June 14, 2017 22:54

Refactor code.

3d92a48

Add comment.

b391b6a

Merge branch 'master' into SPARK-20754-trunc

5456e61

viirya reviewed Jun 21, 2017

View reviewed changes

Support timestamp and string type.

f8b1f44

felixcheung reviewed Jun 27, 2017

View reviewed changes

codegen support String and Timestamp type.

3d40c36

HyukjinKwon reviewed Aug 2, 2017

View reviewed changes

Revert trunc(date, format).

931f07d

wangyum added 3 commits October 28, 2017 14:39

Merge remote-tracking branch 'origin/SPARK-20754-trunc' into SPARK-20754

b59a2df

-trunc

Merge remote-tracking branch 'upstream/master' into SPARK-20754-trunc

679ff98

# Conflicts: # sql/core/src/test/resources/sql-tests/inputs/datetime.sql # sql/core/src/test/resources/sql-tests/results/datetime.sql.out

wangyum closed this Sep 14, 2018

maropu mentioned this pull request Sep 18, 2018

[SPARK-23906][SQL] Add built-in UDF TRUNCATE(number) #22419

Closed

wangyum deleted the SPARK-20754-trunc branch October 8, 2019 04:25

[SPARK-20754][SQL] Support TRUNC (number) #18106

[SPARK-20754][SQL] Support TRUNC (number) #18106

Conversation

wangyum commented May 25, 2017 • edited Loading

What changes were proposed in this pull request?

How was this patch tested?

SparkQA commented May 25, 2017

SparkQA commented May 25, 2017

SparkQA commented May 26, 2017

SparkQA commented May 26, 2017

SparkQA commented Jun 14, 2017

SparkQA commented Jun 15, 2017

wangyum commented Jun 15, 2017

SparkQA commented Jun 15, 2017

wangyum commented Jun 15, 2017

SparkQA commented Jun 20, 2017

wangyum commented Jun 20, 2017

SparkQA commented Jun 20, 2017

shaneknapp commented Jun 21, 2017

SparkQA commented Jun 21, 2017

shaneknapp commented Jun 21, 2017

viirya Jun 21, 2017

Choose a reason for hiding this comment

viirya Jun 21, 2017

Choose a reason for hiding this comment

viirya commented Jun 21, 2017

viirya commented Jun 21, 2017

SparkQA commented Jun 27, 2017

felixcheung Jun 27, 2017

Choose a reason for hiding this comment

HyukjinKwon Jun 27, 2017

Choose a reason for hiding this comment

felixcheung commented Jun 27, 2017

wangyum commented Jun 28, 2017

SparkQA commented Jun 28, 2017

gatorsmile commented Jul 31, 2017

SparkQA commented Jul 31, 2017

gatorsmile commented Aug 1, 2017

wangyum commented Aug 1, 2017

SparkQA commented Aug 2, 2017

wangyum commented Aug 2, 2017

HyukjinKwon Aug 2, 2017

Choose a reason for hiding this comment

holdenk Aug 2, 2017

Choose a reason for hiding this comment

HyukjinKwon Aug 2, 2017

Choose a reason for hiding this comment

SparkQA commented Aug 2, 2017

SparkQA commented Aug 2, 2017

gatorsmile commented Oct 27, 2017

felixcheung commented Oct 28, 2017

SparkQA commented Oct 28, 2017

SparkQA commented Oct 28, 2017

dongjoon-hyun commented Sep 13, 2018

wangyum commented Sep 14, 2018

dongjoon-hyun commented Sep 14, 2018

wangyum commented May 25, 2017 •

edited

Loading