Support date and time functions: date_add, date_sub, day, dayname, dayofweek, dayofyear, from_days, hour, microsecond, minute, month, monthname, quarter, second, subdate, time_to_sec, to_days #746

rupal-bq · 2020-09-18T19:06:30Z

Issue #709

Description of changes:
Added implementation, unit test, manual IT, and doc updates for following date and time functions:

date_add
date_sub
day
dayname
dayofweek
dayofyear
from_days
hour
microsecond
minute
month
monthname
quarter
second
subdate
time_to_sec
to_days
year

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

…ime-functions # Conflicts: # core/src/main/java/com/amazon/opendistroforelasticsearch/sql/expression/datetime/DateTimeFunction.java # core/src/test/java/com/amazon/opendistroforelasticsearch/sql/expression/datetime/DateTimeFunctionTest.java

penghuo · 2020-09-21T16:04:07Z

docs/user/dql/functions.rst

+(DATE, DATE INTERVAL) -> DATE
+(DATE, TIME INTERVAL) -> DATETIME
+(DATETIME/TIMESTAMP, INTERVAL) -> DATETIME


The format is the doc is not correct.

could you add example for each case?

fixed format & added example for each case.

penghuo · 2020-09-21T16:04:56Z

docs/user/dql/functions.rst

+Description
+-----------
+
+Usage: date_add(date, INTERVAL expr unit) adds the time interval expr to date


please add other signature, e.g. data_add(date, expression) also

penghuo · 2020-09-21T16:05:38Z

docs/user/dql/functions.rst

+(DATE, TIME INTERVAL) -> DATETIME
+(DATETIME/TIMESTAMP, INTERVAL) -> DATETIME
+
+Synonyms: ADDDATE


do we support ADDDATE? if not, remove this.

we support ADDDATE

penghuo · 2020-09-21T16:17:45Z

...ain/java/com/amazon/opendistroforelasticsearch/sql/expression/datetime/DateTimeFunction.java

+        impl(nullMissingHandling(DateTimeFunction::exprAddDateInterval), DATE, DATE, INTERVAL),
+        impl(nullMissingHandling(DateTimeFunction::exprAddDateInterval), DATETIME, DATE, INTERVAL),


why the function has same input signature but has different output signature?

some use cases required different return type. E.g select date_add(date('2020-09-16'), interval 1 hour) should return 2020-09-16 01:00:00 but select date_add(date('2020-09-16'), interval 1 day) should return 2020-09-17. But this was always returning datetime in schema anyway so removed it.

penghuo · 2020-09-21T16:18:36Z

...ain/java/com/amazon/opendistroforelasticsearch/sql/expression/datetime/DateTimeFunction.java

+  /**
+   * Specify a start date and add a temporal amount to the date.
+   * The return type depends on the date type and the interval unit. Detailed supported signatures:
+   * (DATE, DATETIME/TIMESTAMP, INTERVAL) -> DATETIME


do you mean (DATE/DATETIME/TIMESTAMP, INTERVAL) -> DATETIME

yes, fixed it.

penghuo · 2020-09-21T16:22:05Z

...ain/java/com/amazon/opendistroforelasticsearch/sql/expression/datetime/DateTimeFunction.java

+   *
+   * @param date ExprValue of Date/Datetime/Timestamp type.
+   * @param expr ExprValue of Interval type, the temporal amount to add.
+   * @return Date/Datetime resulted from expr added to date.


The ExprDatetimeValue only has type DATETIME.

penghuo · 2020-09-21T16:31:55Z

...ain/java/com/amazon/opendistroforelasticsearch/sql/expression/datetime/DateTimeFunction.java

+   */
+  private ExprValue exprDayOfWeek(ExprValue date) {
+    if (date instanceof ExprStringValue) {
+      date = new ExprDateValue(date.stringValue());


The input value should be final

penghuo · 2020-09-21T16:53:18Z

Two questions about String data type.

When the String should be used? is in date_add function, it doesn't accept String as paramater, but in month function it accept String and Date.
Whe the String can be only used to represent DATE?, I see String can be only used to represent DATE, why not applied to DATETIME/TIMESTAMP/TIME?

I think we should have consistent interface for user. There are two options in my mind.

allow user to use String in function which accept DATE/DATETIME/TIMESTAMP/TIME.
2.only allow use to use String for function like date(), time(), timestamp().

penghuo · 2020-09-21T17:04:22Z

...ain/java/com/amazon/opendistroforelasticsearch/sql/expression/datetime/DateTimeFunction.java

+  /**
+   * DAY(DATE). return the day of the month (1-31).
+   */
+  private FunctionResolver day() {


e.g. day(date_add("2020-09-21", 1))
this will throw exception because day function can't accept datetime as paramater?

added datetime & timestamp for all functions

penghuo · 2020-09-21T17:15:08Z

Please also add test cases for PPL

…ime-functions # Conflicts: # core/src/main/java/com/amazon/opendistroforelasticsearch/sql/expression/DSL.java # core/src/main/java/com/amazon/opendistroforelasticsearch/sql/expression/datetime/DateTimeFunction.java # core/src/main/java/com/amazon/opendistroforelasticsearch/sql/expression/function/BuiltinFunctionName.java # core/src/test/java/com/amazon/opendistroforelasticsearch/sql/expression/datetime/DateTimeFunctionTest.java # docs/user/dql/functions.rst # integ-test/src/test/java/com/amazon/opendistroforelasticsearch/sql/sql/DateTimeFunctionIT.java # ppl/src/main/antlr/OpenDistroPPLLexer.g4 # ppl/src/main/antlr/OpenDistroPPLParser.g4 # sql/src/main/antlr/OpenDistroSQLParser.g4

rupal-bq · 2020-09-24T18:22:55Z

Please also add test cases for PPL

Added integration tests for PPL

rupal-bq · 2020-09-24T18:40:08Z

Two questions about String data type.

When the String should be used? is in date_add function, it doesn't accept String as paramater, but in month function it accept String and Date.

Whe the String can be only used to represent DATE?, I see String can be only used to represent DATE, why not applied to DATETIME/TIMESTAMP/TIME?

I think we should have consistent interface for user. There are two options in my mind.

allow user to use String in function which accept DATE/DATETIME/TIMESTAMP/TIME.
2.only allow use to use String for function like date(), time(), timestamp().

added string for all date-time functions.

rupal-bq · 2020-09-24T18:50:15Z

...ain/java/com/amazon/opendistroforelasticsearch/sql/expression/datetime/DateTimeFunction.java

+   * @return Datetime resulted from expr added to date.
+   */
+  private ExprValue exprAddDateInterval(ExprValue date, ExprValue expr) {
+    ExprValue exprValue = new ExprDatetimeValue(date.datetimeValue().plus(expr.intervalValue()));


adddate, date_add, date_sub & subdate need to return date/datetime according to calculated result. Here, I have returned date if time is zero for these functions to get correct result. But the return type will be still be datetime in schema because multiple return type can't be added in define function for the same input parameters.

…ime-functions

penghuo · 2020-09-28T16:51:22Z

core/src/main/java/com/amazon/opendistroforelasticsearch/sql/data/model/ExprStringValue.java

@@ -43,6 +48,40 @@ public String stringValue() {
    return value;
  }

+  @Override


could you add the test cases for these functions?

added test cases for date-time functions in ExprStringValue

…ime-functions # Conflicts: # sql/src/main/antlr/OpenDistroSQLIdentifierParser.g4

penghuo

Thanks for the change.

Rupal Mahajan added 30 commits September 8, 2020 18:29

add functions day, month, quarter, year

db82e29

fix build error

e46da70

fix doctest error

88a878b

fix doctest build error

63ea788

fix doctest

7551dc8

add dayofmonth()

a673fe2

add dayofyear()

7c70c33

add dayofweek()

d87f8f4

fix dayofweek logic & add unit test

242403c

fix doctest for dayofweek()

be7051d

add dayname

6834c10

add monthname

ee02527

fix checkstyle build error

8ab2f63

fix build error

458c34a

fix doctest for monthname

e42aac1

add hour()

960783b

add minute()

4d355b2

add second

0881f24

add microsecond

d199d54

fix datetime & timestamp issue for microsecond

c88a6dd

add time_to_sec

4dd0db0

add subdate & date_sub

988d177

fix doctest error

693211a

fix build error

36449f7

add KeywordsCanBeId for dayofweek

048d5d6

add to_days

c0593d9

add from_days()

7ee0dbc

arrange by alphabetical order

7764a0d

add manual IT

18a6a84

penghuo reviewed Sep 21, 2020

View reviewed changes

Rupal Mahajan added 6 commits September 22, 2020 17:03

update tests & doc

20acde3

fix doc format

42c7222

update tests for adddate

0f3894a

move string conversion to ExprStringValue

a73a8e9

add string type in doc

ea9b190

rupal-bq commented Sep 24, 2020

View reviewed changes

Merge branch 'develop' of https://github.com/rupal-bq/sql into date-t…

61ac4e5

…ime-functions

lyndonbauto approved these changes Sep 25, 2020

View reviewed changes

penghuo reviewed Sep 28, 2020

View reviewed changes

Rupal Mahajan added 4 commits September 28, 2020 11:15

add test cases for datetime function in ExpeStringValue

dc70cd6

Merge branch 'develop' of https://github.com/rupal-bq/sql into date-t…

d93f02b

…ime-functions # Conflicts: # sql/src/main/antlr/OpenDistroSQLIdentifierParser.g4

removing implicit def for keyword in parser

c001615

add dayofweek

104e6b9

penghuo approved these changes Sep 29, 2020

View reviewed changes

dai-chen approved these changes Sep 29, 2020

View reviewed changes

dai-chen merged commit 9fa2b78 into opendistro-for-elasticsearch:develop Sep 29, 2020

lizsnyder mentioned this pull request Jul 16, 2021

Add missing sql settings opendistro/for-elasticsearch-docs#499

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support date and time functions: date_add, date_sub, day, dayname, dayofweek, dayofyear, from_days, hour, microsecond, minute, month, monthname, quarter, second, subdate, time_to_sec, to_days #746

Support date and time functions: date_add, date_sub, day, dayname, dayofweek, dayofyear, from_days, hour, microsecond, minute, month, monthname, quarter, second, subdate, time_to_sec, to_days #746

rupal-bq commented Sep 18, 2020 •

edited

Loading

penghuo Sep 21, 2020

penghuo Sep 21, 2020

rupal-bq Sep 24, 2020

penghuo Sep 21, 2020

rupal-bq Sep 24, 2020

penghuo Sep 21, 2020

rupal-bq Sep 24, 2020

penghuo Sep 21, 2020

rupal-bq Sep 24, 2020

penghuo Sep 21, 2020

rupal-bq Sep 24, 2020

penghuo Sep 21, 2020

penghuo Sep 21, 2020

penghuo commented Sep 21, 2020

penghuo Sep 21, 2020

rupal-bq Sep 24, 2020

penghuo commented Sep 21, 2020

rupal-bq commented Sep 24, 2020

rupal-bq commented Sep 24, 2020

rupal-bq Sep 24, 2020

penghuo Sep 28, 2020

rupal-bq Sep 28, 2020

penghuo left a comment

		impl(nullMissingHandling(DateTimeFunction::exprAddDateInterval), DATE, DATE, INTERVAL),
		impl(nullMissingHandling(DateTimeFunction::exprAddDateInterval), DATETIME, DATE, INTERVAL),

Support date and time functions: date_add, date_sub, day, dayname, dayofweek, dayofyear, from_days, hour, microsecond, minute, month, monthname, quarter, second, subdate, time_to_sec, to_days #746

Support date and time functions: date_add, date_sub, day, dayname, dayofweek, dayofyear, from_days, hour, microsecond, minute, month, monthname, quarter, second, subdate, time_to_sec, to_days #746

Conversation

rupal-bq commented Sep 18, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

penghuo commented Sep 21, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

penghuo commented Sep 21, 2020

rupal-bq commented Sep 24, 2020

rupal-bq commented Sep 24, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

penghuo left a comment

Choose a reason for hiding this comment

rupal-bq commented Sep 18, 2020 •

edited

Loading