SQL: Extend DATE_TRUNC to also operate on intervals(elastic - #46632 ) #47720

musteaf · 2019-10-08T08:56:36Z

The function is extended to operate on intervals according to the PostgreSQL: https://www.postgresql.org/docs/9.1/functions-datetime.html#FUNCTIONS-DATETIME-TRUNC

Closes : #46632

elasticmachine · 2019-10-08T10:44:44Z

Pinging @elastic/es-search (:Search/SQL)

matriv

@musteaf,

First of all: thank you for your contribution!
Nice job so far, I left some comments.

Moreover, I'd like to ask you to update the documentation of the function here:
https://github.com/elastic/elasticsearch/master/docs/reference/sql/functions/date-time.asciidoc regarding the 2nd argument type (add the interval support) along with some example/integ tests
here: https://github.com/elastic/elasticsearch/blob/master/x-pack/plugin/sql/qa/src/main/resources/docs/docs.csv-spec#L2546
Please also add some integration tests here: https://github.com/elastic/elasticsearch/blob/master/x-pack/plugin/sql/qa/src/main/resources/datetime.csv-spec#L143 and if possible also some for filter/groupby/having/orderby below in the same file.

Thank you!

...java/org/elasticsearch/xpack/sql/expression/function/scalar/datetime/DateTruncProcessor.java

...src/main/java/org/elasticsearch/xpack/sql/expression/function/scalar/datetime/DateTrunc.java

...org/elasticsearch/xpack/sql/expression/function/scalar/datetime/DateTruncProcessorTests.java

...src/main/java/org/elasticsearch/xpack/sql/expression/function/scalar/datetime/DateTrunc.java

musteaf · 2019-10-08T14:10:51Z

@matriv thank you for review. I will fix the problems soon as possible.

astefan

Nice work, but it needs a serious formatting step before merging. I've added some comments, as well.

x-pack/plugin/sql/src/main/java/org/elasticsearch/xpack/sql/expression/TypeResolutions.java

...src/main/java/org/elasticsearch/xpack/sql/expression/function/scalar/datetime/DateTrunc.java

...java/org/elasticsearch/xpack/sql/expression/function/scalar/datetime/DateTruncProcessor.java

...org/elasticsearch/xpack/sql/expression/function/scalar/datetime/DateTruncProcessorTests.java

...src/main/java/org/elasticsearch/xpack/sql/expression/function/scalar/datetime/DateTrunc.java

musteaf · 2019-10-14T13:35:16Z

Hi @astefan Thank you very much for your kind review. I started to work on problems.

musteaf · 2019-10-17T01:09:22Z

@matriv
I completed most of the tasks. For integration tests, I want to test manually first. I have a problem with importing test datas for integration tests. Could you please share with me if there is any way to import test datas to elasticsearch? I tried Kibana but v7.4 doesnt work with master version of elasticsearch. If there is any code like " ./gradlew run ......." could you share with me. I could not find any information related to test datas in the documents.

matriv · 2019-10-17T10:20:56Z

@musteaf currently your PR is changed to point to your master branch:

musteaf wants to merge 6 commits into elastic:master from musteaf:master

and we can see no changes anymore.

Our parent class which loads the test_emp data is SpecBaseIntegrationTestCase.

musteaf · 2019-10-29T07:55:43Z

Hi @matriv,
I fixed all issues. I completed the desired things in the review. I committed. Can you see changes now?
But I have a problem with group by with interval type. I think it is related with #43072

This is the query:

dateTruncGroupByWithInterval
SELECT count(*) as count, birth_date + DATE_TRUNC('decade', INTERVAL '1 12:43:21' DAY TO SECONDS) AS dt FROM test_emp GROUP BY dt ORDER BY 2 LIMIT 3;

and error

Original type was [line 1:27: Cannot find grouping for 'birth_date + DATE_TRUNC('decade', INTERVAL '1 12:43:21' DAY TO SECONDS)']. [org.elasticsearch.xpack.sql.planner.FoldingException: line 1:27: Cannot find grouping for 'birth_date + DATE_TRUNC('decade', INTERVAL '1 12:43:21' DAY TO SECONDS)'

Do you have any suggestion?

matriv

@musteaf Thank you for addressing our comments, I left some new though, most of them minor.

Regarding the integ test for GROUP BY, how about not using an alias but repeat the function in the GROUP BY:

SELECT count(*) as count, birth_date + DATE_TRUNC('decade', INTERVAL '1 12:43:21' DAY TO SECONDS) AS dt FROM test_emp GROUP BY birth_date + DATE_TRUNC('decade', INTERVAL '1 12:43:21' DAY TO SECONDS) ORDER BY 2 LIMIT 3;

or even without the alias:

SELECT count(*) as count, birth_date + DATE_TRUNC('decade', INTERVAL '1 12:43:21' DAY TO SECONDS) FROM test_emp GROUP BY birth_date + DATE_TRUNC('decade', INTERVAL '1 12:43:21' DAY TO SECONDS) ORDER BY 2 LIMIT 3;

If neither work please add the test as is but with -Ignore as a suffix and right above it add a comment:

// AwaitsFix https://github.com/elastic/elasticsearch/issues/42041

so we can enable it once the bug is fixed.

docs/reference/sql/functions/date-time.asciidoc

x-pack/plugin/sql/qa/src/main/resources/datetime.csv-spec

.../src/main/java/org/elasticsearch/xpack/sql/expression/function/scalar/datetime/DatePart.java

...src/main/java/org/elasticsearch/xpack/sql/expression/function/scalar/datetime/DateTrunc.java

...java/org/elasticsearch/xpack/sql/expression/function/scalar/datetime/DateTruncProcessor.java

...org/elasticsearch/xpack/sql/expression/function/scalar/datetime/DateTruncProcessorTests.java

matriv · 2019-10-29T22:24:01Z

@elasticmachine test this please

musteaf · 2019-10-30T18:36:31Z

@matriv Thank you for your kind review. I really learned a lot from this PR. Thank you for help. I fixed the issues. But, unfortunately GROUP BY doesn't work.

matriv

I left some more comments, please check.

docs/reference/sql/functions/date-time.asciidoc

...java/org/elasticsearch/xpack/sql/expression/function/scalar/datetime/DateTruncProcessor.java

...org/elasticsearch/xpack/sql/expression/function/scalar/whitelist/InternalSqlScriptUtils.java

matriv · 2019-10-30T22:03:47Z

@elasticmachine test this please

matriv · 2019-10-30T22:04:36Z

But, unfortunately GROUP BY doesn't work.

Don't worry, thx for adding the test as ignored, it will be enabled once the bug is fixed.

matriv · 2019-10-30T22:17:27Z

@elasticmachine run elasticsearch-ci/2

matriv · 2019-10-30T22:24:15Z

@musteaf Please check and fix the checkstyle errors:


:x-pack:plugin:sql:checkstyleTest FAILED
--
[ant:checkstyle] [ERROR] /dev/shm/elastic+elasticsearch+pull-request-2/x-pack/plugin/sql/src/test/java/org/elasticsearch/xpack/sql/analysis/analyzer/VerifierErrorMessagesTests.java:215: Line is longer than 140 characters (found 153). [LineLength]
[ant:checkstyle] [ERROR] /dev/shm/elastic+elasticsearch+pull-request-2/x-pack/plugin/sql/src/test/java/org/elasticsearch/xpack/sql/expression/function/scalar/datetime/DateTruncProcessorTests.java:277: Line is longer than 140 characters (found 164). [LineLength]
[ant:checkstyle] [ERROR] /dev/shm/elastic+elasticsearch+pull-request-2/x-pack/plugin/sql/src/test/java/org/elasticsearch/xpack/sql/expression/function/scalar/datetime/DateTruncProcessorTests.java:281: Line is longer than 140 characters (found 157). [LineLength]
[ant:checkstyle] [ERROR] /dev/shm/elastic+elasticsearch+pull-request-2/x-pack/plugin/sql/src/test/java/org/elasticsearch/xpack/sql/expression/function/scalar/datetime/DateTruncProcessorTests.java:285: Line is longer than 140 characters (found 155). [LineLength]
[ant:checkstyle] [ERROR] /dev/shm/elastic+elasticsearch+pull-request-2/x-pack/plugin/sql/src/test/java/org/elasticsearch/xpack/sql/expression/function/scalar/datetime/DateTruncProcessorTests.java:289: Line is longer than 140 characters (found 156). [LineLength]
[ant:checkstyle] [ERROR] /dev/shm/elastic+elasticsearch+pull-request-2/x-pack/plugin/sql/src/test/java/org/elasticsearch/xpack/sql/expression/function/scalar/datetime/DateTruncProcessorTests.java:293: Line is longer than 140 characters (found 155). [LineLength]

musteaf · 2019-10-31T00:54:29Z

@matriv Thank you for review. I think I fixed the issues. I left comment about asDateTime. Do you have any suggestion about that?

matriv · 2019-10-31T00:56:59Z

@elasticmachine test this please

matriv · 2019-10-31T15:41:12Z

@musteaf the PR #43072 has been merged, so please merge master and enable the group by integration test.

musteaf · 2019-10-31T16:26:48Z

@matriv I merged the master and enabled the group by integration test.

matriv · 2019-10-31T21:02:40Z

@musteaf Please apply the change we discussed in the description here: #47720 (comment)

Please also add references to the 3 examples in the docs.csv-spec file you added to the ascii doc after the examples with Date.

matriv · 2020-03-16T09:55:53Z

@elasticmachine test this please.

matriv · 2020-03-16T10:07:18Z

:x-pack:plugin:sql:checkstyleMain FAILED	
[ant:checkstyle] [ERROR] /dev/shm/elastic+elasticsearch+pull-request-2/x-pack/plugin/sql/src/main/java/org/elasticsearch/xpack/sql/expression/function/scalar/datetime/DateTrunc.java:30: Using the '.*' form of import should be avoided - org.elasticsearch.xpack.ql.util.DateUtils.*. [AvoidStarImport]

matriv · 2020-03-16T18:34:38Z

@elasticmachine test this please

matriv · 2020-03-16T22:29:33Z

@musteaf Please run ./gradlew --parallel -p x-pack/plugin/sql check locally on the root dir of the checked out elasticsearch and check the integ tests errors.

musteaf · 2020-03-17T12:47:15Z

@matriv I fixed the issues. Also if there is a "gradlew" command to check styles, could you share with me for further PR? According to the CONTRIBUTING file, we can use Spotless Gradle plugin to check style but It only supports Eclipse. If there is not any "gradlew" command which works with Intellij, I will use Eclipse then. Thanks in advance.

matriv · 2020-03-18T12:58:14Z

@musteaf I've pasted it earlier: :x-pack:plugin:sql:checkstyleMain & for tests: :x-pack:plugin:sql:checkstyleTest but both are run with the check command I've sent.

matriv · 2020-03-18T12:58:26Z

@elasticmachine test this please

musteaf · 2020-03-18T15:15:07Z

@matriv thank you so much for the code snippet. I examined elasticmachine's build errors but I could not find any related error with this PR.

matriv · 2020-03-18T20:06:50Z

@elasticmachine update branch

matriv · 2020-03-18T20:07:29Z

@elasticmachine test this please

matriv

LGTM

astefan

LGTM. Left just few cosmetics related comments.

astefan · 2020-03-21T09:49:49Z

...src/main/java/org/elasticsearch/xpack/sql/expression/function/scalar/datetime/DateTrunc.java

+        int year = period.getYears();
+        int firstYearOfCentury = year - (year % 100);
+        return new IntervalYearMonth(Period.ZERO.plusYears(firstYearOfCentury), iym.dataType());


Wrong indentation.

astefan · 2020-03-21T10:45:53Z

...src/main/java/org/elasticsearch/xpack/sql/expression/function/scalar/datetime/DateTrunc.java

+import static org.elasticsearch.xpack.ql.util.DateUtils.SECONDS_PER_MINUTE;
+import static org.elasticsearch.xpack.sql.expression.SqlTypeResolutions.isDateOrInterval;
+import static org.elasticsearch.xpack.sql.type.SqlDataTypes.isInterval;
+
 public class DateTrunc extends BinaryDateTimeFunction {

    public enum Part implements DateTimeField {

        MILLENNIUM(dt -> {


Maybe a personal preference, but I think the Part enum values are more easily distinguishable if they are indented in a similar style to this: https://gist.github.com/astefan/a4c27b9bfefb1b074156c86d01d507ff.
At the moment, this is a big block of code where, every few lines, there is a capital letters noun without any clear boundaries.

astefan · 2020-03-21T10:52:54Z

...java/org/elasticsearch/xpack/sql/expression/function/scalar/datetime/DateTruncProcessor.java

+        }
+        if (truncateDateField == Part.WEEK && (timestamp instanceof IntervalDayTime || timestamp instanceof IntervalYearMonth)) {
+            throw new SqlIllegalArgumentException("Truncating intervals is not supported for {} units",
+                truncateTo);


This truncateTo can sit on the previous line.

matriv · 2020-03-21T15:28:37Z

@musteaf Could you please address those last comments?

matriv · 2020-03-23T08:44:54Z

@elasticmachine test this please

matriv · 2020-03-23T11:52:42Z

@musteaf Thank you for contribution!

musteaf · 2020-03-23T13:50:58Z

@matriv Thank you so much for your excellent guidance! That was so educative for me.

#47720) (#53972) The function is extended to operate on intervals according to the PostgreSQL: https://www.postgresql.org/docs/9.1/functions-datetime.html#FUNCTIONS-DATETIME-TRUNC Closes : #46632 (cherry picked from commit 2dc7950) Co-authored-by: musteaf <[email protected]>

musteaf mentioned this pull request Oct 8, 2019

SQL: Extend DATE_TRUNC to also operate on intervals #46632

Closed

musteaf changed the title ~~SQL: Extend DATE_TRUNC to also operate on intervals(elastic#46632)~~ SQL: Extend DATE_TRUNC to also operate on intervals(https://github.com/elastic/elasticsearch/issues/46632) Oct 8, 2019

musteaf changed the title ~~SQL: Extend DATE_TRUNC to also operate on intervals(https://github.com/elastic/elasticsearch/issues/46632)~~ SQL: Extend DATE_TRUNC to also operate on intervals(elastic - https://github.com/elastic/elasticsearch/issues/46632 ) Oct 8, 2019

musteaf changed the title ~~SQL: Extend DATE_TRUNC to also operate on intervals(elastic - https://github.com/elastic/elasticsearch/issues/46632 )~~ SQL: Extend DATE_TRUNC to also operate on intervals(elastic - #46632 ) Oct 8, 2019

matriv requested review from astefan, matriv and costin October 8, 2019 09:31

martijnvg added the :Analytics/SQL SQL querying label Oct 8, 2019

matriv reviewed Oct 8, 2019

View reviewed changes

astefan reviewed Oct 11, 2019

View reviewed changes

matriv reviewed Oct 29, 2019

View reviewed changes

matriv requested a review from astefan October 29, 2019 22:24

matriv reviewed Oct 30, 2019

View reviewed changes

style

d7a8c13

fixed interval ms

615fc86

Merge branch 'master' into master

cf004d7

matriv reviewed Mar 18, 2020

View reviewed changes

astefan approved these changes Mar 21, 2020

View reviewed changes

Mustafa Fırtına added 2 commits March 22, 2020 22:46

format

9ac9839

update

0b63888

matriv added >enhancement v7.7.0 v8.0.0 labels Mar 23, 2020

matriv merged commit 2dc7950 into elastic:master Mar 23, 2020

codebrain mentioned this pull request Apr 1, 2020

7.7.0 meta ticket (Part 2) elastic/elasticsearch-net#4533

Closed

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021

SQL: Extend DATE_TRUNC to also operate on intervals(elastic - #46632 ) #47720

SQL: Extend DATE_TRUNC to also operate on intervals(elastic - #46632 ) #47720

Conversation

musteaf commented Oct 8, 2019 • edited by matriv Loading

elasticmachine commented Oct 8, 2019

matriv left a comment

Choose a reason for hiding this comment

musteaf commented Oct 8, 2019

astefan left a comment

Choose a reason for hiding this comment

musteaf commented Oct 14, 2019

musteaf commented Oct 17, 2019 • edited Loading

matriv commented Oct 17, 2019

musteaf commented Oct 29, 2019 • edited Loading

matriv left a comment

Choose a reason for hiding this comment

matriv commented Oct 29, 2019

musteaf commented Oct 30, 2019

matriv left a comment

Choose a reason for hiding this comment

matriv commented Oct 30, 2019

matriv commented Oct 30, 2019

matriv commented Oct 30, 2019

matriv commented Oct 30, 2019

musteaf commented Oct 31, 2019

matriv commented Oct 31, 2019

matriv commented Oct 31, 2019

musteaf commented Oct 31, 2019 • edited Loading

matriv commented Oct 31, 2019 • edited Loading

matriv commented Mar 16, 2020

matriv commented Mar 16, 2020 • edited Loading

matriv commented Mar 16, 2020

matriv commented Mar 16, 2020

musteaf commented Mar 17, 2020

matriv commented Mar 18, 2020

matriv commented Mar 18, 2020

musteaf commented Mar 18, 2020 • edited Loading

matriv commented Mar 18, 2020

matriv commented Mar 18, 2020

matriv left a comment

Choose a reason for hiding this comment

astefan left a comment

Choose a reason for hiding this comment

astefan Mar 21, 2020

Choose a reason for hiding this comment

astefan Mar 21, 2020

Choose a reason for hiding this comment

astefan Mar 21, 2020

Choose a reason for hiding this comment

matriv commented Mar 21, 2020

matriv commented Mar 23, 2020

matriv commented Mar 23, 2020

musteaf commented Mar 23, 2020

musteaf commented Oct 8, 2019 •

edited by matriv

Loading

musteaf commented Oct 17, 2019 •

edited

Loading

musteaf commented Oct 29, 2019 •

edited

Loading

musteaf commented Oct 31, 2019 •

edited

Loading

matriv commented Oct 31, 2019 •

edited

Loading

matriv commented Mar 16, 2020 •

edited

Loading

musteaf commented Mar 18, 2020 •

edited

Loading