Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SQL: Implement DATE_TRUNC function #46473

Merged
merged 12 commits into from
Sep 11, 2019
75 changes: 74 additions & 1 deletion docs/reference/sql/functions/date-time.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
[[sql-functions-datetime-interval]]
==== Intervals

A common requirement when dealing with date/time in general revolves around
A common requirement when dealing with date/time in general revolves around
the notion of `interval`, a topic that is worth exploring in the context of {es} and {es-sql}.

{es} has comprehensive support for <<date-math, date math>> both inside <<date-math-index-names, index names>> and <<mapping-date-format, queries>>.
Expand Down Expand Up @@ -248,6 +248,79 @@ include-tagged::{sql-specs}/docs/docs.csv-spec[filterNow]
Currently, using a _precision_ greater than 3 doesn't make any difference to the output of the
function as the maximum number of second fractional digits returned is 3 (milliseconds).

[[sql-functions-datetime-trunc]]
==== `DATE_TRUNC`

.Synopsis:
[source, sql]
--------------------------------------------------
DATE_TRUNC(
string_exp, <1>
datetime_exp) <2>
--------------------------------------------------

*Input*:

<1> string expression denoting the unit to which the date/datetime should be truncated
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this phrase be string expression denoting the unit which the date/datetime should be truncated to?

<2> date/datetime expression

*Output*: date/datetime, same as datetime_exp
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

`datetime_exp`


.Description:

Truncate the date/datetime to the specified unit by setting all fields that are less significant than the specified
one to zero (or one, for day, day of week and month).

[cols="^,^"]
|===
2+h|Datetime truncation units

s|unit
s|abbreviations

| millennium | millennia
| century | centuries
| decade | decades
| year | years, yy, yyyy
| quarter | quarters, qq, q
| month | months, mm, m
| week | weeks, wk, ww
| day | days, dd, d
| hour | hours, hh
| minute | minutes, mi, n
| second | seconds, ss, s
| millisecond | milliseconds, ms
| microsecond | microseconds, mcs
| nanosecond | nanoseconds, ns
|===



[source, sql]
--------------------------------------------------
include-tagged::{sql-specs}/docs/docs.csv-spec[truncateDateTimeMillennium]
--------------------------------------------------

[source, sql]
--------------------------------------------------
include-tagged::{sql-specs}/docs/docs.csv-spec[truncateDateTimeWeek]
--------------------------------------------------

[source, sql]
--------------------------------------------------
include-tagged::{sql-specs}/docs/docs.csv-spec[truncateDateTimeMinutes]
--------------------------------------------------

[source, sql]
--------------------------------------------------
include-tagged::{sql-specs}/docs/docs.csv-spec[truncateDateDecades]
--------------------------------------------------

[source, sql]
--------------------------------------------------
include-tagged::{sql-specs}/docs/docs.csv-spec[truncateDateQuarter]
--------------------------------------------------

[[sql-functions-datetime-day]]
==== `DAY_OF_MONTH/DOM/DAY`

Expand Down
1 change: 1 addition & 0 deletions docs/reference/sql/functions/index.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@
** <<sql-functions-current-date>>
** <<sql-functions-current-time>>
** <<sql-functions-current-timestamp>>
** <<sql-functions-datetime-trunc>>
** <<sql-functions-datetime-day>>
** <<sql-functions-datetime-dow>>
** <<sql-functions-datetime-doy>>
Expand Down
17 changes: 9 additions & 8 deletions x-pack/plugin/sql/qa/src/main/resources/command.csv-spec
Original file line number Diff line number Diff line change
Expand Up @@ -39,16 +39,17 @@ CURRENT_DATE |SCALAR
CURRENT_TIME |SCALAR
CURRENT_TIMESTAMP|SCALAR
CURTIME |SCALAR
DATE_TRUNC |SCALAR
DAY |SCALAR
DAYNAME |SCALAR
DAYOFMONTH |SCALAR
DAYOFWEEK |SCALAR
DAYOFYEAR |SCALAR
DAY_NAME |SCALAR
DAY_OF_MONTH |SCALAR
DAY_OF_WEEK |SCALAR
DAY_OF_YEAR |SCALAR
DOM |SCALAR
DAYOFMONTH |SCALAR
DAYOFWEEK |SCALAR
DAYOFYEAR |SCALAR
DAY_NAME |SCALAR
DAY_OF_MONTH |SCALAR
DAY_OF_WEEK |SCALAR
DAY_OF_YEAR |SCALAR
DOM |SCALAR
DOW |SCALAR
DOY |SCALAR
HOUR |SCALAR
Expand Down
96 changes: 95 additions & 1 deletion x-pack/plugin/sql/qa/src/main/resources/datetime.csv-spec
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,100 @@ SELECT WEEK(birth_date) week, birth_date FROM test_emp WHERE WEEK(birth_date) >
2 |1953-01-07T00:00:00.000Z
;

selectDateTruncWithDateTime
schema::dt_hour:ts|dt_min:ts|dt_sec:ts|dt_millis:s|dt_micro:s|dt_nano:s
Copy link
Contributor Author

@matriv matriv Sep 8, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have to cast some columns to string as the .XXX msecs part is not properly read by the CSV infrastructure.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's worth opening another issue for this to fix it long term (either inside the CSV library and elsewhere).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done: #46511

SELECT DATE_TRUNC('hour', '2019-09-04T11:22:33.123Z'::datetime) as dt_hour, DATE_TRUNC('minute', '2019-09-04T11:22:33.123Z'::datetime) as dt_min,
DATE_TRUNC('seconds', '2019-09-04T11:22:33.123Z'::datetime) as dt_sec, DATE_TRUNC('ms', '2019-09-04T11:22:33.123Z'::datetime)::string as dt_millis,
DATE_TRUNC('mcs', '2019-09-04T11:22:33.123Z'::datetime)::string as dt_micro, DATE_TRUNC('nanoseconds', '2019-09-04T11:22:33.123Z'::datetime)::string as dt_nano;

dt_hour | dt_min | dt_sec | dt_millis | dt_micro | dt_nano
-------------------------+---------------------------+--------------------------+--------------------------+--------------------------+-------------------------
2019-09-04T11:00:00.000Z | 2019-09-04T11:22:00.000Z | 2019-09-04T11:22:33.000Z | 2019-09-04T11:22:33.123Z | 2019-09-04T11:22:33.123Z | 2019-09-04T11:22:33.123Z
;

selectDateTruncWithDate
schema::dt_mil:s|dt_cent:s|dt_dec:s|dt_year:s|dt_quarter:s|dt_month:s|dt_week:s|dt_day:s
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have to cast as string here for the time being, working on a solution to properly recognise the DATE type.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be fixed with: 4b119dc

SELECT DATE_TRUNC('millennia', '2019-09-04'::date)::string as dt_mil, DATE_TRUNC('century', '2019-09-04'::date)::string as dt_cent,
DATE_TRUNC('decades', '2019-09-04'::date)::string as dt_dec, DATE_TRUNC('year', '2019-09-04'::date)::string as dt_year,
DATE_TRUNC('quarter', '2019-09-04'::date)::string as dt_quarter, DATE_TRUNC('month', '2019-09-04'::date)::string as dt_month,
DATE_TRUNC('week', '2019-09-04'::date)::string as dt_week, DATE_TRUNC('day', '2019-09-04'::date)::string as dt_day;

dt_mil | dt_cent | dt_dec | dt_year | dt_quarter | dt_month | dt_week | dt_day
-------------------------+--------------------------+--------------------------+--------------------------+--------------------------+--------------------------+--------------------------+-------------------------
2000-01-01T00:00:00.000Z | 2000-01-01T00:00:00.000Z | 2010-01-01T00:00:00.000Z | 2019-01-01T00:00:00.000Z | 2019-07-01T00:00:00.000Z | 2019-09-01T00:00:00.000Z | 2019-09-02T00:00:00.000Z | 2019-09-04T00:00:00.000Z
;

selectDateTruncWithField
schema::emp_no:i|birth_date:ts|dt_mil:ts|dt_cent:ts|dt_dec:ts|dt_year:ts|dt_quarter:ts|dt_month:ts|dt_week:ts|dt_day:ts
SELECT emp_no, birth_date, DATE_TRUNC('millennium', birth_date) as dt_mil, DATE_TRUNC('centuries', birth_date) as dt_cent,
DATE_TRUNC('decades', birth_date) as dt_dec, DATE_TRUNC('year', birth_date) as dt_year, DATE_TRUNC('quarter', birth_date) as dt_quarter,
DATE_TRUNC('month', birth_date) as dt_month, DATE_TRUNC('week', birth_date) as dt_week, DATE_TRUNC('day', birth_date) as dt_day
FROM test_emp WHERE emp_no >= 10032 AND emp_no <= 10042 ORDER BY 1;

emp_no | birth_date | dt_mil | dt_cent | dt_dec | dt_year | dt_quarter | dt_month | dt_week | dt_day
--------+-------------------------+--------------------------+--------------------------+--------------------------+--------------------------+--------------------------+--------------------------+--------------------------+-------------------------
10032 |1960-08-09 00:00:00.000Z | 0999-12-27 00:00:00.000Z | 1900-01-01 00:00:00.000Z | 1960-01-01 00:00:00.000Z | 1960-01-01 00:00:00.000Z | 1960-07-01 00:00:00.000Z | 1960-08-01 00:00:00.000Z | 1960-08-08 00:00:00.000Z | 1960-08-09 00:00:00.000Z
10033 |1956-11-14 00:00:00.000Z | 0999-12-27 00:00:00.000Z | 1900-01-01 00:00:00.000Z | 1950-01-01 00:00:00.000Z | 1956-01-01 00:00:00.000Z | 1956-10-01 00:00:00.000Z | 1956-11-01 00:00:00.000Z | 1956-11-12 00:00:00.000Z | 1956-11-14 00:00:00.000Z
10034 |1962-12-29 00:00:00.000Z | 0999-12-27 00:00:00.000Z | 1900-01-01 00:00:00.000Z | 1960-01-01 00:00:00.000Z | 1962-01-01 00:00:00.000Z | 1962-10-01 00:00:00.000Z | 1962-12-01 00:00:00.000Z | 1962-12-24 00:00:00.000Z | 1962-12-29 00:00:00.000Z
10035 |1953-02-08 00:00:00.000Z | 0999-12-27 00:00:00.000Z | 1900-01-01 00:00:00.000Z | 1950-01-01 00:00:00.000Z | 1953-01-01 00:00:00.000Z | 1953-01-01 00:00:00.000Z | 1953-02-01 00:00:00.000Z | 1953-02-02 00:00:00.000Z | 1953-02-08 00:00:00.000Z
10036 |1959-08-10 00:00:00.000Z | 0999-12-27 00:00:00.000Z | 1900-01-01 00:00:00.000Z | 1950-01-01 00:00:00.000Z | 1959-01-01 00:00:00.000Z | 1959-07-01 00:00:00.000Z | 1959-08-01 00:00:00.000Z | 1959-08-10 00:00:00.000Z | 1959-08-10 00:00:00.000Z
10037 |1963-07-22 00:00:00.000Z | 0999-12-27 00:00:00.000Z | 1900-01-01 00:00:00.000Z | 1960-01-01 00:00:00.000Z | 1963-01-01 00:00:00.000Z | 1963-07-01 00:00:00.000Z | 1963-07-01 00:00:00.000Z | 1963-07-22 00:00:00.000Z | 1963-07-22 00:00:00.000Z
10038 |1960-07-20 00:00:00.000Z | 0999-12-27 00:00:00.000Z | 1900-01-01 00:00:00.000Z | 1960-01-01 00:00:00.000Z | 1960-01-01 00:00:00.000Z | 1960-07-01 00:00:00.000Z | 1960-07-01 00:00:00.000Z | 1960-07-18 00:00:00.000Z | 1960-07-20 00:00:00.000Z
10039 |1959-10-01 00:00:00.000Z | 0999-12-27 00:00:00.000Z | 1900-01-01 00:00:00.000Z | 1950-01-01 00:00:00.000Z | 1959-01-01 00:00:00.000Z | 1959-10-01 00:00:00.000Z | 1959-10-01 00:00:00.000Z | 1959-09-28 00:00:00.000Z | 1959-10-01 00:00:00.000Z
10040 | null | null | null | null | null | null | null | null | null
10041 | null | null | null | null | null | null | null | null | null
10042 | null | null | null | null | null | null | null | null | null
;

dateTruncOrderBy
schema::emp_no:i|hire_date:ts|dt:ts
SELECT emp_no, hire_date, DATE_TRUNC('quarter', hire_date) as dt FROM test_emp ORDER BY dt NULLS LAST, emp_no LIMIT 5;

emp_no | hire_date | dt
--------+--------------------------+-------------------------
10009 | 1985-02-18 00:00:00.000Z | 1985-01-01 00:00:00.000Z
10048 | 1985-02-24 00:00:00.000Z | 1985-01-01 00:00:00.000Z
10098 | 1985-05-13 00:00:00.000Z | 1985-04-01 00:00:00.000Z
10061 | 1985-09-17 00:00:00.000Z | 1985-07-01 00:00:00.000Z
10076 | 1985-07-09 00:00:00.000Z | 1985-07-01 00:00:00.000Z
;

dateTruncFilter
schema::emp_no:i|hire_date:ts|dt:ts
SELECT emp_no, hire_date, DATE_TRUNC('quarter', hire_date) as dt FROM test_emp WHERE DATE_TRUNC('quarter', hire_date) > '1994-07-01T00:00:00.000Z'::timestamp ORDER BY emp_no;

emp_no | hire_date | dt
--------+--------------------------+-------------------------
10016 | 1995-01-27 00:00:00.000Z | 1995-01-01 00:00:00.000Z
10019 | 1999-04-30 00:00:00.000Z | 1999-04-01 00:00:00.000Z
10022 | 1995-08-22 00:00:00.000Z | 1995-07-01 00:00:00.000Z
10024 | 1997-05-19 00:00:00.000Z | 1997-04-01 00:00:00.000Z
10026 | 1995-03-20 00:00:00.000Z | 1995-01-01 00:00:00.000Z
10054 | 1995-03-13 00:00:00.000Z | 1995-01-01 00:00:00.000Z
10084 | 1995-12-15 00:00:00.000Z | 1995-10-01 00:00:00.000Z
10093 | 1996-11-05 00:00:00.000Z | 1996-10-01 00:00:00.000Z
;

dateTruncGroupBy
schema::count:l|dt:ts
SELECT count(*) as count, DATE_TRUNC('decade', hire_date) dt FROM test_emp GROUP BY dt ORDER BY 2;

count | dt
--------+-------------------------
59 | 1980-01-01 00:00:00.000Z
41 | 1990-01-01 00:00:00.000Z
;

dateTruncHaving
schema::gender:s|dt:ts
SELECT gender, max(hire_date) dt FROM test_emp GROUP BY gender HAVING DATE_TRUNC('year', max(hire_date)) >= '1997-01-01T00:00:00.000Z'::timestamp ORDER BY 1;

gender | dt
--------+-------------------------
null | 1999-04-30 00:00:00.000Z
F | 1997-05-19 00:00:00.000Z
;

//
// Aggregate
//
Expand Down Expand Up @@ -404,4 +498,4 @@ SELECT CAST (CAST (birth_date AS VARCHAR) AS TIMESTAMP) a FROM test_emp WHERE YE
a:ts
---------------
1965-01-03T00:00:00Z
;
;
69 changes: 61 additions & 8 deletions x-pack/plugin/sql/qa/src/main/resources/docs/docs.csv-spec
Original file line number Diff line number Diff line change
Expand Up @@ -235,16 +235,17 @@ CURRENT_DATE |SCALAR
CURRENT_TIME |SCALAR
CURRENT_TIMESTAMP|SCALAR
CURTIME |SCALAR
DATE_TRUNC |SCALAR
DAY |SCALAR
DAYNAME |SCALAR
DAYOFMONTH |SCALAR
DAYOFWEEK |SCALAR
DAYOFYEAR |SCALAR
DAY_NAME |SCALAR
DAY_OF_MONTH |SCALAR
DAY_OF_WEEK |SCALAR
DAY_OF_YEAR |SCALAR
DOM |SCALAR
DAYOFMONTH |SCALAR
DAYOFWEEK |SCALAR
DAYOFYEAR |SCALAR
DAY_NAME |SCALAR
DAY_OF_MONTH |SCALAR
DAY_OF_WEEK |SCALAR
DAY_OF_YEAR |SCALAR
DOM |SCALAR
DOW |SCALAR
DOY |SCALAR
HOUR |SCALAR
Expand Down Expand Up @@ -2412,6 +2413,58 @@ SELECT DAY_OF_MONTH(CAST('2018-02-19T10:23:27Z' AS TIMESTAMP)) AS day;
// end::dayOfMonth
;

truncateDateTimeMillennium
// tag::truncateDateTimeMillennium
SELECT DATE_TRUNC('millennium', '2019-09-04T11:22:33.123Z'::datetime) AS millennium;

millennium
------------------------
2000-01-01T00:00:00.000Z
// end::truncateDateTimeMillennium
;

truncateDateTimeWeek
// tag::truncateDateTimeWeek
SELECT DATE_TRUNC('week', '2019-08-24T11:22:33.123Z'::datetime) AS week;

week
------------------------
2019-08-19T00:00:00.000Z
// end::truncateDateTimeWeek
;

truncateDateTimeMinutes
// tag::truncateDateTimeMinutes
SELECT DATE_TRUNC('mi', '2019-09-04T11:22:33.123Z'::datetime) AS mins;

mins
------------------------
2019-09-04T11:22:00.000Z
// end::truncateDateTimeMinutes
;

truncateDateDecades
schema::decades:s
// tag::truncateDateDecades
SELECT DATE_TRUNC('decade', CAST('2019-09-04' AS DATE))::string AS decades;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have to cast as string here for the time being, working on a solution to properly recognise the DATE type.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be fixed with: 4b119dc


decades
------------------------
2010-01-01T00:00:00.000Z
// end::truncateDateDecades
;

truncateDateQuarter
schema::quarter:s
// tag::truncateDateQuarter
SELECT DATE_TRUNC('quarters', CAST('2019-09-04' AS DATE))::string AS quarter;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be fixed with: 4b119dc


quarter
------------------------
2019-07-01T00:00:00.000Z
// end::truncateDateQuarter
;

constantDayOfWeek
// tag::dayOfWeek
SELECT DAY_OF_WEEK(CAST('2018-02-19T10:23:27Z' AS TIMESTAMP)) AS day;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
import static java.time.temporal.ChronoField.HOUR_OF_DAY;
import static java.time.temporal.ChronoField.MILLI_OF_SECOND;
import static java.time.temporal.ChronoField.MINUTE_OF_HOUR;
import static java.time.temporal.ChronoField.NANO_OF_SECOND;
import static java.time.temporal.ChronoField.SECOND_OF_MINUTE;

public final class StringUtils {
Expand All @@ -40,6 +41,19 @@ public final class StringUtils {
.appendOffsetId()
.toFormatter(IsoLocale.ROOT);

public static final DateTimeFormatter ISO_DATE_WITH_NANOS = new DateTimeFormatterBuilder()
.parseCaseInsensitive()
.append(ISO_LOCAL_DATE)
.appendLiteral('T')
.appendValue(HOUR_OF_DAY, 2)
.appendLiteral(':')
.appendValue(MINUTE_OF_HOUR, 2)
.appendLiteral(':')
.appendValue(SECOND_OF_MINUTE, 2)
.appendFraction(NANO_OF_SECOND, 3, 9, true)
.appendOffsetId()
.toFormatter(IsoLocale.ROOT);

public static final DateTimeFormatter ISO_TIME_WITH_MILLIS = new DateTimeFormatterBuilder()
.parseCaseInsensitive()
.appendValue(HOUR_OF_DAY, 2)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,6 @@ public static TypeResolution isNumericOrDateOrTime(Expression e, String operatio
"date", "time", "datetime", "numeric");
}


public static TypeResolution isGeo(Expression e, String operationName, ParamOrdinal paramOrd) {
return isType(e, DataType::isGeo, operationName, paramOrd, "geo_point", "geo_shape");
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@
import org.elasticsearch.xpack.sql.expression.function.scalar.datetime.CurrentDate;
import org.elasticsearch.xpack.sql.expression.function.scalar.datetime.CurrentDateTime;
import org.elasticsearch.xpack.sql.expression.function.scalar.datetime.CurrentTime;
import org.elasticsearch.xpack.sql.expression.function.scalar.datetime.DateTrunc;
import org.elasticsearch.xpack.sql.expression.function.scalar.datetime.DayName;
import org.elasticsearch.xpack.sql.expression.function.scalar.datetime.DayOfMonth;
import org.elasticsearch.xpack.sql.expression.function.scalar.datetime.DayOfWeek;
Expand Down Expand Up @@ -104,8 +105,8 @@
import org.elasticsearch.xpack.sql.expression.predicate.conditional.Case;
import org.elasticsearch.xpack.sql.expression.predicate.conditional.Coalesce;
import org.elasticsearch.xpack.sql.expression.predicate.conditional.Greatest;
import org.elasticsearch.xpack.sql.expression.predicate.conditional.Iif;
import org.elasticsearch.xpack.sql.expression.predicate.conditional.IfNull;
import org.elasticsearch.xpack.sql.expression.predicate.conditional.Iif;
import org.elasticsearch.xpack.sql.expression.predicate.conditional.Least;
import org.elasticsearch.xpack.sql.expression.predicate.conditional.NullIf;
import org.elasticsearch.xpack.sql.expression.predicate.operator.arithmetic.Mod;
Expand Down Expand Up @@ -193,6 +194,7 @@ private void defineDefaultFunctions() {
def(DayOfMonth.class, DayOfMonth::new, "DAY_OF_MONTH", "DAYOFMONTH", "DAY", "DOM"),
def(DayOfWeek.class, DayOfWeek::new, "DAY_OF_WEEK", "DAYOFWEEK", "DOW"),
def(DayOfYear.class, DayOfYear::new, "DAY_OF_YEAR", "DAYOFYEAR", "DOY"),
def(DateTrunc.class, DateTrunc::new, "DATE_TRUNC"),
def(HourOfDay.class, HourOfDay::new, "HOUR_OF_DAY", "HOUR"),
def(IsoDayOfWeek.class, IsoDayOfWeek::new, "ISO_DAY_OF_WEEK", "ISODAYOFWEEK", "ISODOW", "IDOW"),
def(IsoWeekOfYear.class, IsoWeekOfYear::new, "ISO_WEEK_OF_YEAR", "ISOWEEKOFYEAR", "ISOWEEK", "IWOY", "IW"),
Expand Down
Loading