Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add implementation of now, sysdate, localtime and similar functions #92

Merged
merged 18 commits into from
Aug 12, 2022

Conversation

Yury-Fridlyand
Copy link

@Yury-Fridlyand Yury-Fridlyand commented Jul 21, 2022

Signed-off-by: Yury Fridlyand [email protected]

Updated, please re-read

Description

New functions added:

  • now()
  • current_timestamp()
  • localtimestamp()
  • localtime()
  • sysdate()
  • curtime()
  • current_time()
  • curdate()
  • current_date()

New signatures:

  • now() : datetime
  • now(int) : datetime
  • current_timestamp() : datetime
  • current_timestamp(int) : datetime
  • current_timestamp : datetime
  • localtimestamp() : datetime
  • localtimestamp(int) : datetime
  • localtimestamp : datetime
  • localtime() : datetime
  • localtime(int) : datetime
  • localtime : datetime
  • sysdate() : datetime
  • sysdate(int) : datetime
  • curtime() : time
  • curtime(int) : time
  • current_time() : time
  • current_time : time
  • current_time(int) : time
  • curdate() : date
  • current_date() : date
  • current_date : date

If the argument is given it specifies a fractional seconds precision from 0 to 6, the return value includes a fractional seconds part of that many digits. Default precision: 6.
Code updated to always fill padding by zeroes up to 6 to simplify further parsing. Note: double values have no pad filling and might be rounded.

Added automatic type cast rules:

  • datetime -> double
  • time -> double
  • date -> int

So all functions could be used in math operations like
now() + 0

UPD
Reverted, because this affected other functionality.

Tests

SQL

select now(), now(0), now(4), current_timestamp(), current_timestamp(0), current_timestamp(5), localtimestamp(), localtimestamp(1), localtimestamp(2), localtime(), localtime(0), sysdate(), sysdate(3), curdate(), curtime(), curtime(3), curtime(0), current_time(), current_date(), CURRENT_DATE, CURRENT_TIME, CURRENT_TIMESTAMP, LOCALTIME, LOCALTIMESTAMP;
fetched rows / total rows = 1/1
-[ RECORD 1 ]-------------------------
now()                   | 2022-07-20 20:15:25.262409
now(0)                  | 2022-07-20 20:15:25.000000
now(4)                  | 2022-07-20 20:15:25.262400
current_timestamp()     | 2022-07-20 20:15:25.262409
current_timestamp(0)    | 2022-07-20 20:15:25.000000
current_timestamp(5)    | 2022-07-20 20:15:25.262400
localtimestamp()        | 2022-07-20 20:15:25.262409
localtimestamp(1)       | 2022-07-20 20:15:25.200000
localtimestamp(2)       | 2022-07-20 20:15:25.260000
localtime()             | 2022-07-20 20:15:25.262409
localtime(0)            | 2022-07-20 20:15:25.000000
sysdate()               | 2022-07-20 20:15:26.468210
sysdate(3)              | 2022-07-20 20:15:26.468000
curdate()               | 2022-07-20
curtime()               | 20:15:26.469603
curtime(3)              | 20:15:26.469000
curtime(0)              | 20:15:26.000000
current_time()          | 20:15:26.469746
current_date()          | 2022-07-20
CURRENT_DATE            | 2022-07-20
CURRENT_TIME            | 20:15:26.469869
CURRENT_TIMESTAMP       | 2022-07-20 20:15:25.262409
LOCALTIME               | 2022-07-20 20:15:25.262409
LOCALTIMESTAMP          | 2022-07-20 20:15:25.262409

PPL

source=calcs | top 1 zzz | eval `now()` = now(), `now(0)` = now(0), `now(4)` = now(4), `current_timestamp()` = current_timestamp(), `current_timestamp(0)` = current_timestamp(0), `current_timestamp(5)` = current_timestamp(5), `localtimestamp()` = localtimestamp(), `localtimestamp(1)` = localtimestamp(1), `localtimestamp(2)` = localtimestamp(2), `localtime()` = localtime(), `localtime(0)` = localtime(0), `sysdate()` = sysdate(), `sysdate(3)` = sysdate(3), `curdate()` = curdate(), `curtime()` = curtime(), `curtime(3)` = curtime(3), `curtime(0)` = curtime(0), `current_time()` = current_time(), `current_date()` = current_date(), `CURRENT_DATE` = CURRENT_DATE, `CURRENT_TIME` = CURRENT_TIME, `CURRENT_TIMESTAMP` = CURRENT_TIMESTAMP, `LOCALTIME` = LOCALTIME, `LOCALTIMESTAMP` = LOCALTIMESTAMP;
fetched rows / total rows = 1/1
-[ RECORD 1 ]-------------------------
zzz                     | f
now()                   | 2022-07-20 20:17:22.451338
now(0)                  | 2022-07-20 20:17:22.000000
now(4)                  | 2022-07-20 20:17:22.451300
current_timestamp()     | 2022-07-20 20:17:22.451338
current_timestamp(0)    | 2022-07-20 20:17:22.000000
current_timestamp(5)    | 2022-07-20 20:17:22.451330
localtimestamp()        | 2022-07-20 20:17:22.451338
localtimestamp(1)       | 2022-07-20 20:17:22.400000
localtimestamp(2)       | 2022-07-20 20:17:22.450000
localtime()             | 2022-07-20 20:17:22.451338
localtime(0)            | 2022-07-20 20:17:22.000000
sysdate()               | 2022-07-20 20:17:22.760505
sysdate(3)              | 2022-07-20 20:17:22.760000
curdate()               | 2022-07-20
curtime()               | 20:17:22.760584
curtime(3)              | 20:17:22.760000
curtime(0)              | 20:17:22.000000
current_time()          | 20:17:22.760626
current_date()          | 2022-07-20
CURRENT_DATE            | 2022-07-20
CURRENT_TIME            | 20:17:22.760693
CURRENT_TIMESTAMP       | 2022-07-20 20:17:22.451338
LOCALTIME               | 2022-07-20 20:17:22.451338
LOCALTIMESTAMP          | 2022-07-20 20:17:22.451338

Implementation details

NOW and SYSDATE difference
According to MySQL standard

NOW() returns a constant time that indicates the time at which the statement began to execute.

SYSDATE() returns the time at which it executes.

I had to save and then to pick time when plugin receives a query. LogUtils was chosen to store this unless a better candidate introduced.

Two LogUtils
There are LogUtils classes that do the same:
common/src/main/java/org/opensearch/sql/common/utils/LogUtils.java
legacy/src/main/java/org/opensearch/sql/legacy/utils/LogUtils.java
First one is used in PPL entry point and accessible in core.
Second one is used in SQL entry point and not available in core, because core depends on legacy. Adding access to legacy/.../LogUtils in core creates a circular dependency.
I had to duplicate code in both LogUtils until duplicate classes are removed in scope of another task.

UPD
common...LogUtils was renamed to QueryContext.
legacy...LogUtils was deleted. All its call were redirected to QueryContext.

Automatic cast
These changes include

  1. Reorder in ExprCoreType
  2. New widening rules added (listed) in ExprCoreType
  3. Cast rules implemented in TypeCastOperator
  4. Cast rules are also implemented in LuceneQuery *

*) Kind of duplicated code, no idea where it is used

See also WideningTypeRule.java and FunctionResolver.java.

UPD
Reverted in 9f6b66c.

Parser update

  1. datetimeConstantLiteral was added to SQL and PPL parsers. It lists pseudo-constants which actually invoke corresponding functions without (). So CURRENT_DATE is shortcut for CURRENT_DATE() and so on.
  2. AstExpressionBuilder were updated to call a function when datetimeConstantLiteral met.
  3. AstExpressionBuilder in PPL is updated a bit by adding visitFunction method and removing some duplicated code. Copied from AstExpressionBuilder in SQL.
  4. ANTLR rules were copied from SQL to PPL, so PPL got dateLiteral, timeLiteral and timestampLiteral which are not used yet. .

Zero padding
ExprDatetimeValue and ExprTmeValue are updated to fill blank space by zeroes when serialized into json.
Could be reverted if found odd.

UPD
Reverted.

Doctests
Doctests for new functions always fail, because test validate exact string match for result. I added docs, but disabled doctests for them.

See also

Issues Resolved

opensearch-project#46
opensearch-project#260

TODOs

  • Update docs - see TypeConversion.md and add doctests
  • Add unit tests for new functions
  • Add unit tests for implicit cast
  • Add integration tests
  • Fix failing test
  • Update docs on website

Check List

  • New functionality includes testing.
    • All tests pass, including unit test, integration test and doctest
  • New functionality has been documented.
    • New functionality has javadoc added
    • New functionality has user manual doc added
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Comment on lines 21 to 33
import java.math.BigDecimal;
import java.math.RoundingMode;
import java.time.LocalDate;
import java.time.LocalDateTime;
import java.time.ZoneId;
import java.time.ZoneOffset;
import java.time.format.TextStyle;
import java.time.temporal.ChronoUnit;
import java.util.Locale;
import java.util.Optional;
import java.util.concurrent.TimeUnit;
import lombok.experimental.UtilityClass;
import org.opensearch.sql.common.utils.LogUtils;
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove unused imports

Comment on lines 95 to 97
UTC_DATE(FunctionName.of("UTC_DATE")),
UTC_TIME(FunctionName.of("UTC_TIME")),
UTC_TIMESTAMP(FunctionName.of("UTC_TIMESTAMP")),
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fix case

Comment on lines 165 to 183
private FunctionResolver utc_timestamp() {
return define(BuiltinFunctionName.UTC_TIMESTAMP.getName(),
impl(() -> new ExprDatetimeValue(sysDate(Optional.empty()).atZone(ZoneId.of("UTC")).toLocalDateTime()), DATETIME),
impl((v) -> new ExprDatetimeValue(sysDate(Optional.of(v.integerValue())).atZone(ZoneId.of("UTC")).toLocalDateTime()), DATETIME, INTEGER)
);
}

private FunctionResolver utc_time() {
return define(BuiltinFunctionName.UTC_TIME.getName(),
impl(() -> new ExprTimeValue(sysDate(Optional.empty()).atZone(ZoneId.of("UTC")).toLocalTime()), TIME),
impl((v) -> new ExprTimeValue(sysDate(Optional.of(v.integerValue())).atZone(ZoneId.of("UTC")).toLocalTime()), TIME, INTEGER)
);
}

private FunctionResolver utc_date() {
return define(BuiltinFunctionName.UTC_DATE.getName(),
impl(() -> new ExprDateValue(sysDate(Optional.empty()).atZone(ZoneId.of("UTC")).toLocalDate()), DATE),
impl((v) -> new ExprDateValue(sysDate(Optional.of(v.integerValue())).atZone(ZoneId.of("UTC")).toLocalDate()), DATE, INTEGER)
);
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fix UTC functions, they return local time

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some of these functions are covered by other stories. Maybe you should wait to add them?

@@ -42,6 +49,14 @@ public static String getRequestId() {
return requestId;
}

public static void recordProcessingStarted() {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LogUtils is not appropriate container for this data.

A better approach is capture time when SQLService is created.

org/opensearch/sql/legacy/plugin/RestSQLQueryAction.java:99 is a good starting point.

cc @acarbonetto

Copy link

@acarbonetto acarbonetto Jul 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LogUtils was chosen because it currently houses the requestId, which shares a similar responsibility.
I was going to suggest that we refactor all of this code (and remove the LogUtils class)

We want to choose the earliest available moment to create the start time, but I would have assumed SQLService is created when at application startup, not for each request. But then, I haven't looked at the class yet.

return String.format("%s %s", DateTimeFormatter.ISO_DATE.format(datetime),
DateTimeFormatter.ISO_TIME.format((datetime.getNano() == 0)
? datetime.truncatedTo(ChronoUnit.SECONDS) : datetime));
return datetime.format(FORMATTER_ZERO_PAD_NANOS);

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the new datetime formatter same as old formatter?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably not.
@Yury-Fridlyand we do need to consider backwards compatibility too. Make sure we aren't breaking anything 😬

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as ExprTimeValue formatter.

@@ -53,7 +64,7 @@ public ExprTimeValue(String time) {

@Override
public String value() {
return DateTimeFormatter.ISO_LOCAL_TIME.format(time);
return time.format(FORMATTER_ZERO_PAD_NANOS);

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like the new formatter produces different results from the old.

Is that correct? If so, why change it?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change was an experiment. Makes second fraction always printed with precision 6 digits with zero padding.
It could be helpful for cases when user wants to parse DateTIme.

* NOW() returns a constant time that indicates the time at which the statement began to execute
* @return ExprValue that contains LocalDateTime object
*/
private ExprValue exprNow(Optional<Integer> fsp) {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does fsp stand for?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fractional seconds precision https://database.guide/now-examples-mysql/ - But we shouldn't assume the user knows this.
Please add fsp as a @param in the javadoc

}

private Function visitFunction(String functionName, FunctionArgsContext args) {
if (args == null) {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd change this to one return statement and use ?: to construct the funcArgs argument.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Copied from SQL AST builder, so I will update both.

return String.format("%s %s", DateTimeFormatter.ISO_DATE.format(datetime),
DateTimeFormatter.ISO_TIME.format((datetime.getNano() == 0)
? datetime.truncatedTo(ChronoUnit.SECONDS) : datetime));
return datetime.format(FORMATTER_ZERO_PAD_NANOS);

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably not.
@Yury-Fridlyand we do need to consider backwards compatibility too. Make sure we aren't breaking anything 😬

private static final int MIN_FRACTION_SECONDS = 0;
private static final int MAX_FRACTION_SECONDS = 9;
private static final int FRACTION_SECONDS_TO_PRINT = 6;

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow naming conventions (or fix the other conventions if necessary. I think we should call this: MAX_FRACTION_SECONDS_TO_PRINT

Comment on lines 165 to 183
private FunctionResolver utc_timestamp() {
return define(BuiltinFunctionName.UTC_TIMESTAMP.getName(),
impl(() -> new ExprDatetimeValue(sysDate(Optional.empty()).atZone(ZoneId.of("UTC")).toLocalDateTime()), DATETIME),
impl((v) -> new ExprDatetimeValue(sysDate(Optional.of(v.integerValue())).atZone(ZoneId.of("UTC")).toLocalDateTime()), DATETIME, INTEGER)
);
}

private FunctionResolver utc_time() {
return define(BuiltinFunctionName.UTC_TIME.getName(),
impl(() -> new ExprTimeValue(sysDate(Optional.empty()).atZone(ZoneId.of("UTC")).toLocalTime()), TIME),
impl((v) -> new ExprTimeValue(sysDate(Optional.of(v.integerValue())).atZone(ZoneId.of("UTC")).toLocalTime()), TIME, INTEGER)
);
}

private FunctionResolver utc_date() {
return define(BuiltinFunctionName.UTC_DATE.getName(),
impl(() -> new ExprDateValue(sysDate(Optional.empty()).atZone(ZoneId.of("UTC")).toLocalDate()), DATE),
impl((v) -> new ExprDateValue(sysDate(Optional.of(v.integerValue())).atZone(ZoneId.of("UTC")).toLocalDate()), DATE, INTEGER)
);

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some of these functions are covered by other stories. Maybe you should wait to add them?

* NOW() returns a constant time that indicates the time at which the statement began to execute
* @return ExprValue that contains LocalDateTime object
*/
private ExprValue exprNow(Optional<Integer> fsp) {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fractional seconds precision https://database.guide/now-examples-mysql/ - But we shouldn't assume the user knows this.
Please add fsp as a @param in the javadoc

var res = LogUtils.getProcessingStartedTime();
if (fsp.isEmpty())
return new ExprDatetimeValue(res);
var default_precision = 9; // There are 10^9 nanoseconds in one second

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

magic number 9

if (fsp.isEmpty())
return new ExprDatetimeValue(res);
var default_precision = 9; // There are 10^9 nanoseconds in one second
if (fsp.get() < 0 || fsp.get() > 6)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

more magic numbers

@@ -679,4 +755,33 @@ private ExprValue exprYear(ExprValue date) {
return new ExprIntegerValue(date.dateValue().getYear());
}

/**
* NOW() returns a constant time that indicates the time at which the statement began to execute

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

statement or query?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link

@acarbonetto acarbonetto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM


@ParameterizedTest(name = "{1}")
@MethodSource("functionNames")
public void the_test(Function<Expression[], FunctionExpression> function,

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we change this to a more expressive name rather than "the_test"

@Yury-Fridlyand Yury-Fridlyand marked this pull request as ready for review August 3, 2022 01:49
@Yury-Fridlyand Yury-Fridlyand changed the title [WIP] Add implementation of now, sysdate, localtime and similar functions Add implementation of now, sysdate, localtime and similar functions Aug 4, 2022
@codecov
Copy link

codecov bot commented Aug 4, 2022

Codecov Report

Merging #92 (a7b2080) into integ-datetime-now (29f1c52) will increase coverage by 0.08%.
The diff coverage is 100.00%.

@@                   Coverage Diff                    @@
##             integ-datetime-now      #92      +/-   ##
========================================================
+ Coverage                 94.78%   94.87%   +0.08%     
- Complexity                 2880     2936      +56     
========================================================
  Files                       286      287       +1     
  Lines                      7735     7859     +124     
  Branches                    565      572       +7     
========================================================
+ Hits                       7332     7456     +124     
  Misses                      349      349              
  Partials                     54       54              
Flag Coverage Δ
query-workbench 62.76% <ø> (ø)
sql-engine 97.80% <100.00%> (+0.03%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...opensearch/sql/planner/logical/LogicalPlanDSL.java 100.00% <ø> (ø)
...c/main/java/org/opensearch/sql/expression/DSL.java 100.00% <100.00%> (ø)
...arch/sql/expression/datetime/DateTimeFunction.java 100.00% <100.00%> (ø)
...h/sql/expression/function/BuiltinFunctionName.java 100.00% <100.00%> (ø)
...ch/sql/opensearch/client/OpenSearchNodeClient.java 100.00% <100.00%> (ø)
...ch/sql/opensearch/client/OpenSearchRestClient.java 100.00% <100.00%> (ø)
...sql/opensearch/request/OpenSearchQueryRequest.java 100.00% <100.00%> (ø)
...l/opensearch/request/OpenSearchRequestBuilder.java 100.00% <100.00%> (ø)
...ql/opensearch/request/OpenSearchScrollRequest.java 100.00% <100.00%> (ø)
...request/system/OpenSearchDescribeIndexRequest.java 100.00% <100.00%> (ø)
... and 6 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@@ -22,9 +22,6 @@ jobs:
- name: Build with Gradle
run: ./gradlew build assemble

- name: Run backward compatibility tests
run: ./scripts/bwctest.sh

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your PR is showing commits from other work. I'd suggest that you rebase your branch to remove those commits.

Signed-off-by: Yury Fridlyand <[email protected]>
Signed-off-by: Yury Fridlyand <[email protected]>
…LogUtils`. Optimize `now` and `sysdate` calls.

Signed-off-by: Yury Fridlyand <[email protected]>
Signed-off-by: Yury Fridlyand <[email protected]>
Signed-off-by: Yury Fridlyand <[email protected]>
Signed-off-by: Yury Fridlyand <[email protected]>
Signed-off-by: Yury Fridlyand <[email protected]>
Signed-off-by: Yury Fridlyand <[email protected]>
Signed-off-by: Yury Fridlyand <[email protected]>
Signed-off-by: Yury Fridlyand <[email protected]>
Signed-off-by: Yury Fridlyand <[email protected]>
@Yury-Fridlyand
Copy link
Author

Yury-Fridlyand commented Aug 12, 2022

Waiting for QueryContext PR opensearch-project#747 to be merged on upstream...

@Yury-Fridlyand Yury-Fridlyand merged commit 86ace08 into integ-datetime-now Aug 12, 2022
@Yury-Fridlyand Yury-Fridlyand deleted the dev-datetime-now branch August 12, 2022 18:38
Yury-Fridlyand added a commit that referenced this pull request Aug 17, 2022
Yury-Fridlyand added a commit that referenced this pull request Aug 17, 2022
Yury-Fridlyand added a commit that referenced this pull request Sep 15, 2022
Yury-Fridlyand added a commit that referenced this pull request Sep 24, 2022
…ions (opensearch-project#754)

* Add implementation of `now`, `sysdate`, `localtime` and similar functions (#92)

Signed-off-by: Yury-Fridlyand <[email protected]>

* Rework on `now` function implementation (#113)

Signed-off-by: Yury-Fridlyand <[email protected]>

* Minor SQL ANTLR clean-up.

Signed-off-by: Yury-Fridlyand <[email protected]>

Signed-off-by: Yury-Fridlyand <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants