Skip to content

Commit

Permalink
updated docs and addressed review comments
Browse files Browse the repository at this point in the history
  • Loading branch information
fang-xing-esql committed Nov 5, 2024
1 parent 4c39d94 commit a8b1b0f
Show file tree
Hide file tree
Showing 6 changed files with 50 additions and 36 deletions.
40 changes: 26 additions & 14 deletions docs/reference/esql/implicit-casting.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<titleabbrev>Implicit casting</titleabbrev>
++++

Often users will input `datetime`, `ip`, `version`, or geospatial objects as simple strings in their queries for use in predicates, functions, or expressions. {esql} provides <<esql-type-conversion-functions, type conversion functions>> to explicitly convert these strings into the desired data types.
Often users will input `date`, `ip`, `version`, `date_period` or `time_duration` as simple strings in their queries for use in predicates, functions, or expressions. {esql} provides <<esql-type-conversion-functions, type conversion functions>> to explicitly convert these strings into the desired data types.

Without implicit casting users must explicitly code these `to_X` functions in their queries, when string literals don't match the target data types they are assigned or compared to. Here is an example of using `to_datetime` to explicitly perform a data type conversion.

Expand All @@ -18,7 +18,7 @@ FROM employees
| LIMIT 1
----

Implicit casting improves usability, by automatically converting string literals to the target data type. This is most useful when the target data type is `datetime`, `ip`, `version` or a geo spatial. It is natural to specify these as a string in queries.
Implicit casting improves usability, by automatically converting string literals to the target data type. This is most useful when the target data type is `date`, `ip`, `version`, `date_period` or `time_duration`. It is natural to specify these as a string in queries.

The first query can be coded without calling the `to_datetime` function, as follows:

Expand All @@ -38,16 +38,28 @@ The following table details which {esql} operations support implicit casting for

[%header.monospaced.styled,format=dsv,separator=|]
|===
||ScalarFunction|BinaryComparison|ArithmeticOperation|InListPredicate|AggregateFunction
|DATETIME|Y|Y|Y|Y|N
|DOUBLE|Y|N|N|N|N
|LONG|Y|N|N|N|N
|INTEGER|Y|N|N|N|N
|IP|Y|Y|Y|Y|N
|VERSION|Y|Y|Y|Y|N
|GEO_POINT|Y|N|N|N|N
|GEO_SHAPE|Y|N|N|N|N
|CARTESIAN_POINT|Y|N|N|N|N
|CARTESIAN_SHAPE|Y|N|N|N|N
|BOOLEAN|Y|Y|Y|Y|N
||ScalarFunction*|Operator*|<<esql-group-functions, GroupingFunction>>|<<esql-agg-functions, AggregateFunction>>
|DATE|Y|Y|Y|N
|IP|Y|Y|Y|N
|VERSION|Y|Y|Y|N
|BOOLEAN|Y|Y|Y|N
|DATE_PERIOD/TIME_DURATION|Y|N|Y|N
|===

ScalarFunction* includes:

<<esql-conditional-functions-and-expressions, Conditional Functions and Expressions>>

<<esql-date-time-functions, Date and Time Functions>>

<<esql-ip-functions, IP Functions>>


Operator* includes:

<<esql-binary-operators, Binary Operators>>

<<esql-unary-operators, Unary Operator>>

<<esql-in-operator, IN>>

Original file line number Diff line number Diff line change
Expand Up @@ -968,6 +968,8 @@ private BitSet gatherPreAnalysisMetrics(LogicalPlan plan, BitSet b) {
* date = "2024-08-21" + 3 days
* ip == "127.0.0.1"
* version != "1.0"
* bucket(dateField, "1 month")
* date_trunc("1 minute", dateField)
*
* If the inputs to Coalesce are mixed numeric types, cast the rest of the numeric field or value to the first numeric data type if
* applicable. For example, implicit casting converts:
Expand All @@ -992,15 +994,15 @@ private static Expression cast(org.elasticsearch.xpack.esql.core.expression.func
return processIn(in);
}
if (f instanceof EsqlScalarFunction || f instanceof GroupingFunction) { // exclude AggregateFunction until it is needed
return processFunction(f, registry);
return processScalarOrGroupingFunction(f, registry);
}
if (f instanceof EsqlArithmeticOperation || f instanceof BinaryComparison) {
return processBinaryOperator((BinaryOperator) f);
}
return f;
}

private static Expression processFunction(
private static Expression processScalarOrGroupingFunction(
org.elasticsearch.xpack.esql.core.expression.function.Function f,
EsqlFunctionRegistry registry
) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -473,7 +473,7 @@ public List<String> argDescriptions() {
}

/**
* This is called by ImplicitCasting to convert string literals to a target data type.
* Build a list target data types, which is used by ImplicitCasting to convert string literals to a target data type.
*/
public static DataType getTargetType(String[] names) {
List<DataType> types = new ArrayList<>();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -274,11 +274,11 @@ public static TemporalAmount parseTemporalAmount(Object val, DataType expectedTy
return null;
}
StringBuilder value = new StringBuilder();
StringBuilder qualifier = new StringBuilder();
separateValueAndQualifierForTemporalAmount(str.strip(), value, qualifier, errorMessage, expectedType.toString());
if ((value.isEmpty() || qualifier.isEmpty()) == false) {
StringBuilder temporalUnit = new StringBuilder();
separateValueAndTemporalUnitForTemporalAmount(str.strip(), value, temporalUnit, errorMessage, expectedType.toString());
if ((value.isEmpty() || temporalUnit.isEmpty()) == false) {
try {
TemporalAmount result = parseTemporalAmount(Integer.parseInt(value.toString()), qualifier.toString(), Source.EMPTY);
TemporalAmount result = parseTemporalAmount(Integer.parseInt(value.toString()), temporalUnit.toString(), Source.EMPTY);
if (DataType.DATE_PERIOD == expectedType && result instanceof Period
|| DataType.TIME_DURATION == expectedType && result instanceof Duration) {
return result;
Expand All @@ -297,26 +297,26 @@ public static TemporalAmount parseTemporalAmount(Object val, DataType expectedTy
}

public static TemporalAmount maybeParseTemporalAmount(String str) {
// The string literal can be either Date_Period or Time_Duration, derive the data type from its qualifier
// The string literal can be either Date_Period or Time_Duration, derive the data type from its temporal unit
String errorMessage = "Cannot parse [{}] to {}";
String expectedTypes = DATE_PERIOD + " or " + TIME_DURATION;
StringBuilder value = new StringBuilder();
StringBuilder qualifier = new StringBuilder();
separateValueAndQualifierForTemporalAmount(str, value, qualifier, errorMessage, expectedTypes);
if ((value.isEmpty() || qualifier.isEmpty()) == false) {
StringBuilder temporalUnit = new StringBuilder();
separateValueAndTemporalUnitForTemporalAmount(str, value, temporalUnit, errorMessage, expectedTypes);
if ((value.isEmpty() || temporalUnit.isEmpty()) == false) {
try {
return parseTemporalAmount(Integer.parseInt(value.toString()), qualifier.toString(), Source.EMPTY);
return parseTemporalAmount(Integer.parseInt(value.toString()), temporalUnit.toString(), Source.EMPTY);
} catch (NumberFormatException ex) {
throw new ParsingException(Source.EMPTY, errorMessage, str, expectedTypes);
}
}
return null;
}

private static void separateValueAndQualifierForTemporalAmount(
private static void separateValueAndTemporalUnitForTemporalAmount(
String temporalAmount,
StringBuilder value,
StringBuilder qualifier,
StringBuilder temporalUnit,
String errorMessage,
String expectedType
) {
Expand All @@ -325,7 +325,7 @@ private static void separateValueAndQualifierForTemporalAmount(
for (char c : temporalAmount.toCharArray()) {
if (c == ' ') {
if (lastWasSpace == false) {
nextBuffer = nextBuffer == value ? qualifier : null;
nextBuffer = nextBuffer == value ? temporalUnit : null;
}
lastWasSpace = true;
continue;
Expand Down Expand Up @@ -420,10 +420,10 @@ public static DataType commonType(DataType left, DataType right) {
}

// generally supporting abbreviations from https://en.wikipedia.org/wiki/Unit_of_time
public static TemporalAmount parseTemporalAmount(Number value, String qualifier, Source source) throws InvalidArgumentException,
public static TemporalAmount parseTemporalAmount(Number value, String temporalUnit, Source source) throws InvalidArgumentException,
ArithmeticException, ParsingException {
try {
return switch (INTERVALS.valueOf(qualifier.toUpperCase(Locale.ROOT))) {
return switch (INTERVALS.valueOf(temporalUnit.toUpperCase(Locale.ROOT))) {
case MILLISECOND, MILLISECONDS, MS -> Duration.ofMillis(safeToLong(value));
case SECOND, SECONDS, SEC, S -> Duration.ofSeconds(safeToLong(value));
case MINUTE, MINUTES, MIN -> Duration.ofMinutes(safeToLong(value));
Expand All @@ -436,7 +436,7 @@ public static TemporalAmount parseTemporalAmount(Number value, String qualifier,
case YEAR, YEARS, YR, Y -> Period.ofYears(safeToInt(safeToLong(value)));
};
} catch (IllegalArgumentException e) {
throw new ParsingException(source, "Unexpected time interval qualifier: '{}'", qualifier);
throw new ParsingException(source, "Unexpected temporal unit: '{}'", temporalUnit);
}
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1578,15 +1578,15 @@ public void testIntervalAsString() {

// Bucket
assertEquals(
"1:52: Cannot convert string [1 yar] to [DATE_PERIOD or TIME_DURATION], error [Unexpected time interval qualifier: 'yar']",
"1:52: Cannot convert string [1 yar] to [DATE_PERIOD or TIME_DURATION], error [Unexpected temporal unit: 'yar']",
error("from test | stats max(emp_no) by bucket(hire_date, \"1 yar\")")
);
assertEquals(
"1:52: Cannot convert string [1 hur] to [DATE_PERIOD or TIME_DURATION], error [Unexpected time interval qualifier: 'hur']",
"1:52: Cannot convert string [1 hur] to [DATE_PERIOD or TIME_DURATION], error [Unexpected temporal unit: 'hur']",
error("from test | stats max(emp_no) by bucket(hire_date, \"1 hur\")")
);
assertEquals(
"1:58: Cannot convert string [1 mu] to [DATE_PERIOD or TIME_DURATION], error [Unexpected time interval qualifier: 'mu']",
"1:58: Cannot convert string [1 mu] to [DATE_PERIOD or TIME_DURATION], error [Unexpected temporal unit: 'mu']",
error("from test | stats max = max(emp_no) by bucket(hire_date, \"1 mu\") | sort max ")
);
assertEquals(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -431,7 +431,7 @@ public void testDatePeriodLiterals() {
}

public void testUnknownNumericQualifier() {
assertParsingException(() -> whereExpression("1 decade"), "Unexpected time interval qualifier: 'decade'");
assertParsingException(() -> whereExpression("1 decade"), "Unexpected temporal unit: 'decade'");
}

public void testQualifiedDecimalLiteral() {
Expand Down

0 comments on commit a8b1b0f

Please sign in to comment.