Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ES|QL bit_length function #115792

Merged
merged 32 commits into from
Nov 7, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
6f59227
Add ES|QL BIT_LENGTH support.
timgrein Oct 28, 2024
3de0143
Update docs/changelog/115792.yaml
timgrein Oct 28, 2024
7b69556
Use "str" as typed data name in BitLengthTests
timgrein Oct 28, 2024
a5da6c4
Merge remote-tracking branch 'origin/esql-string-bit-length-function'…
timgrein Oct 28, 2024
7a2ecf9
Update number of functions in 60_usage.yml
timgrein Oct 28, 2024
cbcd332
Merge branch 'main' into esql-string-bit-length-function
elasticmachine Oct 28, 2024
2621b3c
Merge branch 'main' into esql-string-bit-length-function
elasticmachine Oct 29, 2024
ae82768
Merge branch 'main' into esql-string-bit-length-function
elasticmachine Oct 29, 2024
2583805
Add FN_BIT_LENGTH capability and guard csv tests accordingly
timgrein Oct 29, 2024
4626195
Merge remote-tracking branch 'origin/esql-string-bit-length-function'…
timgrein Oct 29, 2024
d6e1c66
Use new stream reading/writing approach
timgrein Oct 29, 2024
a27f94a
Use Byte.SIZE instead of magic number
timgrein Oct 29, 2024
7a94caa
Tryout remote
timgrein Oct 30, 2024
0a4130c
Use Byte.SIZE instead of magic number
timgrein Oct 30, 2024
666198e
Remove FN_GREATEST capability and only use the most recent in bitLeng…
timgrein Oct 30, 2024
db7931a
Use TestCaseSupplier.stringCases as test cases
timgrein Oct 30, 2024
a0d2aee
Merge branch 'main' into esql-string-bit-length-function
elasticmachine Oct 30, 2024
b800d4f
Update x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/…
timgrein Oct 30, 2024
db02eb7
Add bit length capability to usage integration tests
timgrein Oct 30, 2024
3cdde43
Catch ArithmeticException in Evaluator and turn expression result int…
timgrein Oct 30, 2024
f4c0abe
Use stringTypes() instead of manually constructing the data type array
timgrein Oct 30, 2024
a9cfaf0
Use ternary operator for type resolution
timgrein Oct 30, 2024
762be47
Only use most recent capability in usage tests
timgrein Oct 30, 2024
9c9511b
Ignore ESQL usage output tests on older branches as checking the exac…
timgrein Oct 31, 2024
6dbadf2
Keep ignore old test for now
timgrein Oct 31, 2024
b88922e
Fix expected function number for current main
timgrein Oct 31, 2024
d868e08
Reintroduce snapshot capabilities into usage tests
timgrein Oct 31, 2024
989c770
Merge branch 'main' into esql-string-bit-length-function
elasticmachine Oct 31, 2024
b36fa7b
Merge branch 'main' into esql-string-bit-length-function
elasticmachine Nov 6, 2024
c708715
Add required_capability: fn_bit_length to docs.csv-spec
timgrein Nov 6, 2024
7f480bd
Merge remote-tracking branch 'origin/esql-string-bit-length-function'…
timgrein Nov 6, 2024
28ad3e9
Merge branch 'main' into esql-string-bit-length-function
elasticmachine Nov 6, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions docs/changelog/115792.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
pr: 115792
summary: Add ES|QL `bit_length` function
area: ES|QL
type: enhancement
issues: []
5 changes: 5 additions & 0 deletions docs/reference/esql/functions/description/bit_length.asciidoc

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

13 changes: 13 additions & 0 deletions docs/reference/esql/functions/examples/bit_length.asciidoc

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

12 changes: 12 additions & 0 deletions docs/reference/esql/functions/kibana/docs/bit_length.md

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

15 changes: 15 additions & 0 deletions docs/reference/esql/functions/layout/bit_length.asciidoc

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 6 additions & 0 deletions docs/reference/esql/functions/parameters/bit_length.asciidoc

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions docs/reference/esql/functions/signature/bit_length.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 2 additions & 0 deletions docs/reference/esql/functions/string-functions.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
{esql} supports these string functions:

// tag::string_list[]
* <<esql-bit_length>>
* <<esql-concat>>
* <<esql-ends_with>>
* <<esql-from_base64>>
Expand All @@ -30,6 +31,7 @@
* <<esql-trim>>
// end::string_list[]

include::layout/bit_length.asciidoc[]
include::layout/concat.asciidoc[]
include::layout/ends_with.asciidoc[]
include::layout/from_base64.asciidoc[]
Expand Down
10 changes: 10 additions & 0 deletions docs/reference/esql/functions/types/bit_length.asciidoc

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 3 additions & 1 deletion x-pack/plugin/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -82,8 +82,10 @@ tasks.named("precommit").configure {

tasks.named("yamlRestCompatTestTransform").configure({ task ->
task.skipTest("security/10_forbidden/Test bulk response with invalid credentials", "warning does not exist for compatibility")
task.skipTest("inference/inference_crud/Test get all", "Assertions on number of inference models break due to default configs")
task.skipTest("esql/60_usage/Basic ESQL usage output (telemetry)", "The telemetry output changed. We dropped a column. That's safe.")
task.skipTest("inference/inference_crud/Test get all", "Assertions on number of inference models break due to default configs")
task.skipTest("esql/60_usage/Basic ESQL usage output (telemetry) snapshot version", "The number of functions is constantly increasing")
task.skipTest("esql/60_usage/Basic ESQL usage output (telemetry) non-snapshot version", "The number of functions is constantly increasing")
task.skipTest("esql/80_text/reverse text", "The output type changed from TEXT to KEYWORD.")
task.skipTest("esql/80_text/values function", "The output type changed from TEXT to KEYWORD.")
})
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -656,3 +656,22 @@ FROM sample_data

@timestamp:date | client_ip:ip | event_duration:long | message:keyword
;

docsBitLength
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you need required_capability: fn_bit_length.

required_capability: fn_bit_length
// tag::bitLength[]
FROM employees
| KEEP first_name, last_name
| EVAL fn_bit_length = BIT_LENGTH(first_name)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: could add a fn_length = LENGTH(first_name) for a quick /8 (reviewing) check, but can stay as is too.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As these are docs examples it's probably fine to let it solely be focused on bit_length IMHO.

// end::bitLength[]
| SORT first_name
| LIMIT 3
;

// tag::bitLength-result[]
first_name:keyword | last_name:keyword | fn_bit_length:integer
Alejandro |McAlpine |72
Amabile |Gomatam |56
Anneke |Preusig |48
// end::bitLength-result[]
;
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,56 @@ emp_no:integer | l:integer
10003 | 5
;

bitLength
required_capability: fn_bit_length
row a = "hello", b = "" | eval y = bit_length(a) + bit_length(b);

a:keyword | b:keyword | y:integer
hello | | 40
;

bitLengthWithNonAsciiChars
required_capability: fn_bit_length
row a = "¡", b = "❗️" | eval y = bit_length(a) | eval z = bit_length(b);

a:keyword | b:keyword | y:integer | z:integer
¡ | ❗️ | 16 | 48
;

foldBitLength
required_capability: fn_bit_length
row a = 1 | eval b = bit_length("hello");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: could be row b = bit_length("hello"), but can stay as is too.


a:integer | b:integer
1 | 40
;

bitLengthAndSourceQuoting
required_capability: fn_bit_length
from "employees" | sort emp_no | limit 3 | eval l = bit_length(first_name) | keep emp_no, l;

emp_no:integer | l:integer
10001 | 48
10002 | 56
10003 | 40
;

bitLengthInsideOtherFunction
required_capability: fn_bit_length
row a = "abc", b = "de" | eval g = greatest(bit_length(a), bit_length(b), bit_length("fghi"));

a:keyword | b:keyword | g:integer
abc | de | 32
;

bitLengthNull
required_capability: fn_bit_length
row a = "abc" | eval l = bit_length(null);

a:string | l:integer
abc | null
;

startsWithConstant
from employees | sort emp_no | limit 10 | eval f_S = starts_with(first_name, "S") | keep emp_no, first_name, f_S;

Expand Down

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,12 @@
*/
public class EsqlCapabilities {
public enum Cap {

/**
* Support for function {@code BIT_LENGTH}. Done in #115792
*/
FN_BIT_LENGTH,

/**
* Support for function {@code REVERSE}.
*/
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,7 @@
import org.elasticsearch.xpack.esql.expression.function.scalar.spatial.StDistance;
import org.elasticsearch.xpack.esql.expression.function.scalar.spatial.StX;
import org.elasticsearch.xpack.esql.expression.function.scalar.spatial.StY;
import org.elasticsearch.xpack.esql.expression.function.scalar.string.BitLength;
import org.elasticsearch.xpack.esql.expression.function.scalar.string.Concat;
import org.elasticsearch.xpack.esql.expression.function.scalar.string.EndsWith;
import org.elasticsearch.xpack.esql.expression.function.scalar.string.LTrim;
Expand Down Expand Up @@ -305,6 +306,7 @@ private FunctionDefinition[][] functions() {
def(Tau.class, Tau::new, "tau") },
// string
new FunctionDefinition[] {
def(BitLength.class, BitLength::new, "bit_length"),
def(Concat.class, Concat::new, "concat"),
def(EndsWith.class, EndsWith::new, "ends_with"),
def(LTrim.class, LTrim::new, "ltrim"),
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@
import org.elasticsearch.xpack.esql.expression.function.scalar.math.Tau;
import org.elasticsearch.xpack.esql.expression.function.scalar.nulls.Coalesce;
import org.elasticsearch.xpack.esql.expression.function.scalar.spatial.BinarySpatialFunction;
import org.elasticsearch.xpack.esql.expression.function.scalar.string.BitLength;
import org.elasticsearch.xpack.esql.expression.function.scalar.string.Concat;
import org.elasticsearch.xpack.esql.expression.function.scalar.string.EndsWith;
import org.elasticsearch.xpack.esql.expression.function.scalar.string.Left;
Expand Down Expand Up @@ -74,6 +75,7 @@ public static List<NamedWriteableRegistry.Entry> getNamedWriteables() {
List<NamedWriteableRegistry.Entry> entries = new ArrayList<>();
entries.add(And.ENTRY);
entries.add(Atan2.ENTRY);
entries.add(BitLength.ENTRY);
entries.add(Bucket.ENTRY);
entries.add(Case.ENTRY);
entries.add(Categorize.ENTRY);
Expand Down
Loading