ES|QL kql function. #116764

afoucret · 2024-11-13T17:57:07Z

This PR introduces support for the KQL query language as an ES|QL function.
Our main goal is to provides users a way forward to migrate from KQL to ES|QL by integrating KQL queries directly into ES|QL queries.

We recently added the support of KQL directly form the query DSL:

GET movies/_search
{
  "query": {
    "kql": {
      "query": "actors: \"Kurt Russel\" AND title: \"Big trouble\""
    }
  }
}

This PR expose this new capacity of the query DSL through a new KQL fulltext filtering function

POST _query
{
  "query": """
    FROM movies
    | WHERE kql("actors: \"Kurt Russel\" AND title: \"Big trouble\"")
  """
}

The KQL function is conceptually very similar to the QSTR function and come with the same limitations.

The new function is enabled only in snapshots builds (same for the KQL query in the query DSL)

github-actions · 2024-11-13T17:57:20Z

Documentation preview:

✨ Changed pages

…esql-kql-function

fang-xing-esql

Thanks for putting these together @afoucret ! It looks pretty good to me, I added some comments.

Here are my understanding of the KQL function, and please correct me if I missed anything:

In terms of the interface, the KQL function is very similar to QSTR so it makes sense to follow a similar approach as QSTR to support it in the ES|QL.
However they are different in syntax according to this google doc, although their syntax look very similar to me, I won't be surprised that I might make mistakes on the syntax when using them. From users perspective, it will be helpful to have a page(or a link to existing ES docs) in the ES|QL doc to describe when to use KQL and when to use QSTR, and have examples to describe the usages. Especially they are now both available through ES|QL we may want to suggest the best practice to reduce confusions. Perhaps it is only me, but I'm not clear when I should use which function. This is not a needed for a snapshot function, but it will be helpful to get us prepared for a tech preview or GA.
I suggest to add ES|QL-ui label to notify kibana about this new function, it is near the FF of 8.17, it will be nice to give them a heads up if this is targeting to 8.17.
I suggest to add test-release label to run this through release tests, sometimes it does catch surprise for us.

fang-xing-esql · 2024-11-20T16:21:15Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/Verifier.java

@@ -788,6 +789,14 @@ private static void checkFullTextQueryFunctions(LogicalPlan plan, Set<Failure> f
                qsf -> "[" + qsf.functionName() + "] " + qsf.functionType(),
                failures
            );
+            checkCommandsBeforeExpression(


We could do a refactor here to reduce repetitive code, if Kql and QueryString have the same restrictions, they can be added in a collection and call checkCommandsBeforeExpression in a loop.

Added a loop that remove the duplicated code.

fang-xing-esql · 2024-11-20T16:22:16Z

...sql/src/main/java/org/elasticsearch/xpack/esql/expression/function/EsqlFunctionRegistry.java

@@ -409,7 +410,8 @@ private static FunctionDefinition[][] snapshotFunctions() {
                // This is an experimental function and can be removed without notice.
                def(Delay.class, Delay::new, "delay"),
                def(Categorize.class, Categorize::new, "categorize"),
-                def(Rate.class, Rate::withUnresolvedTimestamp, "rate") } };
+                def(Rate.class, Rate::withUnresolvedTimestamp, "rate"),
+                def(Kql.class, Kql::new, "kql") } };


Shall we follow the alphabetical order here?

Done. Thank you.

fang-xing-esql · 2024-11-20T16:23:08Z

...plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/fulltext/Kql.java

+    @Override
+    public String functionName() {
+        return "KQL";
+    }


The override to functionName can be skipped, as the function name is the same as class name.

Removed the override.

fang-xing-esql · 2024-11-20T16:29:29Z

...n/esql/src/test/java/org/elasticsearch/xpack/esql/expression/function/fulltext/KqlTests.java

+import java.util.List;
+import java.util.function.Supplier;
+
+@FunctionName("kql")


This annotation can be skipped, as the class name is the same as function name.

Removed the annotation.

…esql-kql-function

elasticsearchmachine · 2024-11-21T14:58:20Z

Pinging @elastic/kibana-esql (ES|QL-ui)

afoucret · 2024-11-21T14:58:43Z

@fang-xing-esql Thanks for the detailed review and feedback!

You're absolutely right that the KQL and QSTR functions are quite similar.
Both allow to leverage the _search infrastructure to execute another query language (respectively the Kibana Query Language and Lucene query string).

That said, for KQL, our intention is not to promote it as a standalone feature but rather to facilitate users migrating to ES|QL. This is why we’ve kept communication about it relatively low-key.
I will work on enriching the documentation by adding a link to the existing KQL page.

Few other points:

I updated the code per your suggestions.
I added the test-release label to ensure those tests are sucessful
I updated the version label to target 8.18.0 instead of 8.17.0
I added the ES|QL-ui label to keep the Kibana team informed of the progress

fang-xing-esql

Thank you @afoucret! LGTM. And thanks to @carlosdelest for reviewing as well.

…esql-kql-function

elasticsearchmachine · 2024-11-25T13:23:23Z

💚 Backport successful

Status	Branch	Result
✅	8.x

* ES|QL kql function. (#116764) * Fix test compile error in branch 8.x

elasticsearchmachine added the v9.0.0 label Nov 13, 2024

afoucret force-pushed the esql-kql-function branch 2 times, most recently from e13af7c to 44bb5a0 Compare November 14, 2024 11:10

afoucret added 5 commits November 14, 2024 12:19

Adding KqlQuery to esql-core query dsl supported queries.

ed1e528

Adding KQL_FUNCTION capability.

b7e2a56

Change visibility for KqlQueryBuilder parser fields.

eb81cd8

Creating the KQL fulltext function.

37d0f38

Register the KQL function in various places.

a271aea

afoucret force-pushed the esql-kql-function branch from 44bb5a0 to a271aea Compare November 14, 2024 11:19

afoucret and others added 12 commits November 14, 2024 17:43

Merge branch 'elastic:main' into esql-kql-function

31f45db

Refactoring dependencies on KQL

e3c1737

Remove moved tests.

1659381

Adding some Verifier tests.

949e8e0

Fixing telemetry.

27afe86

Add integration test for the KQL function.

5f6e321

Adding CSV tests.

21a37c8

Regenerate documentation for the KQL function.

41f0276

Merge branch 'main' of https://github.com/elastic/elasticsearch into …

be48540

…esql-kql-function

Add LogicalPhysicalPlanOptimizer test cases for KQL.

fc0e1e3

Fixing some errors in the qstr csv test spec.

9370d55

Fix in the doc.

f02f716

afoucret added :Analytics/ES|QL AKA ESQL :Search Relevance/Search Catch all for Search Relevance v8.17.0 >non-issue auto-backport Automatically create backport pull requests when merged labels Nov 19, 2024

afoucret requested review from tteofili, carlosdelest and ioanatia November 19, 2024 10:24

fang-xing-esql reviewed Nov 20, 2024

View reviewed changes

afoucret added the test-release Trigger CI checks against release build label Nov 20, 2024

afoucret added 5 commits November 20, 2024 20:22

Remove duplicated Verifier code for QSTR and KQL.

7bf2d14

Sort functions by alphabetical order.

f66b657

Remove useless override.

247d188

Remove useless annotation.

47f8851

Merge branch 'main' of https://github.com/elastic/elasticsearch into …

02ea75e

…esql-kql-function

elasticsearchmachine added v8.18.0 and removed v8.17.0 labels Nov 20, 2024

afoucret added the ES|QL-ui Impacts ES|QL UI label Nov 21, 2024

fang-xing-esql approved these changes Nov 21, 2024

View reviewed changes

afoucret force-pushed the esql-kql-function branch from 8bd0239 to 2354e07 Compare November 22, 2024 13:59

afoucret requested review from a team as code owners November 22, 2024 13:59

afoucret force-pushed the esql-kql-function branch from 2354e07 to 02ea75e Compare November 22, 2024 14:00

afoucret added 6 commits November 22, 2024 15:07

Merge branch 'main' of https://github.com/elastic/elasticsearch into …

2577576

…esql-kql-function

Remove gen files added by mistake

abad995

Fix test compile after merge from main.

d096e9e

Lint fix.

cab9a30

Merge branch 'main' of https://github.com/elastic/elasticsearch into …

f88fc8f

…esql-kql-function

Update the number of functions in telemetry.

680014a

afoucret merged commit ff58d89 into elastic:main Nov 25, 2024
17 checks passed

afoucret mentioned this pull request Nov 25, 2024

[8.x] ES|QL kql function. (#116764) #117474

Merged

afoucret added a commit to afoucret/elasticsearch that referenced this pull request Nov 25, 2024

ES|QL kql function. (elastic#116764)

d63eb58

stratoula mentioned this pull request Nov 25, 2024

[ES|QL] Autocomplete for the new KQL function elastic/kibana#201623

Closed

elasticsearchmachine pushed a commit that referenced this pull request Nov 25, 2024

[8.x] ES|QL kql function. (#116764) (#117474)

891d3e5

* ES|QL kql function. (#116764) * Fix test compile error in branch 8.x

alexey-ivanov-es pushed a commit to alexey-ivanov-es/elasticsearch that referenced this pull request Nov 28, 2024

ES|QL kql function. (elastic#116764)

40c67c8

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ES|QL kql function. #116764

ES|QL kql function. #116764

afoucret commented Nov 13, 2024 •

edited

Loading

github-actions bot commented Nov 13, 2024

fang-xing-esql left a comment •

edited

Loading

fang-xing-esql Nov 20, 2024

afoucret Nov 20, 2024

fang-xing-esql Nov 20, 2024

afoucret Nov 20, 2024

fang-xing-esql Nov 20, 2024

afoucret Nov 20, 2024

fang-xing-esql Nov 20, 2024

afoucret Nov 20, 2024

elasticsearchmachine commented Nov 21, 2024

afoucret commented Nov 21, 2024

fang-xing-esql left a comment

elasticsearchmachine commented Nov 25, 2024

ES|QL kql function. #116764

ES|QL kql function. #116764

Conversation

afoucret commented Nov 13, 2024 • edited Loading

github-actions bot commented Nov 13, 2024

fang-xing-esql left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticsearchmachine commented Nov 21, 2024

afoucret commented Nov 21, 2024

fang-xing-esql left a comment

Choose a reason for hiding this comment

elasticsearchmachine commented Nov 25, 2024

💚 Backport successful

afoucret commented Nov 13, 2024 •

edited

Loading

fang-xing-esql left a comment •

edited

Loading