ESQL: change from quoting from backtick to quote #108395

costin · 2024-05-08T03:43:17Z

For historical reasons, the source declaration inside FROM command is
treated as an identifier, using backticks (``) for escaping the value.
This is inconsistent since the source is not an identifier (field name)
but an index name which has different semantics.
index means a field name index while "index" means a literal with
said value.

In case of FROM, the index name/location is more like a literal (also in
unquoted form) than an identifier (that is a reference to a value).

This PR tweaks the grammar and plugs in the quoted string logic so that
both the single quote (") and triple quote (""") are allowed.

For historical reasons, the source declaration inside FROM command is treated as an identifier, using backticks (`) for escaping the value. This is inconsistent since the source is not an identifier (field name) but an index name which has different semantics. `index` means a field name index while "index" means a literal with said value. In case of FROM, the index name/location is more like a literal (also in unquoted form) than an identifier (that is a reference to a value). This PR tweaks the grammar and plugs in the quoted string logic so that both the single quote (") and triple quote (""") are allowed.

elasticsearchmachine · 2024-05-08T03:43:40Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

elasticsearchmachine · 2024-05-08T03:43:41Z

Hi @costin, I've created a changelog YAML for you.

alex-spies

I didn't get to fully review the PR yet, but I noticed it is a breaking language change. We need to use our language version mechanism to avoid breaking existing queries.

alex-spies · 2024-05-08T08:32:58Z

x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/parser/StatementParserTests.java

        assertIdentifierAsIndexPattern("foo,test,xyz", "from foo,   test,xyz");
        assertIdentifierAsIndexPattern(
            "<logstash-{now/M{yyyy.MM}}>,<logstash-{now/d{yyyy.MM.dd|+12:00}}>",
-            "from <logstash-{now/M{yyyy.MM}}>, `<logstash-{now/d{yyyy.MM.dd|+12:00}}>`"
+            "from <logstash-{now/M{yyyy.MM}}>, \"<logstash-{now/d{yyyy.MM.dd|+12:00}}>\""
        );


I am surprised we don't need to update any csv tests - I guess we should add one or two to ensure this change also works correctly end-to-end.

And use these csv tests in the docs for FROM; currently, I think they contain incorrect syntax (missing backticks/quotes):

It'd also be great to add some tests with backticks now.

x-pack/plugin/esql/src/main/antlr/EsqlBaseLexer.g4

alex-spies · 2024-05-08T08:40:47Z

Also, we'll have to update the docs for FROM.

alex-spies

Another thing I noticed is that for

FROM `testidx`

we include the backticks in the parsed index pattern/name. I think backticks should not be allowed here.

bpintea · 2024-05-08T13:50:25Z

we include the backticks in the parsed index pattern/name. I think backticks should not be allowed here.

I think they should be allowed (maybe surprisingly).
(Also, PUT "my_index" fails with: "reason": """Invalid index name ["my_index"], must not contain the following characters ['<','*','?','>','|',',','/','\',' ','"']""")

bpintea

On one hand, this feels good since quotes aren't allowed in index names, so it feels natural to use them for quoting. OTOH, index names feel closer to identifiers than string literals and conceptually, would be easier to understand them together, as a language user.

It'd be good to have more tests with backticks now, as well as csv-specs, as noted by Alex.

It'd also be good to get it in 8.14, if we can sync this with other clients (like Kibana).

bpintea · 2024-05-08T13:54:34Z

x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/parser/StatementParserTests.java

        assertIdentifierAsIndexPattern("foo,test,xyz", "from foo,   test,xyz");
        assertIdentifierAsIndexPattern(
            "<logstash-{now/M{yyyy.MM}}>,<logstash-{now/d{yyyy.MM.dd|+12:00}}>",
-            "from <logstash-{now/M{yyyy.MM}}>, `<logstash-{now/d{yyyy.MM.dd|+12:00}}>`"
+            "from <logstash-{now/M{yyyy.MM}}>, \"<logstash-{now/d{yyyy.MM.dd|+12:00}}>\""
        );


It'd also be great to add some tests with backticks now.

astefan

I think it's ok to have these as strings/constants (vs. identifiers). Until now, the backticks were more like a way of emphasizing that the index names are not regular "strings", but based on your explanation, they in essence are and I agree.

alex-spies

Since quoting was disabled in this PR, this PR does not break bwc anymore, so my change request is moot :)

elasticsearchmachine · 2024-06-04T02:37:33Z

Pinging @elastic/kibana-esql (ES|QL-ui)

costin · 2024-06-04T02:37:40Z

I've added the target for this PR to be 8.15 (no backport to 8.14 necessary).
Since quoting (with backticks, in the old form) was removed in 8.14, this PR is just about adding regular quotes to indices for 8.15 and there's no backwards compatibility to worry about.

…fier

elasticsearchmachine · 2024-06-26T08:59:22Z

Hi @costin, I've created a changelog YAML for you.

bpintea · 2024-06-26T14:15:09Z

@elasticmachine update branch

elasticmachine · 2024-06-26T14:15:12Z

merge conflict between base and head

…ad-of-identifier

bpintea · 2024-06-26T16:25:15Z

I think this should be followed up by updating the docs for FROM; esp. they should use examples from csv tests (currently hard-coded, manual examples) and document + illustrate the use of quotes.

I've updated the docs with just a small section on using them. This can be extended in a followup.

Since we also generalize the quoting for LOOKUP and METRICS, we should probably add some tests to that, too.

I've added some more tests, which uncovered one LOOKUP issue (unquoting).

docs/reference/esql/source-commands/from.asciidoc

…ad-of-identifier

bpintea · 2024-06-30T14:38:42Z

@elasticmachine run elasticsearch-ci/part-3

Co-authored-by: marciw <[email protected]>

astefan · 2024-06-30T15:33:46Z

@elasticmachine run elasticsearch-ci/part-3

astefan · 2024-06-30T16:04:43Z

@elasticmachine update branch

…fier

## Summary Follow up of elastic/elasticsearch#108395 ### Checklist - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios --------- Co-authored-by: Søren Louv-Jansen <[email protected]>

## Summary Follow up of elastic/elasticsearch#108395 ### Checklist - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios --------- Co-authored-by: Søren Louv-Jansen <[email protected]> (cherry picked from commit 7b635f3)

# Backport This will backport the following commits from `main` to `8.15`: - [[Obs AI assistant] Replace with double quotes (#187327)](#187327)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  Co-authored-by: Stratoula Kalafateli <[email protected]>

…dex names (#187212) Related to elastic/elasticsearch#108395. This PR updates the ES|QL grammars (lexer and parser) to match the latest version in Elasticsearch. The PR adds support for index names wrapped by `""`, and triple quotes `"""index-name"""` to the ES|QL text editor and removes backticks. <img width="1392" alt="Screenshot 2024-07-09 at 09 25 22" src="https://github.com/elastic/kibana/assets/43350163/4e93f503-9053-42a6-99c5-beda7bb4ba4c"> <img width="1392" alt="Screenshot 2024-07-09 at 09 24 54" src="https://github.com/elastic/kibana/assets/43350163/eec6292c-f3c0-49f0-9864-cb3e6717e7a7"> <img width="1392" alt="Screenshot 2024-07-09 at 09 25 03" src="https://github.com/elastic/kibana/assets/43350163/66dca108-a535-4df7-b1c0-42705c7913dd"> --------- Co-authored-by: Quynh Nguyen <[email protected]> Co-authored-by: Elastic Machine <[email protected]>

…dex names (elastic#187212) Related to elastic/elasticsearch#108395. This PR updates the ES|QL grammars (lexer and parser) to match the latest version in Elasticsearch. The PR adds support for index names wrapped by `""`, and triple quotes `"""index-name"""` to the ES|QL text editor and removes backticks. <img width="1392" alt="Screenshot 2024-07-09 at 09 25 22" src="https://github.com/elastic/kibana/assets/43350163/4e93f503-9053-42a6-99c5-beda7bb4ba4c"> <img width="1392" alt="Screenshot 2024-07-09 at 09 24 54" src="https://github.com/elastic/kibana/assets/43350163/eec6292c-f3c0-49f0-9864-cb3e6717e7a7"> <img width="1392" alt="Screenshot 2024-07-09 at 09 25 03" src="https://github.com/elastic/kibana/assets/43350163/66dca108-a535-4df7-b1c0-42705c7913dd"> --------- Co-authored-by: Quynh Nguyen <[email protected]> Co-authored-by: Elastic Machine <[email protected]> (cherry picked from commit 5bc0c78)

… for index names (#187212) (#187895) # Backport This will backport the following commits from `main` to `8.15`: - [[ES|QL] Update grammars: change quoting from backtick to quote for index names (#187212)](#187212)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)

costin added >bug :Analytics/ES|QL AKA ESQL ES|QL-ui Impacts ES|QL UI v8.14.1 v8.15.0 labels May 8, 2024

costin requested review from astefan, bpintea, alex-spies and fang-xing-esql May 8, 2024 03:43

elasticsearchmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label May 8, 2024

alex-spies requested changes May 8, 2024

View reviewed changes

alex-spies self-requested a review May 8, 2024 08:40

alex-spies reviewed May 8, 2024

View reviewed changes

alex-spies self-requested a review May 8, 2024 10:59

bpintea approved these changes May 8, 2024

View reviewed changes

astefan approved these changes May 8, 2024

View reviewed changes

astefan mentioned this pull request May 9, 2024

[ES|QL] Support Named and Positional Parameters in EsqlQueryRequest #108421

Merged

alex-spies approved these changes May 10, 2024

View reviewed changes

costin removed the v8.14.1 label Jun 4, 2024

costin added 2 commits June 4, 2024 09:25

Merge branch 'main' into esql/from-source-as-string-instead-of-identi…

1bcb7dd

…fier

Update grammar

9d97583

costin force-pushed the esql/from-source-as-string-instead-of-identifier branch from c3dd636 to 9d97583 Compare June 4, 2024 21:37

Add more tests

27a94d9

Update docs/changelog/108395.yaml

2dfc172

bpintea added 2 commits June 26, 2024 16:21

Merge branch 'main' into fork/costin/esql/from-source-as-string-inste…

b77c516

…ad-of-identifier

Adress review comments

093a726

costin assigned astefan and unassigned bpintea Jun 26, 2024

Add doc note

a714672

marciw reviewed Jun 26, 2024

View reviewed changes

docs/reference/esql/source-commands/from.asciidoc Outdated Show resolved Hide resolved

bpintea added 4 commits June 27, 2024 09:01

Merge branch 'main' into fork/costin/esql/from-source-as-string-inste…

eecdbc7

…ad-of-identifier

Revert test rename

dcf4767

Fix quoting with remote cluster

8fd2be7

Merge branch 'main' into fork/costin/esql/from-source-as-string-inste…

ea6ed00

…ad-of-identifier

Update docs/reference/esql/source-commands/from.asciidoc

167c2d3

Co-authored-by: marciw <[email protected]>

Merge branch 'main' into esql/from-source-as-string-instead-of-identi…

a096616

…fier

astefan merged commit b906ce3 into elastic:main Jun 30, 2024
15 checks passed

stratoula mentioned this pull request Jul 2, 2024

[Obs AI assistant] Replace with double quotes elastic/kibana#187327

Merged

1 task

qn895 mentioned this pull request Jul 9, 2024

[ES|QL] Update grammars: change quoting from backtick to quote for index names elastic/kibana#187212

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ESQL: change from quoting from backtick to quote #108395

ESQL: change from quoting from backtick to quote #108395

costin commented May 8, 2024 •

edited by astefan

Loading

elasticsearchmachine commented May 8, 2024

elasticsearchmachine commented May 8, 2024

alex-spies left a comment

alex-spies May 8, 2024

alex-spies May 8, 2024

bpintea May 8, 2024

alex-spies commented May 8, 2024

alex-spies left a comment

bpintea commented May 8, 2024

bpintea left a comment

bpintea May 8, 2024

astefan left a comment

alex-spies left a comment

elasticsearchmachine commented Jun 4, 2024

costin commented Jun 4, 2024 •

edited

Loading

elasticsearchmachine commented Jun 26, 2024

bpintea commented Jun 26, 2024

elasticmachine commented Jun 26, 2024

bpintea commented Jun 26, 2024 •

edited

Loading

bpintea commented Jun 30, 2024

astefan commented Jun 30, 2024

astefan commented Jun 30, 2024

ESQL: change from quoting from backtick to quote #108395

ESQL: change from quoting from backtick to quote #108395

Conversation

costin commented May 8, 2024 • edited by astefan Loading

elasticsearchmachine commented May 8, 2024

elasticsearchmachine commented May 8, 2024

alex-spies left a comment

Choose a reason for hiding this comment

alex-spies May 8, 2024

Choose a reason for hiding this comment

alex-spies May 8, 2024

Choose a reason for hiding this comment

bpintea May 8, 2024

Choose a reason for hiding this comment

alex-spies commented May 8, 2024

alex-spies left a comment

Choose a reason for hiding this comment

bpintea commented May 8, 2024

bpintea left a comment

Choose a reason for hiding this comment

bpintea May 8, 2024

Choose a reason for hiding this comment

astefan left a comment

Choose a reason for hiding this comment

alex-spies left a comment

Choose a reason for hiding this comment

elasticsearchmachine commented Jun 4, 2024

costin commented Jun 4, 2024 • edited Loading

elasticsearchmachine commented Jun 26, 2024

bpintea commented Jun 26, 2024

elasticmachine commented Jun 26, 2024

bpintea commented Jun 26, 2024 • edited Loading

bpintea commented Jun 30, 2024

astefan commented Jun 30, 2024

astefan commented Jun 30, 2024

costin commented May 8, 2024 •

edited by astefan

Loading

costin commented Jun 4, 2024 •

edited

Loading

bpintea commented Jun 26, 2024 •

edited

Loading