Support SELECT * and FROM clause in new SQL parser #573

dai-chen · 2020-07-13T22:41:41Z

Issue #, if available: #575, #504, #322

Description of changes: Change SQL grammar to integrate with new engine and support SELECT * and FROM clause.

Change SQL grammar: split identifier rule to an individual file so it can be overridden in fuzzy testing.
Build Relation node and QualifiedName expression in AST builder.

Testing: Add new UT and IT for different types of identifier (index name). Because of problem with date, text and keyword in JDBC driver, select * is not added to comparison test.

Note a major change on integration test: A new integTestWithNewEngine task is added to run PPL and new, legacy, comparison SQL ITs with new engine enabled. The integTest depends on it and only runs legacy SQL ITs with new engine disabled. Because more and more SQL queries will start being handled by new engine after FROM clause added, this makes sure that our legacy ITs can work with/without new engine enabled.

Documentation: Checked out MySQL, SQLServer and ANSI SQL and organize our doc in this way: https://github.com/dai-chen/sql/blob/support-select-all-from/docs/user/general/identifiers.rst

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

sql/src/test/java/com/amazon/opendistroforelasticsearch/sql/sql/antlr/SQLSyntaxParserTest.java

docs/user/general/identifiers.rst

chloe-zh

LGTM, thanks!

penghuo

Thanks for the change.

kedargn · 2020-09-02T17:30:43Z

@dai-chen I am facing the same issue. I tried to enable 'opendistro.sql.engine.new.enabled' setting in open distro ES version 1.8.0. I got an error

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "transient setting [opendistro.sql.engine.new.enabled], not recognized"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "transient setting [opendistro.sql.engine.new.enabled], not recognized"
  },
  "status": 400
}

Do I have to update to version 1.9.0 to enable new SQL engine?

dai-chen · 2020-09-02T21:39:46Z

@dai-chen I am facing the same issue. I tried to enable 'opendistro.sql.engine.new.enabled' setting in open distro ES version 1.8.0. I got an error
{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "transient setting [opendistro.sql.engine.new.enabled], not recognized"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "transient setting [opendistro.sql.engine.new.enabled], not recognized"
  },
  "status": 400
}
Do I have to update to version 1.9.0 to enable new SQL engine?

Thanks for reaching out! This setting and new query engine is available in 1.9.0.1+: https://github.com/opendistro-for-elasticsearch/sql/blob/opendistro-1.9.0.1/docs/user/admin/settings.rst#opendistro-sql-engine-new-enabled. Please try to upgrade to 1.9.0.1 or latest 1.10. Thanks!

kedargn · 2020-09-02T21:51:10Z

@dai-chen I am facing the same issue. I tried to enable 'opendistro.sql.engine.new.enabled' setting in open distro ES version 1.8.0. I got an error
{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "transient setting [opendistro.sql.engine.new.enabled], not recognized"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "transient setting [opendistro.sql.engine.new.enabled], not recognized"
  },
  "status": 400
}
Do I have to update to version 1.9.0 to enable new SQL engine?
Thanks for reaching out! This setting and new query engine is available in 1.9.0.1+: https://github.com/opendistro-for-elasticsearch/sql/blob/opendistro-1.9.0.1/docs/user/admin/settings.rst#opendistro-sql-engine-new-enabled. Please try to upgrade to 1.9.0.1 or latest 1.10. Thanks!

@dai-chen Thank you for your reply! Latest docker image for open distro ES I can find is 1.9.0. I am not able to find 1.10. Am I missing something here? https://hub.docker.com/r/amazon/opendistro-for-elasticsearch/tags

dai-chen · 2020-09-02T22:42:43Z

@dai-chen I am facing the same issue. I tried to enable 'opendistro.sql.engine.new.enabled' setting in open distro ES version 1.8.0. I got an error
{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "transient setting [opendistro.sql.engine.new.enabled], not recognized"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "transient setting [opendistro.sql.engine.new.enabled], not recognized"
  },
  "status": 400
}
Do I have to update to version 1.9.0 to enable new SQL engine?
Thanks for reaching out! This setting and new query engine is available in 1.9.0.1+: https://github.com/opendistro-for-elasticsearch/sql/blob/opendistro-1.9.0.1/docs/user/admin/settings.rst#opendistro-sql-engine-new-enabled. Please try to upgrade to 1.9.0.1 or latest 1.10. Thanks!
@dai-chen Thank you for your reply! Latest docker image for open distro ES I can find is 1.9.0. I am not able to find 1.10. Am I missing something here? https://hub.docker.com/r/amazon/opendistro-for-elasticsearch/tags

I can see other artifacts updated to 1.9.0.1 already. Could you try to check out 1.9.0.1 docker image?

kedargn · 2020-09-02T22:49:35Z

@dai-chen I am facing the same issue. I tried to enable 'opendistro.sql.engine.new.enabled' setting in open distro ES version 1.8.0. I got an error
{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "transient setting [opendistro.sql.engine.new.enabled], not recognized"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "transient setting [opendistro.sql.engine.new.enabled], not recognized"
  },
  "status": 400
}
Do I have to update to version 1.9.0 to enable new SQL engine?
Thanks for reaching out! This setting and new query engine is available in 1.9.0.1+: https://github.com/opendistro-for-elasticsearch/sql/blob/opendistro-1.9.0.1/docs/user/admin/settings.rst#opendistro-sql-engine-new-enabled. Please try to upgrade to 1.9.0.1 or latest 1.10. Thanks!
@dai-chen Thank you for your reply! Latest docker image for open distro ES I can find is 1.9.0. I am not able to find 1.10. Am I missing something here? https://hub.docker.com/r/amazon/opendistro-for-elasticsearch/tags
I can see other artifacts updated to 1.9.0.1 already. Could you try to check out 1.9.0.1 docker image?

When I try to pull 1.9.0.1 it says not found.

docker image pull "amazon/opendistro-for-elasticsearch:1.9.0.1"
Error response from daemon: manifest for amazon/opendistro-for-elasticsearch:1.9.0.1 not found

However, I can see 1.9.0

1.9.0: Pulling from amazon/opendistro-for-elasticsearch
Digest: sha256:878e6947b3a50487f5a57453a7ba8c1e7da6488f997c6fc257c657ad04874d29
Status: Image is up to date for amazon/opendistro-for-elasticsearch:1.9.0

Also, I don't see version 1.9.0.1 here https://hub.docker.com/r/amazon/opendistro-for-elasticsearch/tags. Am I doing something wrong?

dai-chen · 2020-09-02T23:52:00Z

@dai-chen I am facing the same issue. I tried to enable 'opendistro.sql.engine.new.enabled' setting in open distro ES version 1.8.0. I got an error
{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "transient setting [opendistro.sql.engine.new.enabled], not recognized"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "transient setting [opendistro.sql.engine.new.enabled], not recognized"
  },
  "status": 400
}
Do I have to update to version 1.9.0 to enable new SQL engine?
Thanks for reaching out! This setting and new query engine is available in 1.9.0.1+: https://github.com/opendistro-for-elasticsearch/sql/blob/opendistro-1.9.0.1/docs/user/admin/settings.rst#opendistro-sql-engine-new-enabled. Please try to upgrade to 1.9.0.1 or latest 1.10. Thanks!
@dai-chen Thank you for your reply! Latest docker image for open distro ES I can find is 1.9.0. I am not able to find 1.10. Am I missing something here? https://hub.docker.com/r/amazon/opendistro-for-elasticsearch/tags
I can see other artifacts updated to 1.9.0.1 already. Could you try to check out 1.9.0.1 docker image?
When I try to pull 1.9.0.1 it says not found.
docker image pull "amazon/opendistro-for-elasticsearch:1.9.0.1"
Error response from daemon: manifest for amazon/opendistro-for-elasticsearch:1.9.0.1 not found
However, I can see 1.9.0
1.9.0: Pulling from amazon/opendistro-for-elasticsearch
Digest: sha256:878e6947b3a50487f5a57453a7ba8c1e7da6488f997c6fc257c657ad04874d29
Status: Image is up to date for amazon/opendistro-for-elasticsearch:1.9.0
Also, I don't see version 1.9.0.1 here https://hub.docker.com/r/amazon/opendistro-for-elasticsearch/tags. Am I doing something wrong?

I see. Although other artifacts are available, it seems we haven't announced the 1.9.0.1 yet: https://opendistro.github.io/for-elasticsearch/blog/. Sorry for the confusion and please stay tune for the upcoming release. You should be able to enable and play with new query engine soon.

dai-chen added 11 commits July 7, 2020 10:56

Support from

aef681b

Add more UT

bbb8caa

Merge branch 'develop' into support-select-all-from

6848f97

Merge branch 'develop' into support-select-all-from

2d89a35

Merge branch 'develop' into support-select-all-from

bde1b51

Update doc

92fe877

Merge branch 'develop' into support-select-all-from

7dc8a0e

Update doc

683d53b

Merge branch 'develop' into support-select-all-from

9fffdd0

Add doctest

ae3d756

Add IT

a17651f

dai-chen added the SQL label Jul 13, 2020

dai-chen self-assigned this Jul 13, 2020

dai-chen changed the title ~~Support select all from~~ Support SELECT * and FROM clause Jul 13, 2020

dai-chen changed the title ~~Support SELECT * and FROM clause~~ Support SELECT * and FROM clause in new engine Jul 14, 2020

dai-chen changed the title ~~Support SELECT * and FROM clause in new engine~~ Support SELECT * and FROM clause in new SQL parser Jul 14, 2020

dai-chen added 5 commits July 14, 2020 17:43

Change doc and grammar for ANSI SQL

af1422f

Change doc and grammar

f32dcb9

Split grammar file

3f98057

Prepare PR

54d7722

Prepare PR

b6bb289

dai-chen marked this pull request as ready for review July 16, 2020 14:46

dai-chen requested review from penghuo and chloe-zh July 16, 2020 14:46

Run IT with/without new engine

8b2d212

penghuo reviewed Jul 16, 2020

View reviewed changes

sql/src/test/java/com/amazon/opendistroforelasticsearch/sql/sql/antlr/SQLSyntaxParserTest.java Show resolved Hide resolved

penghuo reviewed Jul 16, 2020

View reviewed changes

docs/user/general/identifiers.rst Outdated Show resolved Hide resolved

chloe-zh approved these changes Jul 17, 2020

View reviewed changes

dai-chen added 2 commits July 20, 2020 11:44

Address PR comments

85c4b05

Address PR comments

fe867f4

Merge branch 'develop' into support-select-all-from

e18ee9c

penghuo approved these changes Jul 21, 2020

View reviewed changes

Merge branch 'develop' into support-select-all-from

6a33e69

dai-chen merged commit 0a878ab into opendistro-for-elasticsearch:develop Jul 21, 2020

dai-chen deleted the support-select-all-from branch July 21, 2020 17:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support SELECT * and FROM clause in new SQL parser #573

Support SELECT * and FROM clause in new SQL parser #573

dai-chen commented Jul 13, 2020 •

edited

Loading

chloe-zh left a comment

penghuo left a comment

kedargn commented Sep 2, 2020 •

edited

Loading

dai-chen commented Sep 2, 2020

kedargn commented Sep 2, 2020 •

edited

Loading

dai-chen commented Sep 2, 2020

kedargn commented Sep 2, 2020 •

edited

Loading

dai-chen commented Sep 2, 2020

Support SELECT * and FROM clause in new SQL parser #573

Support SELECT * and FROM clause in new SQL parser #573

Conversation

dai-chen commented Jul 13, 2020 • edited Loading

chloe-zh left a comment

Choose a reason for hiding this comment

penghuo left a comment

Choose a reason for hiding this comment

kedargn commented Sep 2, 2020 • edited Loading

dai-chen commented Sep 2, 2020

kedargn commented Sep 2, 2020 • edited Loading

dai-chen commented Sep 2, 2020

kedargn commented Sep 2, 2020 • edited Loading

dai-chen commented Sep 2, 2020

dai-chen commented Jul 13, 2020 •

edited

Loading

kedargn commented Sep 2, 2020 •

edited

Loading

kedargn commented Sep 2, 2020 •

edited

Loading

kedargn commented Sep 2, 2020 •

edited

Loading