Add support for wildcard_query function #156

GumpacG · 2022-11-04T15:48:57Z

Description

Add support for wildcard_query function:

wildcard_query and wildcardquery are supported in SQL.
wildcard_query is supported in PPL.
Wildcard characters that are currently supported are * and ? as per https://opensearch.org/docs/latest/opensearch/query-dsl/term/#wildcards
Wildcard conversions % to * and _ to ? is supported as per the legacy implementation.

Issues Resolved

Check List

New functionality includes testing.
- All tests pass, including unit test, integration test and doctest
New functionality has been documented.
- New functionality has javadoc added
- New functionality has user manual doc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Guian Gumpac <[email protected]>

codecov · 2022-11-04T15:57:46Z

Codecov Report

Merging #156 (9da6140) into integ-add-wildcardquery (50669eb) will decrease coverage by 2.50%.
The diff coverage is 100.00%.

@@                      Coverage Diff                      @@
##             integ-add-wildcardquery     #156      +/-   ##
=============================================================
- Coverage                      98.29%   95.79%   -2.51%     
- Complexity                      3465     3473       +8     
=============================================================
  Files                            347      358      +11     
  Lines                           8645     9322     +677     
  Branches                         550      669     +119     
=============================================================
+ Hits                            8498     8930     +432     
- Misses                           142      334     +192     
- Partials                           5       58      +53

Flag	Coverage Δ
query-workbench	`62.76% <ø> (?)`
sql-engine	`98.30% <100.00%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...arch/storage/script/filter/FilterQueryBuilder.java	`100.00% <ø> (ø)`
...c/main/java/org/opensearch/sql/expression/DSL.java	`100.00% <100.00%> (ø)`
...h/sql/expression/function/BuiltinFunctionName.java	`100.00% <100.00%> (ø)`
...h/sql/expression/function/OpenSearchFunctions.java	`100.00% <100.00%> (ø)`
...search/storage/script/filter/lucene/LikeQuery.java	`100.00% <100.00%> (ø)`
.../lucene/relevance/FunctionParameterRepository.java	`100.00% <100.00%> (ø)`
.../script/filter/lucene/relevance/WildcardQuery.java	`100.00% <100.00%> (ø)`
...h/public/components/QueryLanguageSwitch/Switch.tsx	`85.71% <0.00%> (ø)`
workbench/public/components/Header/Header.tsx	`100.00% <0.00%> (ø)`
...ch/public/components/QueryResults/QueryResults.tsx	`61.60% <0.00%> (ø)`
... and 7 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

...java/org/opensearch/sql/opensearch/storage/script/filter/lucene/relevance/WildcardQuery.java

Signed-off-by: Guian Gumpac <[email protected]>

docs/user/dql/functions.rst

docs/user/ppl/functions/relevance.rst

...java/org/opensearch/sql/opensearch/storage/script/filter/lucene/relevance/WildcardQuery.java

...rc/test/java/org/opensearch/sql/opensearch/storage/script/filter/FilterQueryBuilderTest.java

.../test/java/org/opensearch/sql/opensearch/storage/script/filter/lucene/WildcardQueryTest.java

...earch/sql/opensearch/storage/script/filter/lucene/relevance/FunctionParameterRepository.java

core/src/test/java/org/opensearch/sql/expression/function/OpenSearchFunctionsTest.java

docs/user/dql/functions.rst

integ-test/src/test/java/org/opensearch/sql/ppl/WildcardQueryIT.java

integ-test/src/test/java/org/opensearch/sql/sql/WildcardQueryIT.java

Signed-off-by: Guian Gumpac <[email protected]>

...earch/sql/opensearch/storage/script/filter/lucene/relevance/FunctionParameterRepository.java

...java/org/opensearch/sql/opensearch/storage/script/filter/lucene/relevance/WildcardQuery.java

Signed-off-by: Guian Gumpac <[email protected]>

Yury-Fridlyand · 2022-11-14T23:22:32Z

Please, fix failed integration tests.

docs/user/dql/functions.rst

integ-test/src/test/java/org/opensearch/sql/sql/WildcardQueryIT.java

…arch-project-sql into dev-add-wildcardquery

Signed-off-by: Guian Gumpac <[email protected]>

...arch/src/main/java/org/opensearch/sql/opensearch/storage/script/filter/lucene/LikeQuery.java

Signed-off-by: Guian Gumpac <[email protected]>

acarbonetto · 2022-11-22T23:01:54Z

Can we look into fixing the coverage for this please?

docs/user/dql/functions.rst

...arch/src/main/java/org/opensearch/sql/opensearch/storage/script/filter/lucene/LikeQuery.java

Signed-off-by: Guian Gumpac <[email protected]>

MaxKsyunz · 2022-11-24T21:33:05Z

integ-test/src/test/java/org/opensearch/sql/ppl/LikeQueryIT.java

+  }
+
+  @Test
+  public void test_like_with_percent() throws IOException {


❤️ I like how you split tests by scenario.

MaxKsyunz · 2022-11-24T23:03:22Z

...arch/src/main/java/org/opensearch/sql/opensearch/storage/script/filter/lucene/LikeQuery.java

+  @Override
+  public QueryBuilder build(FunctionExpression func) {
+    ReferenceExpression ref = (ReferenceExpression) func.getArguments().get(0);
+    String field = convertTextToKeyword(ref.toString(), ref.type());


I'd use ref.getAttr() here instead of toString().

They happen to be the same in this case but toString() can return anything.

Thanks. Addressed in b54a934

MaxKsyunz · 2022-11-24T23:13:17Z

.../test/java/org/opensearch/sql/opensearch/storage/script/filter/lucene/WildcardQueryTest.java

+  }
+
+  @Test
+  public void test_escaping_sql_wildcards() {


A better home for this test is in the common/src/test/java/org.opensearch.sql.common.utils -- in same module where the StringUtils.convertSqlWildcardToLucene is defined.

Thanks. Addressed in b54a934

core/src/main/java/org/opensearch/sql/expression/DSL.java

Signed-off-by: Guian Gumpac <[email protected]>

…nsearch-project#1108) Signed-off-by: Guian Gumpac <[email protected]>

Guian Gumpac added 8 commits November 2, 2022 14:42

Implemented wildcard_query and added tests in core

a96a740

Signed-off-by: Guian Gumpac <[email protected]>

Implemented and added tests for sql

885f04c

Signed-off-by: Guian Gumpac <[email protected]>

Implemented and added tests for ppl

4d0dadb

Signed-off-by: Guian Gumpac <[email protected]>

Implemented and added tests for lucene

9c25e66

Signed-off-by: Guian Gumpac <[email protected]>

Fixed test for like expression

efe0ee5

Signed-off-by: Guian Gumpac <[email protected]>

Added parameters to wildcard_query

bedf662

Signed-off-by: Guian Gumpac <[email protected]>

Added integration tests for ppl and sql

afbeb44

Signed-off-by: Guian Gumpac <[email protected]>

Added docs for doctests

e25d4da

Signed-off-by: Guian Gumpac <[email protected]>

GumpacG marked this pull request as ready for review November 4, 2022 16:22

Yury-Fridlyand requested review from Yury-Fridlyand, acarbonetto, forestmvey, margarit-h, MaxKsyunz and MitchellGale November 4, 2022 17:16

Yury-Fridlyand reviewed Nov 4, 2022

View reviewed changes

...java/org/opensearch/sql/opensearch/storage/script/filter/lucene/relevance/WildcardQuery.java Outdated Show resolved Hide resolved

Guian Gumpac added 4 commits November 7, 2022 15:04

Pulled from 2.x

84fcf88

Signed-off-by: Guian Gumpac <[email protected]>

Fixed issues introduced during merging

4708ae2

Signed-off-by: Guian Gumpac <[email protected]>

Addressed PR comment

e05e5c5

Signed-off-by: Guian Gumpac <[email protected]>

Added annotation that was deleted from merging

988b40b

Signed-off-by: Guian Gumpac <[email protected]>

Yury-Fridlyand mentioned this pull request Nov 8, 2022

SQL ingestion and ops - standup opensearch-project/sql#791

Open

Fixed merge conflict issues

5c225ce

Signed-off-by: Guian Gumpac <[email protected]>

MaxKsyunz reviewed Nov 8, 2022

View reviewed changes

Yury-Fridlyand reviewed Nov 8, 2022

View reviewed changes

forestmvey reviewed Nov 9, 2022

View reviewed changes

integ-test/src/test/java/org/opensearch/sql/ppl/WildcardQueryIT.java Outdated Show resolved Hide resolved

forestmvey reviewed Nov 9, 2022

View reviewed changes

integ-test/src/test/java/org/opensearch/sql/sql/WildcardQueryIT.java Outdated Show resolved Hide resolved

forestmvey reviewed Nov 9, 2022

View reviewed changes

integ-test/src/test/java/org/opensearch/sql/sql/WildcardQueryIT.java Outdated Show resolved Hide resolved

Addressed some PR comments and handled escaping wildcards

0b9752c

Signed-off-by: Guian Gumpac <[email protected]>

MitchellGale reviewed Nov 9, 2022

View reviewed changes

...earch/sql/opensearch/storage/script/filter/lucene/relevance/FunctionParameterRepository.java Show resolved Hide resolved

...java/org/opensearch/sql/opensearch/storage/script/filter/lucene/relevance/WildcardQuery.java Outdated Show resolved Hide resolved

MaxKsyunz approved these changes Nov 14, 2022

View reviewed changes

MaxKsyunz self-requested a review November 14, 2022 22:13

Guian Gumpac added 2 commits November 14, 2022 14:31

Made namedArgument a static function

a9f5be7

Signed-off-by: Guian Gumpac <[email protected]>

Removed extra space

bba63ce

Signed-off-by: Guian Gumpac <[email protected]>

acarbonetto approved these changes Nov 15, 2022

View reviewed changes

Guian Gumpac added 2 commits November 15, 2022 12:07

Merge branch 'integ-add-wildcardquery' of github.com:Bit-Quill/opense…

8b7b442

…arch-project-sql into dev-add-wildcardquery

Fixed LIKE query

4af9eef

Signed-off-by: Guian Gumpac <[email protected]>

Yury-Fridlyand reviewed Nov 17, 2022

View reviewed changes

...arch/src/main/java/org/opensearch/sql/opensearch/storage/script/filter/lucene/LikeQuery.java Show resolved Hide resolved

Guian Gumpac added 4 commits November 18, 2022 11:23

Fixed LIKE tests and added more tests

57aac8e

Signed-off-by: Guian Gumpac <[email protected]>

Addressed PR comments

0a4af9d

Signed-off-by: Guian Gumpac <[email protected]>

Implemented converting text field to keyword. Still needs testing

288e29c

Signed-off-by: Guian Gumpac <[email protected]>

Added test cases for LIKE in sql and ppl

1fb4973

Signed-off-by: Guian Gumpac <[email protected]>

forestmvey approved these changes Nov 23, 2022

View reviewed changes

Yury-Fridlyand requested changes Nov 24, 2022

View reviewed changes

docs/user/dql/functions.rst Outdated Show resolved Hide resolved

docs/user/dql/functions.rst Outdated Show resolved Hide resolved

...arch/src/main/java/org/opensearch/sql/opensearch/storage/script/filter/lucene/LikeQuery.java Outdated Show resolved Hide resolved

Guian Gumpac added 3 commits November 24, 2022 14:46

Addressed PR comments regarding docs

34ffe13

Signed-off-by: Guian Gumpac <[email protected]>

Fixed backslashes in docs

eed3294

Signed-off-by: Guian Gumpac <[email protected]>

Added missed backticks in docs

86fa737

Signed-off-by: Guian Gumpac <[email protected]>

MaxKsyunz approved these changes Nov 24, 2022

View reviewed changes

Yury-Fridlyand reviewed Nov 25, 2022

View reviewed changes

core/src/main/java/org/opensearch/sql/expression/DSL.java Outdated Show resolved Hide resolved

Yury-Fridlyand approved these changes Nov 25, 2022

View reviewed changes

Guian Gumpac added 2 commits November 25, 2022 08:29

Moved escaping wildcard test to common/utils

b54a934

Signed-off-by: Guian Gumpac <[email protected]>

Fixed merge conflicts

fede123

Signed-off-by: Guian Gumpac <[email protected]>

Yury-Fridlyand approved these changes Nov 25, 2022

View reviewed changes

MaxKsyunz approved these changes Nov 25, 2022

View reviewed changes

Fixed checkstyle error

9da6140

Signed-off-by: Guian Gumpac <[email protected]>

GumpacG merged commit 2cd6921 into integ-add-wildcardquery Nov 25, 2022

margarit-h pushed a commit that referenced this pull request Dec 8, 2022

Add support for wildcard_query function to the new engine (#156) (ope…

2af7321

…nsearch-project#1108) Signed-off-by: Guian Gumpac <[email protected]>

GumpacG deleted the dev-add-wildcardquery branch January 21, 2023 17:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for wildcard_query function #156

Add support for wildcard_query function #156

GumpacG commented Nov 4, 2022 •

edited

Loading

codecov bot commented Nov 4, 2022 •

edited

Loading

Yury-Fridlyand commented Nov 14, 2022

acarbonetto commented Nov 22, 2022

MaxKsyunz Nov 24, 2022

MaxKsyunz Nov 24, 2022

GumpacG Nov 25, 2022

MaxKsyunz Nov 24, 2022

GumpacG Nov 25, 2022

Add support for wildcard_query function #156

Add support for wildcard_query function #156

Conversation

GumpacG commented Nov 4, 2022 • edited Loading

Description

Issues Resolved

Check List

codecov bot commented Nov 4, 2022 • edited Loading

Codecov Report

Yury-Fridlyand commented Nov 14, 2022

acarbonetto commented Nov 22, 2022

MaxKsyunz Nov 24, 2022

Choose a reason for hiding this comment

MaxKsyunz Nov 24, 2022

Choose a reason for hiding this comment

GumpacG Nov 25, 2022

Choose a reason for hiding this comment

MaxKsyunz Nov 24, 2022

Choose a reason for hiding this comment

GumpacG Nov 25, 2022

Choose a reason for hiding this comment

GumpacG commented Nov 4, 2022 •

edited

Loading

codecov bot commented Nov 4, 2022 •

edited

Loading