Add Nested Function Support in SELECT Clause #243

forestmvey · 2023-03-09T21:47:17Z

Description

Syntax: nested( [field} | [field,path] )

Nested function allows the query of nested types of an index. The path parameter is determined dynamically if not provided by the user, and the output query should be identical assuming the user input the correct parameter values. The condition parameter is not supported when the nested function is used in the SELECT clause. Nested types structure is flattened making the full path of an object the key, and the object it refers to the value.

   Sample input=
   keys = ['comments.likes']
   row = comments: {
   likes: 2
   }
   output =
   flattenedRow = {comment.likes: 2}

Example Queries

Simple nested query, array of `message.info` is flattened into two rows.

SELECT nested(message.info) FROM nested_objects;

After flattening notice `comment.data` has repeating `ab`

SELECT nested(message.info), nested(comment.data) FROM nested_objects;

someField is of type object

SELECT nested(message.info), someField FROM nested_objects;

To Do

nested function nested in other function call (Not currently supported in legacy)

Not Supported

Using condition parameter in the SELECT clause.

Changes from Legacy Functionality

Nested function does not support condition parameter in SELECT clause.
Updated partiql.rst to handle object arrays same as V2 engine.

Issues Resolved

Check List

New functionality includes testing.
- All tests pass, including unit test, integration test and doctest
New functionality has been documented.
- New functionality has javadoc added
- New functionality has user manual doc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

codecov · 2023-03-09T22:00:29Z

Codecov Report

Merging #243 (9b803d4) into integ-nested-select-clause (23cc0f6) will increase coverage by 0.03%.
The diff coverage is 100.00%.

❗ Current head 9b803d4 differs from pull request most recent head 48553f3. Consider uploading reports for the commit 48553f3 to get more accurate results

@@                       Coverage Diff                        @@
##             integ-nested-select-clause     #243      +/-   ##
================================================================
+ Coverage                         98.46%   98.50%   +0.03%     
- Complexity                         3869     3964      +95     
================================================================
  Files                               345      351       +6     
  Lines                              9603     9845     +242     
  Branches                            616      653      +37     
================================================================
+ Hits                               9456     9698     +242     
  Misses                              142      142              
  Partials                              5        5

Flag	Coverage Δ
sql-engine	`98.50% <100.00%> (+0.03%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...rg/opensearch/sql/analysis/ExpressionAnalyzer.java	`100.00% <ø> (ø)`
...ch/sql/planner/optimizer/LogicalPlanOptimizer.java	`100.00% <ø> (ø)`
...ain/java/org/opensearch/sql/analysis/Analyzer.java	`100.00% <100.00%> (ø)`
...va/org/opensearch/sql/analysis/NestedAnalyzer.java	`100.00% <100.00%> (ø)`
...c/main/java/org/opensearch/sql/expression/DSL.java	`100.00% <100.00%> (ø)`
...opensearch/sql/expression/ReferenceExpression.java	`100.00% <100.00%> (ø)`
...h/sql/expression/function/BuiltinFunctionName.java	`100.00% <100.00%> (ø)`
...ql/expression/function/NestedFunctionResolver.java	`100.00% <100.00%> (ø)`
...h/sql/expression/function/OpenSearchFunctions.java	`100.00% <100.00%> (ø)`
...ensearch/sql/expression/nested/NestedFunction.java	`100.00% <100.00%> (ø)`
... and 19 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Yury-Fridlyand

Cool!
Doctests? Demo? Nested party?

core/src/main/java/org/opensearch/sql/analysis/Analyzer.java

core/src/main/java/org/opensearch/sql/expression/nested/NestedFunctions.java

...rc/main/java/org/opensearch/sql/opensearch/storage/scan/OpenSearchIndexScanQueryBuilder.java

sql/src/main/java/org/opensearch/sql/sql/parser/AstBuilder.java

sql/src/main/java/org/opensearch/sql/sql/parser/AstExpressionBuilder.java

sql/src/main/java/org/opensearch/sql/sql/parser/AstUnnestBuilder.java

integ-test/src/test/java/org/opensearch/sql/sql/NestedIT.java

core/src/main/java/org/opensearch/sql/analysis/NestedAnalyzer.java

GumpacG · 2023-03-20T22:33:46Z

Nested with date type subfields are supported however it returns the wrong type. Maybe date types need to be included in NestedFunction.java.
Legacy:

V2:

sql/src/main/antlr/OpenSearchSQLParser.g4

GabeFernandez310 · 2023-03-22T21:26:14Z

Does documentation/doctests need to be added?

acarbonetto · 2023-03-23T20:54:36Z

core/src/main/java/org/opensearch/sql/planner/physical/UnnestOperator.java

+        result = flatten(field, inputValue, result, true);
+      }
+
+      if (result.isEmpty()) {


when does this happen?

Kept in place in-case of null values in data.

core/src/main/java/org/opensearch/sql/planner/physical/UnnestOperator.java

acarbonetto · 2023-03-23T21:02:25Z

core/src/main/java/org/opensearch/sql/planner/physical/UnnestOperator.java

+      ExprValue row,
+      List<Map<String,
+      ExprValue>> prevList,
+      boolean supportArrays


can we make supportArrays == false a separate function? Feels like these have different behaviours anyways

We can but we will need another function that is almost the same as getNested.

core/src/test/java/org/opensearch/sql/analysis/AnalyzerTest.java

sql/src/main/antlr/OpenSearchSQLParser.g4

forestmvey · 2023-03-27T23:42:51Z

Does documentation/doctests need to be added?

Added doctest in 10ee84d14f28b1c7e5bfa92a76e92cf9c2d74579

Yury-Fridlyand · 2023-03-29T01:44:24Z

core/src/main/java/org/opensearch/sql/analysis/Analyzer.java

+      NestedAnalyzer nestedAnalyzer = new NestedAnalyzer(
+          namedExpressions, expressionAnalyzer, child
+      );
+      child = nestedAnalyzer.analyze(expr, context);


You overwrite child set by highlightAnalyzer, which overwrites child set by windowAnalyzer, which .... Is it safe? Is nested compatible with highlight?
Do you have [IT] tests for that?

Don't believe we have any requirement for nested to work with added V2 functionality like highlight. Goal is legacy deprecation and we want to abandon the NLPChina implementation for a better standard like PartiQL. I can pushDown highlight and nested, I'm not sure what a valid query would be though?

select nested(...), highlight(...) from ... where match(...)
Can you try this?

Not working with Relevance Queries. Issue here for tracking.
1488

core/src/main/java/org/opensearch/sql/analysis/NestedAnalyzer.java

core/src/main/java/org/opensearch/sql/expression/nested/NestedFunction.java

docs/user/dql/functions.rst

doctest/test_mapping/nested_objects.json

integ-test/src/test/java/org/opensearch/sql/sql/NestedIT.java

sql/src/main/antlr/OpenSearchSQLParser.g4

core/src/test/java/org/opensearch/sql/planner/physical/UnnestOperatorTest.java

core/src/main/java/org/opensearch/sql/expression/function/OpenSearchFunctions.java

Signed-off-by: forestmvey <[email protected]>

Yury-Fridlyand reviewed Mar 9, 2023

View reviewed changes

GumpacG reviewed Mar 20, 2023

View reviewed changes

core/src/main/java/org/opensearch/sql/analysis/NestedAnalyzer.java Outdated Show resolved Hide resolved

forestmvey marked this pull request as ready for review March 20, 2023 18:38

forestmvey requested a review from a team March 20, 2023 18:38

GumpacG reviewed Mar 20, 2023

View reviewed changes

sql/src/main/antlr/OpenSearchSQLParser.g4 Show resolved Hide resolved

GumpacG reviewed Mar 21, 2023

View reviewed changes

sql/src/main/antlr/OpenSearchSQLParser.g4 Outdated Show resolved Hide resolved

forestmvey force-pushed the dev-nested-cleanup branch 2 times, most recently from e5221ba to 03cd356 Compare March 22, 2023 16:15

forestmvey requested a review from MaxKsyunz as a code owner March 22, 2023 16:15

acarbonetto approved these changes Mar 23, 2023

View reviewed changes

GabeFernandez310 reviewed Mar 24, 2023

View reviewed changes

sql/src/main/antlr/OpenSearchSQLParser.g4 Show resolved Hide resolved

GabeFernandez310 approved these changes Mar 28, 2023

View reviewed changes

forestmvey force-pushed the dev-nested-cleanup branch 3 times, most recently from c7229e9 to 26aa7c6 Compare March 28, 2023 22:17

Yury-Fridlyand reviewed Mar 29, 2023

View reviewed changes

forestmvey force-pushed the dev-nested-cleanup branch 2 times, most recently from 19f5c95 to dc718e2 Compare March 29, 2023 20:24

MaxKsyunz reviewed Mar 29, 2023

View reviewed changes

core/src/test/java/org/opensearch/sql/planner/physical/UnnestOperatorTest.java Outdated Show resolved Hide resolved

core/src/main/java/org/opensearch/sql/expression/function/OpenSearchFunctions.java Outdated Show resolved Hide resolved

Yury-Fridlyand approved these changes Mar 30, 2023

View reviewed changes

forestmvey force-pushed the dev-nested-cleanup branch from 9b803d4 to c02fa2a Compare March 30, 2023 18:02

Adding nested function support in SELECT clause.

48553f3

Signed-off-by: forestmvey <[email protected]>

forestmvey force-pushed the dev-nested-cleanup branch from c02fa2a to 48553f3 Compare March 30, 2023 21:07

forestmvey merged commit 4982979 into integ-nested-select-clause Mar 30, 2023

forestmvey deleted the dev-nested-cleanup branch March 30, 2023 21:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Nested Function Support in SELECT Clause #243

Add Nested Function Support in SELECT Clause #243

forestmvey commented Mar 9, 2023 •

edited

Loading

codecov bot commented Mar 9, 2023 •

edited

Loading

Yury-Fridlyand left a comment

GumpacG commented Mar 20, 2023 •

edited

Loading

GabeFernandez310 commented Mar 22, 2023

acarbonetto Mar 23, 2023

forestmvey Mar 27, 2023 •

edited

Loading

acarbonetto Mar 23, 2023

forestmvey Mar 27, 2023

forestmvey commented Mar 27, 2023

Yury-Fridlyand Mar 29, 2023

forestmvey Mar 29, 2023

Yury-Fridlyand Mar 30, 2023

forestmvey Mar 30, 2023 •

edited

Loading

Add Nested Function Support in SELECT Clause #243

Add Nested Function Support in SELECT Clause #243

Conversation

forestmvey commented Mar 9, 2023 • edited Loading

Description

Example Queries

Simple nested query, array of message.info is flattened into two rows.

After flattening notice comment.data has repeating ab

someField is of type object

To Do

Not Supported

Changes from Legacy Functionality

Issues Resolved

Check List

codecov bot commented Mar 9, 2023 • edited Loading

Codecov Report

Yury-Fridlyand left a comment

Choose a reason for hiding this comment

GumpacG commented Mar 20, 2023 • edited Loading

GabeFernandez310 commented Mar 22, 2023

acarbonetto Mar 23, 2023

Choose a reason for hiding this comment

forestmvey Mar 27, 2023 • edited Loading

Choose a reason for hiding this comment

acarbonetto Mar 23, 2023

Choose a reason for hiding this comment

forestmvey Mar 27, 2023

Choose a reason for hiding this comment

forestmvey commented Mar 27, 2023

Yury-Fridlyand Mar 29, 2023

Choose a reason for hiding this comment

forestmvey Mar 29, 2023

Choose a reason for hiding this comment

Yury-Fridlyand Mar 30, 2023

Choose a reason for hiding this comment

forestmvey Mar 30, 2023 • edited Loading

Choose a reason for hiding this comment

forestmvey commented Mar 9, 2023 •

edited

Loading

Simple nested query, array of `message.info` is flattened into two rows.

After flattening notice `comment.data` has repeating `ab`

codecov bot commented Mar 9, 2023 •

edited

Loading

GumpacG commented Mar 20, 2023 •

edited

Loading

forestmvey Mar 27, 2023 •

edited

Loading

forestmvey Mar 30, 2023 •

edited

Loading