Lookup #2698

lukasz-soszynski-eliatra · 2024-05-29T17:55:34Z

Description

[Describe what this change achieves]
Implementation of the new lookup command.

Issues Resolved

[List any issues this PR will resolve]
#2651

Check List

New functionality includes testing.
- All tests pass, including unit test, integration test and doctest
New functionality has been documented.
- New functionality has javadoc added
- New functionality has user manual doc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

rupal-bq · 2024-06-07T18:39:02Z

Is this ready for review now? Can you please rebase with latest main to fix CI?

salyh · 2024-06-12T11:24:48Z

@rupal-bq I will rebase.

rupal-bq · 2024-06-13T06:16:51Z

Can you please add reference manual for lookup command? (e.g. https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/cmd/dedup.rst)

salyh · 2024-06-13T09:34:10Z

I rebased and squashed the commits and fixed DCO

salyh · 2024-06-25T11:25:27Z

Can you please add reference manual for lookup command? (e.g. https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/cmd/dedup.rst)

done

rupal-bq · 2024-07-01T20:53:19Z

@salyh some tests failed. Can you please take a look?

rupal-bq · 2024-07-02T05:37:37Z

docs/user/ppl/cmd/lookup.rst

+    | 19               | Jack     | finance    | true   |
+    +------------------+----------+------------+--------+
+
+    os> source=accounts | lookup hr employee_number AS account_number, dep AS department appendonly=true name AS given_name, active AS is_active ;


Can you please also add same example with appendonly=false ? I think that will help clarify how appendonly works. source=accounts | lookup hr employee_number AS account_number, dep AS department appendonly=false name AS given_name, active AS is_active ;

Will add it

rupal-bq · 2024-07-02T05:37:52Z

core/src/main/java/org/opensearch/sql/analysis/Analyzer.java

+    for (Map resultMap : inputMap) {
+      Expression origin = expressionAnalyzer.analyze(resultMap.getOrigin(), lookupTableContext);
+      if (resultMap.getTarget() instanceof Field) {
+        Expression targerExpression =


typo- targerExpression

We will fix this tomorrow together with a bugfix for this code segment we just discovered.

rupal-bq · 2024-07-02T05:47:20Z

...main/java/org/opensearch/sql/opensearch/executor/protector/OpenSearchExecutionProtector.java

+      for (Map.Entry<String, Object> f : matchMap.entrySet()) {
+        BoolQueryBuilder orQueryBuilder = new BoolQueryBuilder();
+
+        // Todo: Search with term and a match query? Or terms only?


Not sure about this. Can you share an example query in which term and match both are required?

When the lookup field in the lookup index is a "keyword" field (e.g. not analyzed) we need a term query. If the field is an analyzed field (full text field) we need a match query. Because we don't know the mapping of the field we just do both. Its way cheaper than first check the mapping.

rupal-bq · 2024-07-02T05:49:27Z

core/src/main/java/org/opensearch/sql/analysis/Analyzer.java

+  public LogicalPlan visitLookup(Lookup node, AnalysisContext queryContext) {
+    LogicalPlan child = node.getChild().get(0).accept(this, queryContext);
+    List<Argument> options = node.getOptions();
+    // Todo, refactor the option.


Why is this a TODO?

This is just copied from

sql/core/src/main/java/org/opensearch/sql/analysis/Analyzer.java

Line 495 in 4326396

// Todo, refactor the option.

and not added by us. Seems there is a geneal need to refactor the options - but thats not specific to this PR

salyh · 2024-07-03T15:17:54Z

@rupal-bq We have issues with the CI which are not caused by the code changes, for example:

 | Exception in thread "main" java.lang.IllegalStateException: opensearch-security requires Java 21:, your system: 17.0.11+9
| 	at org.opensearch.bootstrap.JarHell.checkJavaVersion(JarHell.java:278)
| 	at org.opensearch.plugins.PluginsService.verifyCompatibility(PluginsService.java:403)
| 	at org.opensearch.plugins.InstallPluginCommand.loadPluginInfo(InstallPluginCommand.java:819)
| 	at org.opensearch.plugins.InstallPluginCommand.installPlugin(InstallPluginCommand.java:874)
| 	at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:276)
| 	at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:250)
| 	at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
| 	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
| 	at org.opensearch.cli.MultiCommand.execute(MultiCommand.java:104)
| 	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
| 	at org.opensearch.cli.Command.main(Command.java:101)
| 	at org.opensearch.plugins.PluginCli.main(PluginCli.java:66)

https://github.com/opensearch-project/sql/actions/runs/9764080193/job/26953624605?pr=2698

rupal-bq · 2024-07-09T05:18:47Z

@rupal-bq We have issues with the CI which are not caused by the code changes, for example:

 | Exception in thread "main" java.lang.IllegalStateException: opensearch-security requires Java 21:, your system: 17.0.11+9
| 	at org.opensearch.bootstrap.JarHell.checkJavaVersion(JarHell.java:278)
| 	at org.opensearch.plugins.PluginsService.verifyCompatibility(PluginsService.java:403)
| 	at org.opensearch.plugins.InstallPluginCommand.loadPluginInfo(InstallPluginCommand.java:819)
| 	at org.opensearch.plugins.InstallPluginCommand.installPlugin(InstallPluginCommand.java:874)
| 	at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:276)
| 	at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:250)
| 	at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
| 	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
| 	at org.opensearch.cli.MultiCommand.execute(MultiCommand.java:104)
| 	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
| 	at org.opensearch.cli.Command.main(Command.java:101)
| 	at org.opensearch.plugins.PluginCli.main(PluginCli.java:66)

https://github.com/opensearch-project/sql/actions/runs/9764080193/job/26953624605?pr=2698

Issue is fixed in main now. Can you please rebase?

Signed-off-by: Hendrik Saly <[email protected]>

Signed-off-by: Lukasz Soszynski <[email protected]>

Signed-off-by: Hendrik Saly <[email protected]>

Signed-off-by: Lukasz Soszynski <[email protected]>

Signed-off-by: Hendrik Saly <[email protected]> Fix typo Signed-off-by: Hendrik Saly <[email protected]>

Signed-off-by: Lukasz Soszynski <[email protected]>

Signed-off-by: Hendrik Saly <[email protected]>

penghuo · 2024-07-18T22:50:44Z

docs/user/ppl/cmd/lookup.rst

@@ -0,0 +1,153 @@
+=============


to enable doc_test, please add lookup.rst to catagory.json ppl_cli section,

penghuo · 2024-07-18T23:05:18Z

docs/user/ppl/cmd/lookup.rst

+* lookup-index: mandatory. the name of the lookup index. If more than one is provided, all of them must match.
+* lookup-field: mandatory. the name of the lookup field. Must be existing in the lookup-index. It is used to match to a local field (in the current search) to get the lookup document. When there is no lookup document matching it is a no-op. If there is more than one an exception is thrown.
+* local-lookup-field: optional. the name of a field in the current search to match against the lookup-field. **Default:** value of lookup-field.
+* appendonly: optional. indicates if the values to copy over to the search result from the lookup document should overwrite existing values. If true no existing values are overwritten. **Default:** false.


why named it appendonly? does overwrite more straitforward?

Maybe - I have no personal preference here

penghuo · 2024-07-18T23:06:43Z

docs/user/ppl/cmd/lookup.rst

+* lookup-field: mandatory. the name of the lookup field. Must be existing in the lookup-index. It is used to match to a local field (in the current search) to get the lookup document. When there is no lookup document matching it is a no-op. If there is more than one an exception is thrown.
+* local-lookup-field: optional. the name of a field in the current search to match against the lookup-field. **Default:** value of lookup-field.
+* appendonly: optional. indicates if the values to copy over to the search result from the lookup document should overwrite existing values. If true no existing values are overwritten. **Default:** false.
+* source-field: optional. the fields to copy over from the lookup document to the search result. If no such fields are given all fields are copied. **Default:** all fields


could you add test on source-field?

if appendonly is ommit, how does parser idenitfy differenate lookup-field and source-field.

https://github.com/eliatra/sql/blob/f2410e8698b064dc6dfb90cbb054349ab4a0c719/integ-test/src/test/java/org/opensearch/sql/ppl/LookupCommandIT.java#L114

whitespace (and keep in mind that lookup-field ist mandatory). Can you provide a concrete example where we probbaly run into ambiguity

penghuo · 2024-07-18T23:16:57Z

opensearch/src/main/java/org/opensearch/sql/opensearch/storage/OpenSearchIndex.java

+
+    BiFunction<String, Map<String, Object>, Map<String, Object>> lookup() {
+
+      if (client.getNodeClient() == null) {


why we can not use OpenSearchClient?

penghuo · 2024-07-18T23:24:35Z

core/src/main/java/org/opensearch/sql/planner/physical/LookupOperator.java

+        finalMap.put("_copy", copyMap.keySet());
+      }
+
+      Map<String, Object> lookupResult = lookup.apply(indexName, finalMap);


The complexity is O(N), N is rows in input dataset? should we rewrite query as LEFT JOIN?
Do we have performance test result?

No peformance tests yet.
Running the lookup command as a join has also perf related implications like triggering the circuit breaker when both indices contains lot of documents and fields. If we have perf related concerns maybe keeping the current logic and adding a simple LRU cache is an option? And how would one implement then "appendonly" flag with a join?

…hIndex Signed-off-by: Lukasz Soszynski <[email protected]>

Signed-off-by: Lukasz Soszynski <[email protected]>

Signed-off-by: Hendrik Saly <[email protected]>

lukasz-soszynski-eliatra requested review from ps48, kavithacm, derek-ho, joshuali925, dai-chen, YANG-DB, rupal-bq, mengweieric, vmmusings, Swiddis, penghuo, seankao-az, MaxKsyunz, Yury-Fridlyand, anirudha, forestmvey, acarbonetto, GumpacG and ykmr1224 as code owners May 29, 2024 17:55

lukasz-soszynski-eliatra marked this pull request as draft May 30, 2024 08:09

salyh force-pushed the lookup branch 2 times, most recently from e7c8535 to d4f88c3 Compare June 13, 2024 09:33

lukasz-soszynski-eliatra marked this pull request as ready for review June 18, 2024 08:16

lukasz-soszynski-eliatra mentioned this pull request Jun 25, 2024

Lookup operator tests #2775

Closed

6 tasks

salyh force-pushed the lookup branch from 9c731d9 to d6a1fb4 Compare June 28, 2024 13:59

rupal-bq reviewed Jul 2, 2024

View reviewed changes

salyh force-pushed the lookup branch from 5b65e61 to 3a60e16 Compare July 9, 2024 09:24

lukasz-soszynski-eliatra mentioned this pull request Jul 9, 2024

Lookup simplified analyze copy fields #2809

Closed

6 tasks

salyh and others added 9 commits July 16, 2024 13:11

Initial commit for PPL lookup

039949e

Signed-off-by: Hendrik Saly <[email protected]>

Tests related to lookup and Analyzer class.

036e07a

Signed-off-by: Lukasz Soszynski <[email protected]>

Removed System.out leftover

1c2a8f6

Signed-off-by: Hendrik Saly <[email protected]>

Added implementation in QueryDataAnonymizer and fixed related tests

1de0bdf

Signed-off-by: Hendrik Saly <[email protected]>

Add documentation

cda7a80

Signed-off-by: Hendrik Saly <[email protected]>

Added more lookup operator tests

488c9af

Signed-off-by: Lukasz Soszynski <[email protected]>

Fix typo

c2edb51

Signed-off-by: Hendrik Saly <[email protected]> Fix typo Signed-off-by: Hendrik Saly <[email protected]>

One unit test and integration tests for the lookup operator.

9da10e3

Signed-off-by: Lukasz Soszynski <[email protected]>

Added "appendonly" example to docs

e9d8389

Signed-off-by: Hendrik Saly <[email protected]>

salyh force-pushed the lookup branch from 3a60e16 to e9d8389 Compare July 16, 2024 11:12

salyh requested a review from LantaoJin as a code owner July 16, 2024 11:12

Move implementation from ExecutionProtector to OpenSearchIndex class

0c4e488

Signed-off-by: Hendrik Saly <[email protected]>

salyh mentioned this pull request Jul 16, 2024

[FEATURE] PPL lookup Functionality #2651

Open

penghuo reviewed Jul 18, 2024

View reviewed changes

lukasz-soszynski-eliatra and others added 3 commits July 19, 2024 13:36

Corrections related to LookupCommandIT and tests related to OpenSearc…

ad96d22

…hIndex Signed-off-by: Lukasz Soszynski <[email protected]>

Typo corrected.

f2410e8

Signed-off-by: Lukasz Soszynski <[email protected]>

Enable doc test

127f22b

Signed-off-by: Hendrik Saly <[email protected]>

This was referenced Dec 2, 2024

Lookup join #3179

Closed

Lookup - ppl command #3181

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Lookup #2698

Lookup #2698

lukasz-soszynski-eliatra commented May 29, 2024

rupal-bq commented Jun 7, 2024 •

edited

Loading

salyh commented Jun 12, 2024

rupal-bq commented Jun 13, 2024

salyh commented Jun 13, 2024 •

edited

Loading

salyh commented Jun 25, 2024

rupal-bq commented Jul 1, 2024 •

edited

Loading

rupal-bq Jul 2, 2024

salyh Jul 2, 2024

salyh Jul 9, 2024

rupal-bq Jul 2, 2024

salyh Jul 2, 2024

rupal-bq Jul 2, 2024

salyh Jul 2, 2024

rupal-bq Jul 2, 2024

salyh Jul 2, 2024

salyh commented Jul 3, 2024

rupal-bq commented Jul 9, 2024

penghuo Jul 18, 2024

salyh Jul 23, 2024

penghuo Jul 18, 2024

salyh Jul 23, 2024

penghuo Jul 18, 2024

salyh Jul 23, 2024

penghuo Jul 18, 2024

penghuo Jul 18, 2024

salyh Jul 23, 2024


		BiFunction<String, Map<String, Object>, Map<String, Object>> lookup() {

		if (client.getNodeClient() == null) {

Lookup #2698

Are you sure you want to change the base?

Lookup #2698

Conversation

lukasz-soszynski-eliatra commented May 29, 2024

Description

Issues Resolved

Check List

rupal-bq commented Jun 7, 2024 • edited Loading

salyh commented Jun 12, 2024

rupal-bq commented Jun 13, 2024

salyh commented Jun 13, 2024 • edited Loading

salyh commented Jun 25, 2024

rupal-bq commented Jul 1, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

salyh commented Jul 3, 2024

rupal-bq commented Jul 9, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rupal-bq commented Jun 7, 2024 •

edited

Loading

salyh commented Jun 13, 2024 •

edited

Loading

rupal-bq commented Jul 1, 2024 •

edited

Loading