Add pit for multi query #2753

rupal-bq · 2024-06-14T06:05:52Z

Description

Added option to use search after with pit instead of scroll api for multi query

Issues Resolved

Check List

New functionality includes testing.
- All tests pass, including unit test, integration test and doctest
New functionality has been documented.
- New functionality has javadoc added
- New functionality has user manual doc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Rupal Mahajan <[email protected]>

…l into join-search-after Signed-off-by: Rupal Mahajan <[email protected]>

Signed-off-by: Rupal Mahajan <[email protected]>

dai-chen · 2024-07-05T22:28:23Z

legacy/src/main/java/org/opensearch/sql/legacy/executor/multi/MinusExecutor.java

+        pit.create();
+      }
+
+      if (this.useTermsOptimization && this.fieldsOrderFirstTable.length != 1) {


The logic starting from this become even more complex than before. Could you explain why and think about if any better approach?

Here I am creating pit at start of run and deleting at end (similar to join). Only moved run code to try block but no changes in it.

I see. I recall we have this similar logic in your previous PR? If all ElasticHitsExecutor has such logic, can we move it to parent class?

No, not all ElasticHitsExecutor has similar logic. Join and multiquery has this pit logic but there is no scrolling in UnionExecutor so it doesn't have such logic.

legacy/src/main/java/org/opensearch/sql/legacy/executor/multi/MinusExecutor.java

Signed-off-by: Rupal Mahajan <[email protected]>

…search-after-pit Signed-off-by: Rupal Mahajan <[email protected]>

dai-chen · 2024-07-08T20:44:58Z

legacy/src/main/java/org/opensearch/sql/legacy/executor/multi/MinusExecutor.java

+        pit.create();
+      }
+
+      if (this.useTermsOptimization && this.fieldsOrderFirstTable.length != 1) {


I see. I recall we have this similar logic in your previous PR? If all ElasticHitsExecutor has such logic, can we move it to parent class?

legacy/src/main/java/org/opensearch/sql/legacy/executor/ElasticHitsExecutor.java

Signed-off-by: Rupal Mahajan <[email protected]>

dai-chen

Approved. This https://github.com/opensearch-project/sql/pull/2753/files#r1669287520 is still open.

Signed-off-by: Rupal Mahajan <[email protected]>

* Add pit for join queries (#2703) * Add search after for join Signed-off-by: Rupal Mahajan <[email protected]> * Enable search after by default Signed-off-by: Rupal Mahajan <[email protected]> * Add pit Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Fix tests Signed-off-by: Rupal Mahajan <[email protected]> * ignore joinWithGeoIntersectNL Signed-off-by: Rupal Mahajan <[email protected]> * Rerun CI with scroll Signed-off-by: Rupal Mahajan <[email protected]> * Remove unused code and retrigger CI with search_after true Signed-off-by: Rupal Mahajan <[email protected]> * Address comments Signed-off-by: Rupal Mahajan <[email protected]> * Remove unused code change Signed-off-by: Rupal Mahajan <[email protected]> * Update pit keep alive time with SQL_CURSOR_KEEP_ALIVE Signed-off-by: Rupal Mahajan <[email protected]> * Fix scroll condition Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Add pit before query execution Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Move pit from join request builder to executor Signed-off-by: Rupal Mahajan <[email protected]> * Remove unused methods Signed-off-by: Rupal Mahajan <[email protected]> * Add pit in parent class's run() Signed-off-by: Rupal Mahajan <[email protected]> * Add comment for fetching subsequent result in NestedLoopsElasticExecutor Signed-off-by: Rupal Mahajan <[email protected]> * Update comment Signed-off-by: Rupal Mahajan <[email protected]> * Add javadoc for pit handler Signed-off-by: Rupal Mahajan <[email protected]> * Add pit interface Signed-off-by: Rupal Mahajan <[email protected]> * Add pit handler unit test Signed-off-by: Rupal Mahajan <[email protected]> * Fix failed unit test CI Signed-off-by: Rupal Mahajan <[email protected]> * Fix spotless error Signed-off-by: Rupal Mahajan <[email protected]> * Rename pit class and add logs Signed-off-by: Rupal Mahajan <[email protected]> * Fix pit delete unit test Signed-off-by: Rupal Mahajan <[email protected]> --------- Signed-off-by: Rupal Mahajan <[email protected]> * Add pit for multi query (#2753) * Add search after for join Signed-off-by: Rupal Mahajan <[email protected]> * Enable search after by default Signed-off-by: Rupal Mahajan <[email protected]> * Add pit Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Fix tests Signed-off-by: Rupal Mahajan <[email protected]> * ignore joinWithGeoIntersectNL Signed-off-by: Rupal Mahajan <[email protected]> * Rerun CI with scroll Signed-off-by: Rupal Mahajan <[email protected]> * draft Signed-off-by: Rupal Mahajan <[email protected]> * Remove unused code and retrigger CI with search_after true Signed-off-by: Rupal Mahajan <[email protected]> * Address comments Signed-off-by: Rupal Mahajan <[email protected]> * Remove unused code change Signed-off-by: Rupal Mahajan <[email protected]> * Update pit keep alive time with SQL_CURSOR_KEEP_ALIVE Signed-off-by: Rupal Mahajan <[email protected]> * Fix scroll condition Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Add pit before query execution Signed-off-by: Rupal Mahajan <[email protected]> * Refactor get response with pit method Signed-off-by: Rupal Mahajan <[email protected]> * Update remaining scroll search calls Signed-off-by: Rupal Mahajan <[email protected]> * Fix integ test failures Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Move pit from join request builder to executor Signed-off-by: Rupal Mahajan <[email protected]> * Remove unused methods Signed-off-by: Rupal Mahajan <[email protected]> * Move pit from request to executor Signed-off-by: Rupal Mahajan <[email protected]> * Fix pit.delete call missed while merge Signed-off-by: Rupal Mahajan <[email protected]> * Move getResponseWithHits method to util class Signed-off-by: Rupal Mahajan <[email protected]> * add try catch for create delete pit in minus executor Signed-off-by: Rupal Mahajan <[email protected]> * move all common fields to ElasticHitsExecutor Signed-off-by: Rupal Mahajan <[email protected]> * add javadoc for ElasticHitsExecutor Signed-off-by: Rupal Mahajan <[email protected]> * Add missing javadoc Signed-off-by: Rupal Mahajan <[email protected]> * Forcing an empty commit as last commit is stuck processing updates Signed-off-by: Rupal Mahajan <[email protected]> --------- Signed-off-by: Rupal Mahajan <[email protected]> --------- Signed-off-by: Rupal Mahajan <[email protected]> Signed-off-by: Manasvini B S <[email protected]> Co-authored-by: Rupal Mahajan <[email protected]>

* Add pit for join queries (#2703) * Add search after for join Signed-off-by: Rupal Mahajan <[email protected]> * Enable search after by default Signed-off-by: Rupal Mahajan <[email protected]> * Add pit Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Fix tests Signed-off-by: Rupal Mahajan <[email protected]> * ignore joinWithGeoIntersectNL Signed-off-by: Rupal Mahajan <[email protected]> * Rerun CI with scroll Signed-off-by: Rupal Mahajan <[email protected]> * Remove unused code and retrigger CI with search_after true Signed-off-by: Rupal Mahajan <[email protected]> * Address comments Signed-off-by: Rupal Mahajan <[email protected]> * Remove unused code change Signed-off-by: Rupal Mahajan <[email protected]> * Update pit keep alive time with SQL_CURSOR_KEEP_ALIVE Signed-off-by: Rupal Mahajan <[email protected]> * Fix scroll condition Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Add pit before query execution Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Move pit from join request builder to executor Signed-off-by: Rupal Mahajan <[email protected]> * Remove unused methods Signed-off-by: Rupal Mahajan <[email protected]> * Add pit in parent class's run() Signed-off-by: Rupal Mahajan <[email protected]> * Add comment for fetching subsequent result in NestedLoopsElasticExecutor Signed-off-by: Rupal Mahajan <[email protected]> * Update comment Signed-off-by: Rupal Mahajan <[email protected]> * Add javadoc for pit handler Signed-off-by: Rupal Mahajan <[email protected]> * Add pit interface Signed-off-by: Rupal Mahajan <[email protected]> * Add pit handler unit test Signed-off-by: Rupal Mahajan <[email protected]> * Fix failed unit test CI Signed-off-by: Rupal Mahajan <[email protected]> * Fix spotless error Signed-off-by: Rupal Mahajan <[email protected]> * Rename pit class and add logs Signed-off-by: Rupal Mahajan <[email protected]> * Fix pit delete unit test Signed-off-by: Rupal Mahajan <[email protected]> --------- Signed-off-by: Rupal Mahajan <[email protected]> * Add pit for multi query (#2753) * Add search after for join Signed-off-by: Rupal Mahajan <[email protected]> * Enable search after by default Signed-off-by: Rupal Mahajan <[email protected]> * Add pit Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Fix tests Signed-off-by: Rupal Mahajan <[email protected]> * ignore joinWithGeoIntersectNL Signed-off-by: Rupal Mahajan <[email protected]> * Rerun CI with scroll Signed-off-by: Rupal Mahajan <[email protected]> * draft Signed-off-by: Rupal Mahajan <[email protected]> * Remove unused code and retrigger CI with search_after true Signed-off-by: Rupal Mahajan <[email protected]> * Address comments Signed-off-by: Rupal Mahajan <[email protected]> * Remove unused code change Signed-off-by: Rupal Mahajan <[email protected]> * Update pit keep alive time with SQL_CURSOR_KEEP_ALIVE Signed-off-by: Rupal Mahajan <[email protected]> * Fix scroll condition Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Add pit before query execution Signed-off-by: Rupal Mahajan <[email protected]> * Refactor get response with pit method Signed-off-by: Rupal Mahajan <[email protected]> * Update remaining scroll search calls Signed-off-by: Rupal Mahajan <[email protected]> * Fix integ test failures Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Move pit from join request builder to executor Signed-off-by: Rupal Mahajan <[email protected]> * Remove unused methods Signed-off-by: Rupal Mahajan <[email protected]> * Move pit from request to executor Signed-off-by: Rupal Mahajan <[email protected]> * Fix pit.delete call missed while merge Signed-off-by: Rupal Mahajan <[email protected]> * Move getResponseWithHits method to util class Signed-off-by: Rupal Mahajan <[email protected]> * add try catch for create delete pit in minus executor Signed-off-by: Rupal Mahajan <[email protected]> * move all common fields to ElasticHitsExecutor Signed-off-by: Rupal Mahajan <[email protected]> * add javadoc for ElasticHitsExecutor Signed-off-by: Rupal Mahajan <[email protected]> * Add missing javadoc Signed-off-by: Rupal Mahajan <[email protected]> * Forcing an empty commit as last commit is stuck processing updates Signed-off-by: Rupal Mahajan <[email protected]> --------- Signed-off-by: Rupal Mahajan <[email protected]> --------- Signed-off-by: Rupal Mahajan <[email protected]> Signed-off-by: Manasvini B S <[email protected]> Co-authored-by: Rupal Mahajan <[email protected]> (cherry picked from commit 7815c96) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Add search after for join Signed-off-by: Rupal Mahajan <[email protected]> * Enable search after by default Signed-off-by: Rupal Mahajan <[email protected]> * Add pit Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Fix tests Signed-off-by: Rupal Mahajan <[email protected]> * ignore joinWithGeoIntersectNL Signed-off-by: Rupal Mahajan <[email protected]> * Rerun CI with scroll Signed-off-by: Rupal Mahajan <[email protected]> * draft Signed-off-by: Rupal Mahajan <[email protected]> * Remove unused code and retrigger CI with search_after true Signed-off-by: Rupal Mahajan <[email protected]> * Address comments Signed-off-by: Rupal Mahajan <[email protected]> * Remove unused code change Signed-off-by: Rupal Mahajan <[email protected]> * Update pit keep alive time with SQL_CURSOR_KEEP_ALIVE Signed-off-by: Rupal Mahajan <[email protected]> * Fix scroll condition Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Add pit before query execution Signed-off-by: Rupal Mahajan <[email protected]> * Refactor get response with pit method Signed-off-by: Rupal Mahajan <[email protected]> * Update remaining scroll search calls Signed-off-by: Rupal Mahajan <[email protected]> * Fix integ test failures Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Move pit from join request builder to executor Signed-off-by: Rupal Mahajan <[email protected]> * Remove unused methods Signed-off-by: Rupal Mahajan <[email protected]> * Move pit from request to executor Signed-off-by: Rupal Mahajan <[email protected]> * Fix pit.delete call missed while merge Signed-off-by: Rupal Mahajan <[email protected]> * Move getResponseWithHits method to util class Signed-off-by: Rupal Mahajan <[email protected]> * add try catch for create delete pit in minus executor Signed-off-by: Rupal Mahajan <[email protected]> * move all common fields to ElasticHitsExecutor Signed-off-by: Rupal Mahajan <[email protected]> * add javadoc for ElasticHitsExecutor Signed-off-by: Rupal Mahajan <[email protected]> * Add missing javadoc Signed-off-by: Rupal Mahajan <[email protected]> * Forcing an empty commit as last commit is stuck processing updates Signed-off-by: Rupal Mahajan <[email protected]> --------- Signed-off-by: Rupal Mahajan <[email protected]>

* Add pit for join queries (#2703) * Add search after for join Signed-off-by: Rupal Mahajan <[email protected]> * Enable search after by default Signed-off-by: Rupal Mahajan <[email protected]> * Add pit Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Fix tests Signed-off-by: Rupal Mahajan <[email protected]> * ignore joinWithGeoIntersectNL Signed-off-by: Rupal Mahajan <[email protected]> * Rerun CI with scroll Signed-off-by: Rupal Mahajan <[email protected]> * Remove unused code and retrigger CI with search_after true Signed-off-by: Rupal Mahajan <[email protected]> * Address comments Signed-off-by: Rupal Mahajan <[email protected]> * Remove unused code change Signed-off-by: Rupal Mahajan <[email protected]> * Update pit keep alive time with SQL_CURSOR_KEEP_ALIVE Signed-off-by: Rupal Mahajan <[email protected]> * Fix scroll condition Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Add pit before query execution Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Move pit from join request builder to executor Signed-off-by: Rupal Mahajan <[email protected]> * Remove unused methods Signed-off-by: Rupal Mahajan <[email protected]> * Add pit in parent class's run() Signed-off-by: Rupal Mahajan <[email protected]> * Add comment for fetching subsequent result in NestedLoopsElasticExecutor Signed-off-by: Rupal Mahajan <[email protected]> * Update comment Signed-off-by: Rupal Mahajan <[email protected]> * Add javadoc for pit handler Signed-off-by: Rupal Mahajan <[email protected]> * Add pit interface Signed-off-by: Rupal Mahajan <[email protected]> * Add pit handler unit test Signed-off-by: Rupal Mahajan <[email protected]> * Fix failed unit test CI Signed-off-by: Rupal Mahajan <[email protected]> * Fix spotless error Signed-off-by: Rupal Mahajan <[email protected]> * Rename pit class and add logs Signed-off-by: Rupal Mahajan <[email protected]> * Fix pit delete unit test Signed-off-by: Rupal Mahajan <[email protected]> --------- Signed-off-by: Rupal Mahajan <[email protected]> * Add pit for multi query (#2753) * Add search after for join Signed-off-by: Rupal Mahajan <[email protected]> * Enable search after by default Signed-off-by: Rupal Mahajan <[email protected]> * Add pit Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Fix tests Signed-off-by: Rupal Mahajan <[email protected]> * ignore joinWithGeoIntersectNL Signed-off-by: Rupal Mahajan <[email protected]> * Rerun CI with scroll Signed-off-by: Rupal Mahajan <[email protected]> * draft Signed-off-by: Rupal Mahajan <[email protected]> * Remove unused code and retrigger CI with search_after true Signed-off-by: Rupal Mahajan <[email protected]> * Address comments Signed-off-by: Rupal Mahajan <[email protected]> * Remove unused code change Signed-off-by: Rupal Mahajan <[email protected]> * Update pit keep alive time with SQL_CURSOR_KEEP_ALIVE Signed-off-by: Rupal Mahajan <[email protected]> * Fix scroll condition Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Add pit before query execution Signed-off-by: Rupal Mahajan <[email protected]> * Refactor get response with pit method Signed-off-by: Rupal Mahajan <[email protected]> * Update remaining scroll search calls Signed-off-by: Rupal Mahajan <[email protected]> * Fix integ test failures Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Move pit from join request builder to executor Signed-off-by: Rupal Mahajan <[email protected]> * Remove unused methods Signed-off-by: Rupal Mahajan <[email protected]> * Move pit from request to executor Signed-off-by: Rupal Mahajan <[email protected]> * Fix pit.delete call missed while merge Signed-off-by: Rupal Mahajan <[email protected]> * Move getResponseWithHits method to util class Signed-off-by: Rupal Mahajan <[email protected]> * add try catch for create delete pit in minus executor Signed-off-by: Rupal Mahajan <[email protected]> * move all common fields to ElasticHitsExecutor Signed-off-by: Rupal Mahajan <[email protected]> * add javadoc for ElasticHitsExecutor Signed-off-by: Rupal Mahajan <[email protected]> * Add missing javadoc Signed-off-by: Rupal Mahajan <[email protected]> * Forcing an empty commit as last commit is stuck processing updates Signed-off-by: Rupal Mahajan <[email protected]> --------- Signed-off-by: Rupal Mahajan <[email protected]> * Add pit to default cursor Signed-off-by: Rupal Mahajan <[email protected]> * Run CI without pit unit test Signed-off-by: Rupal Mahajan <[email protected]> * Rerun CI without pit unit test Signed-off-by: Rupal Mahajan <[email protected]> * FIx unit tests for PIT changes Signed-off-by: Manasvini B S <[email protected]> * Addressed comments Signed-off-by: Manasvini B S <[email protected]> --------- Signed-off-by: Rupal Mahajan <[email protected]> Signed-off-by: Manasvini B S <[email protected]> Co-authored-by: Rupal Mahajan <[email protected]>

* Add pit for join queries (opensearch-project#2703) * Add search after for join Signed-off-by: Rupal Mahajan <[email protected]> * Enable search after by default Signed-off-by: Rupal Mahajan <[email protected]> * Add pit Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Fix tests Signed-off-by: Rupal Mahajan <[email protected]> * ignore joinWithGeoIntersectNL Signed-off-by: Rupal Mahajan <[email protected]> * Rerun CI with scroll Signed-off-by: Rupal Mahajan <[email protected]> * Remove unused code and retrigger CI with search_after true Signed-off-by: Rupal Mahajan <[email protected]> * Address comments Signed-off-by: Rupal Mahajan <[email protected]> * Remove unused code change Signed-off-by: Rupal Mahajan <[email protected]> * Update pit keep alive time with SQL_CURSOR_KEEP_ALIVE Signed-off-by: Rupal Mahajan <[email protected]> * Fix scroll condition Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Add pit before query execution Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Move pit from join request builder to executor Signed-off-by: Rupal Mahajan <[email protected]> * Remove unused methods Signed-off-by: Rupal Mahajan <[email protected]> * Add pit in parent class's run() Signed-off-by: Rupal Mahajan <[email protected]> * Add comment for fetching subsequent result in NestedLoopsElasticExecutor Signed-off-by: Rupal Mahajan <[email protected]> * Update comment Signed-off-by: Rupal Mahajan <[email protected]> * Add javadoc for pit handler Signed-off-by: Rupal Mahajan <[email protected]> * Add pit interface Signed-off-by: Rupal Mahajan <[email protected]> * Add pit handler unit test Signed-off-by: Rupal Mahajan <[email protected]> * Fix failed unit test CI Signed-off-by: Rupal Mahajan <[email protected]> * Fix spotless error Signed-off-by: Rupal Mahajan <[email protected]> * Rename pit class and add logs Signed-off-by: Rupal Mahajan <[email protected]> * Fix pit delete unit test Signed-off-by: Rupal Mahajan <[email protected]> --------- Signed-off-by: Rupal Mahajan <[email protected]> * Add pit for multi query (opensearch-project#2753) * Add search after for join Signed-off-by: Rupal Mahajan <[email protected]> * Enable search after by default Signed-off-by: Rupal Mahajan <[email protected]> * Add pit Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Fix tests Signed-off-by: Rupal Mahajan <[email protected]> * ignore joinWithGeoIntersectNL Signed-off-by: Rupal Mahajan <[email protected]> * Rerun CI with scroll Signed-off-by: Rupal Mahajan <[email protected]> * draft Signed-off-by: Rupal Mahajan <[email protected]> * Remove unused code and retrigger CI with search_after true Signed-off-by: Rupal Mahajan <[email protected]> * Address comments Signed-off-by: Rupal Mahajan <[email protected]> * Remove unused code change Signed-off-by: Rupal Mahajan <[email protected]> * Update pit keep alive time with SQL_CURSOR_KEEP_ALIVE Signed-off-by: Rupal Mahajan <[email protected]> * Fix scroll condition Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Add pit before query execution Signed-off-by: Rupal Mahajan <[email protected]> * Refactor get response with pit method Signed-off-by: Rupal Mahajan <[email protected]> * Update remaining scroll search calls Signed-off-by: Rupal Mahajan <[email protected]> * Fix integ test failures Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Move pit from join request builder to executor Signed-off-by: Rupal Mahajan <[email protected]> * Remove unused methods Signed-off-by: Rupal Mahajan <[email protected]> * Move pit from request to executor Signed-off-by: Rupal Mahajan <[email protected]> * Fix pit.delete call missed while merge Signed-off-by: Rupal Mahajan <[email protected]> * Move getResponseWithHits method to util class Signed-off-by: Rupal Mahajan <[email protected]> * add try catch for create delete pit in minus executor Signed-off-by: Rupal Mahajan <[email protected]> * move all common fields to ElasticHitsExecutor Signed-off-by: Rupal Mahajan <[email protected]> * add javadoc for ElasticHitsExecutor Signed-off-by: Rupal Mahajan <[email protected]> * Add missing javadoc Signed-off-by: Rupal Mahajan <[email protected]> * Forcing an empty commit as last commit is stuck processing updates Signed-off-by: Rupal Mahajan <[email protected]> --------- Signed-off-by: Rupal Mahajan <[email protected]> --------- Signed-off-by: Rupal Mahajan <[email protected]> Signed-off-by: Manasvini B S <[email protected]> Co-authored-by: Rupal Mahajan <[email protected]>

* Add pit for join queries (opensearch-project#2703) * Add search after for join Signed-off-by: Rupal Mahajan <[email protected]> * Enable search after by default Signed-off-by: Rupal Mahajan <[email protected]> * Add pit Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Fix tests Signed-off-by: Rupal Mahajan <[email protected]> * ignore joinWithGeoIntersectNL Signed-off-by: Rupal Mahajan <[email protected]> * Rerun CI with scroll Signed-off-by: Rupal Mahajan <[email protected]> * Remove unused code and retrigger CI with search_after true Signed-off-by: Rupal Mahajan <[email protected]> * Address comments Signed-off-by: Rupal Mahajan <[email protected]> * Remove unused code change Signed-off-by: Rupal Mahajan <[email protected]> * Update pit keep alive time with SQL_CURSOR_KEEP_ALIVE Signed-off-by: Rupal Mahajan <[email protected]> * Fix scroll condition Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Add pit before query execution Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Move pit from join request builder to executor Signed-off-by: Rupal Mahajan <[email protected]> * Remove unused methods Signed-off-by: Rupal Mahajan <[email protected]> * Add pit in parent class's run() Signed-off-by: Rupal Mahajan <[email protected]> * Add comment for fetching subsequent result in NestedLoopsElasticExecutor Signed-off-by: Rupal Mahajan <[email protected]> * Update comment Signed-off-by: Rupal Mahajan <[email protected]> * Add javadoc for pit handler Signed-off-by: Rupal Mahajan <[email protected]> * Add pit interface Signed-off-by: Rupal Mahajan <[email protected]> * Add pit handler unit test Signed-off-by: Rupal Mahajan <[email protected]> * Fix failed unit test CI Signed-off-by: Rupal Mahajan <[email protected]> * Fix spotless error Signed-off-by: Rupal Mahajan <[email protected]> * Rename pit class and add logs Signed-off-by: Rupal Mahajan <[email protected]> * Fix pit delete unit test Signed-off-by: Rupal Mahajan <[email protected]> --------- Signed-off-by: Rupal Mahajan <[email protected]> * Add pit for multi query (opensearch-project#2753) * Add search after for join Signed-off-by: Rupal Mahajan <[email protected]> * Enable search after by default Signed-off-by: Rupal Mahajan <[email protected]> * Add pit Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Fix tests Signed-off-by: Rupal Mahajan <[email protected]> * ignore joinWithGeoIntersectNL Signed-off-by: Rupal Mahajan <[email protected]> * Rerun CI with scroll Signed-off-by: Rupal Mahajan <[email protected]> * draft Signed-off-by: Rupal Mahajan <[email protected]> * Remove unused code and retrigger CI with search_after true Signed-off-by: Rupal Mahajan <[email protected]> * Address comments Signed-off-by: Rupal Mahajan <[email protected]> * Remove unused code change Signed-off-by: Rupal Mahajan <[email protected]> * Update pit keep alive time with SQL_CURSOR_KEEP_ALIVE Signed-off-by: Rupal Mahajan <[email protected]> * Fix scroll condition Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Add pit before query execution Signed-off-by: Rupal Mahajan <[email protected]> * Refactor get response with pit method Signed-off-by: Rupal Mahajan <[email protected]> * Update remaining scroll search calls Signed-off-by: Rupal Mahajan <[email protected]> * Fix integ test failures Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Move pit from join request builder to executor Signed-off-by: Rupal Mahajan <[email protected]> * Remove unused methods Signed-off-by: Rupal Mahajan <[email protected]> * Move pit from request to executor Signed-off-by: Rupal Mahajan <[email protected]> * Fix pit.delete call missed while merge Signed-off-by: Rupal Mahajan <[email protected]> * Move getResponseWithHits method to util class Signed-off-by: Rupal Mahajan <[email protected]> * add try catch for create delete pit in minus executor Signed-off-by: Rupal Mahajan <[email protected]> * move all common fields to ElasticHitsExecutor Signed-off-by: Rupal Mahajan <[email protected]> * add javadoc for ElasticHitsExecutor Signed-off-by: Rupal Mahajan <[email protected]> * Add missing javadoc Signed-off-by: Rupal Mahajan <[email protected]> * Forcing an empty commit as last commit is stuck processing updates Signed-off-by: Rupal Mahajan <[email protected]> --------- Signed-off-by: Rupal Mahajan <[email protected]> * Add pit to default cursor Signed-off-by: Rupal Mahajan <[email protected]> * Run CI without pit unit test Signed-off-by: Rupal Mahajan <[email protected]> * Rerun CI without pit unit test Signed-off-by: Rupal Mahajan <[email protected]> * FIx unit tests for PIT changes Signed-off-by: Manasvini B S <[email protected]> * Addressed comments Signed-off-by: Manasvini B S <[email protected]> --------- Signed-off-by: Rupal Mahajan <[email protected]> Signed-off-by: Manasvini B S <[email protected]> Co-authored-by: Rupal Mahajan <[email protected]>

* Add pit for join queries (#2703) * Add search after for join * Enable search after by default * Add pit * nit * Fix tests * ignore joinWithGeoIntersectNL * Rerun CI with scroll * Remove unused code and retrigger CI with search_after true * Address comments * Remove unused code change * Update pit keep alive time with SQL_CURSOR_KEEP_ALIVE * Fix scroll condition * nit * Add pit before query execution * nit * Move pit from join request builder to executor * Remove unused methods * Add pit in parent class's run() * Add comment for fetching subsequent result in NestedLoopsElasticExecutor * Update comment * Add javadoc for pit handler * Add pit interface * Add pit handler unit test * Fix failed unit test CI * Fix spotless error * Rename pit class and add logs * Fix pit delete unit test --------- * Add pit for multi query (#2753) * Add search after for join * Enable search after by default * Add pit * nit * Fix tests * ignore joinWithGeoIntersectNL * Rerun CI with scroll * draft * Remove unused code and retrigger CI with search_after true * Address comments * Remove unused code change * Update pit keep alive time with SQL_CURSOR_KEEP_ALIVE * Fix scroll condition * nit * Add pit before query execution * Refactor get response with pit method * Update remaining scroll search calls * Fix integ test failures * nit * Move pit from join request builder to executor * Remove unused methods * Move pit from request to executor * Fix pit.delete call missed while merge * Move getResponseWithHits method to util class * add try catch for create delete pit in minus executor * move all common fields to ElasticHitsExecutor * add javadoc for ElasticHitsExecutor * Add missing javadoc * Forcing an empty commit as last commit is stuck processing updates --------- --------- (cherry picked from commit 7815c96) Signed-off-by: Rupal Mahajan <[email protected]> Signed-off-by: Manasvini B S <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Rupal Mahajan <[email protected]>

* Add pit for join queries (#2703) * Add search after for join Signed-off-by: Rupal Mahajan <[email protected]> * Enable search after by default Signed-off-by: Rupal Mahajan <[email protected]> * Add pit Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Fix tests Signed-off-by: Rupal Mahajan <[email protected]> * ignore joinWithGeoIntersectNL Signed-off-by: Rupal Mahajan <[email protected]> * Rerun CI with scroll Signed-off-by: Rupal Mahajan <[email protected]> * Remove unused code and retrigger CI with search_after true Signed-off-by: Rupal Mahajan <[email protected]> * Address comments Signed-off-by: Rupal Mahajan <[email protected]> * Remove unused code change Signed-off-by: Rupal Mahajan <[email protected]> * Update pit keep alive time with SQL_CURSOR_KEEP_ALIVE Signed-off-by: Rupal Mahajan <[email protected]> * Fix scroll condition Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Add pit before query execution Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Move pit from join request builder to executor Signed-off-by: Rupal Mahajan <[email protected]> * Remove unused methods Signed-off-by: Rupal Mahajan <[email protected]> * Add pit in parent class's run() Signed-off-by: Rupal Mahajan <[email protected]> * Add comment for fetching subsequent result in NestedLoopsElasticExecutor Signed-off-by: Rupal Mahajan <[email protected]> * Update comment Signed-off-by: Rupal Mahajan <[email protected]> * Add javadoc for pit handler Signed-off-by: Rupal Mahajan <[email protected]> * Add pit interface Signed-off-by: Rupal Mahajan <[email protected]> * Add pit handler unit test Signed-off-by: Rupal Mahajan <[email protected]> * Fix failed unit test CI Signed-off-by: Rupal Mahajan <[email protected]> * Fix spotless error Signed-off-by: Rupal Mahajan <[email protected]> * Rename pit class and add logs Signed-off-by: Rupal Mahajan <[email protected]> * Fix pit delete unit test Signed-off-by: Rupal Mahajan <[email protected]> --------- Signed-off-by: Rupal Mahajan <[email protected]> * Add pit for multi query (#2753) * Add search after for join Signed-off-by: Rupal Mahajan <[email protected]> * Enable search after by default Signed-off-by: Rupal Mahajan <[email protected]> * Add pit Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Fix tests Signed-off-by: Rupal Mahajan <[email protected]> * ignore joinWithGeoIntersectNL Signed-off-by: Rupal Mahajan <[email protected]> * Rerun CI with scroll Signed-off-by: Rupal Mahajan <[email protected]> * draft Signed-off-by: Rupal Mahajan <[email protected]> * Remove unused code and retrigger CI with search_after true Signed-off-by: Rupal Mahajan <[email protected]> * Address comments Signed-off-by: Rupal Mahajan <[email protected]> * Remove unused code change Signed-off-by: Rupal Mahajan <[email protected]> * Update pit keep alive time with SQL_CURSOR_KEEP_ALIVE Signed-off-by: Rupal Mahajan <[email protected]> * Fix scroll condition Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Add pit before query execution Signed-off-by: Rupal Mahajan <[email protected]> * Refactor get response with pit method Signed-off-by: Rupal Mahajan <[email protected]> * Update remaining scroll search calls Signed-off-by: Rupal Mahajan <[email protected]> * Fix integ test failures Signed-off-by: Rupal Mahajan <[email protected]> * nit Signed-off-by: Rupal Mahajan <[email protected]> * Move pit from join request builder to executor Signed-off-by: Rupal Mahajan <[email protected]> * Remove unused methods Signed-off-by: Rupal Mahajan <[email protected]> * Move pit from request to executor Signed-off-by: Rupal Mahajan <[email protected]> * Fix pit.delete call missed while merge Signed-off-by: Rupal Mahajan <[email protected]> * Move getResponseWithHits method to util class Signed-off-by: Rupal Mahajan <[email protected]> * add try catch for create delete pit in minus executor Signed-off-by: Rupal Mahajan <[email protected]> * move all common fields to ElasticHitsExecutor Signed-off-by: Rupal Mahajan <[email protected]> * add javadoc for ElasticHitsExecutor Signed-off-by: Rupal Mahajan <[email protected]> * Add missing javadoc Signed-off-by: Rupal Mahajan <[email protected]> * Forcing an empty commit as last commit is stuck processing updates Signed-off-by: Rupal Mahajan <[email protected]> --------- Signed-off-by: Rupal Mahajan <[email protected]> * Add pit to default cursor Signed-off-by: Rupal Mahajan <[email protected]> * Run CI without pit unit test Signed-off-by: Rupal Mahajan <[email protected]> * Rerun CI without pit unit test Signed-off-by: Rupal Mahajan <[email protected]> * FIx unit tests for PIT changes Signed-off-by: Manasvini B S <[email protected]> * Addressed comments Signed-off-by: Manasvini B S <[email protected]> --------- Signed-off-by: Rupal Mahajan <[email protected]> Signed-off-by: Manasvini B S <[email protected]> Co-authored-by: Rupal Mahajan <[email protected]> (cherry picked from commit 69853fe) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Add pit for join queries (#2703) * Add search after for join * Enable search after by default * Add pit * nit * Fix tests * ignore joinWithGeoIntersectNL * Rerun CI with scroll * Remove unused code and retrigger CI with search_after true * Address comments * Remove unused code change * Update pit keep alive time with SQL_CURSOR_KEEP_ALIVE * Fix scroll condition * nit * Add pit before query execution * nit * Move pit from join request builder to executor * Remove unused methods * Add pit in parent class's run() * Add comment for fetching subsequent result in NestedLoopsElasticExecutor * Update comment * Add javadoc for pit handler * Add pit interface * Add pit handler unit test * Fix failed unit test CI * Fix spotless error * Rename pit class and add logs * Fix pit delete unit test --------- * Add pit for multi query (#2753) * Add search after for join * Enable search after by default * Add pit * nit * Fix tests * ignore joinWithGeoIntersectNL * Rerun CI with scroll * draft * Remove unused code and retrigger CI with search_after true * Address comments * Remove unused code change * Update pit keep alive time with SQL_CURSOR_KEEP_ALIVE * Fix scroll condition * nit * Add pit before query execution * Refactor get response with pit method * Update remaining scroll search calls * Fix integ test failures * nit * Move pit from join request builder to executor * Remove unused methods * Move pit from request to executor * Fix pit.delete call missed while merge * Move getResponseWithHits method to util class * add try catch for create delete pit in minus executor * move all common fields to ElasticHitsExecutor * add javadoc for ElasticHitsExecutor * Add missing javadoc * Forcing an empty commit as last commit is stuck processing updates --------- * Add pit to default cursor * Run CI without pit unit test * Rerun CI without pit unit test * FIx unit tests for PIT changes * Addressed comments --------- (cherry picked from commit 69853fe) Signed-off-by: Rupal Mahajan <[email protected]> Signed-off-by: Manasvini B S <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Rupal Mahajan <[email protected]>

rupal-bq added 30 commits May 28, 2024 23:29

Add search after for join

e225977

Signed-off-by: Rupal Mahajan <[email protected]>

Enable search after by default

d69feda

Signed-off-by: Rupal Mahajan <[email protected]>

Add pit

b75d782

Signed-off-by: Rupal Mahajan <[email protected]>

Merge branch 'opensearch-project:main' into join-search-after

a336e3b

nit

afd24d8

Signed-off-by: Rupal Mahajan <[email protected]>

Fix tests

8f840ae

Signed-off-by: Rupal Mahajan <[email protected]>

ignore joinWithGeoIntersectNL

070b40f

Signed-off-by: Rupal Mahajan <[email protected]>

Rerun CI with scroll

58c96f5

Signed-off-by: Rupal Mahajan <[email protected]>

Merge branch 'opensearch-project:main' into join-search-after

3dad30d

draft

b5035ac

Signed-off-by: Rupal Mahajan <[email protected]>

Remove unused code and retrigger CI with search_after true

1404cd5

Signed-off-by: Rupal Mahajan <[email protected]>

Merge branch 'opensearch-project:main' into join-search-after

fd09367

Address comments

fcb584a

Signed-off-by: Rupal Mahajan <[email protected]>

Merge branch 'join-search-after' of github.com:rupal-bq/opensearch-sq…

12f5abb

…l into join-search-after Signed-off-by: Rupal Mahajan <[email protected]>

Remove unused code change

08a6a29

Signed-off-by: Rupal Mahajan <[email protected]>

Merge branch 'main' into join-search-after

030f4b5

Signed-off-by: Rupal Mahajan <[email protected]>

Update pit keep alive time with SQL_CURSOR_KEEP_ALIVE

39f727a

Signed-off-by: Rupal Mahajan <[email protected]>

Fix scroll condition

be8d986

Signed-off-by: Rupal Mahajan <[email protected]>

nit

7bb89d8

Signed-off-by: Rupal Mahajan <[email protected]>

Add pit before query execution

60bd8fe

Signed-off-by: Rupal Mahajan <[email protected]>

Merge branch 'opensearch-project:main' into join-search-after

17d1d0e

Merge branch 'join-search-after' into multi-query-search-after-pit

3c511b2

Signed-off-by: Rupal Mahajan <[email protected]>

Refactor get response with pit method

281153f

Signed-off-by: Rupal Mahajan <[email protected]>

Update remaining scroll search calls

5bcd134

Signed-off-by: Rupal Mahajan <[email protected]>

Fix integ test failures

87f611b

Signed-off-by: Rupal Mahajan <[email protected]>

nit

3af87c4

Signed-off-by: Rupal Mahajan <[email protected]>

Move pit from join request builder to executor

bee4863

Signed-off-by: Rupal Mahajan <[email protected]>

Remove unused methods

0857a6b

Signed-off-by: Rupal Mahajan <[email protected]>

Merge branch 'join-search-after' into multi-query-search-after-pit

bcd38b0

Signed-off-by: Rupal Mahajan <[email protected]>

Merge branch 'opensearch-project:main' into join-search-after

8e1c101

rupal-bq requested review from Yury-Fridlyand, anirudha, forestmvey, acarbonetto, GumpacG and ykmr1224 as code owners June 26, 2024 22:01

dai-chen reviewed Jul 5, 2024

View reviewed changes

rupal-bq added 4 commits July 8, 2024 10:20

move all common fields to ElasticHitsExecutor

783c24f

Signed-off-by: Rupal Mahajan <[email protected]>

Merge branch 'main' into multi-query-search-after-pit

c60a396

Signed-off-by: Rupal Mahajan <[email protected]>

Merge branch 'feature/pit' into multi-query-search-after-pit

ff3cae4

Signed-off-by: Rupal Mahajan <[email protected]>

Merge remote-tracking branch 'upstream/feature/pit' into multi-query-…

bed6a13

…search-after-pit Signed-off-by: Rupal Mahajan <[email protected]>

dai-chen reviewed Jul 8, 2024

View reviewed changes

add javadoc for ElasticHitsExecutor

4fbc0ec

Signed-off-by: Rupal Mahajan <[email protected]>

dai-chen approved these changes Jul 9, 2024

View reviewed changes

rupal-bq added 2 commits July 9, 2024 16:28

Add missing javadoc

037b146

Signed-off-by: Rupal Mahajan <[email protected]>

Forcing an empty commit as last commit is stuck processing updates

81e2af9

Signed-off-by: Rupal Mahajan <[email protected]>

rupal-bq force-pushed the multi-query-search-after-pit branch from 3023ea8 to 81e2af9 Compare July 9, 2024 23:48

vamsimanohar approved these changes Jul 10, 2024

View reviewed changes

rupal-bq merged commit 8c6dc0c into opensearch-project:feature/pit Jul 10, 2024
10 checks passed

manasvinibs mentioned this pull request Aug 19, 2024

Manual tests for Sample queries for Scroll to PIT pagination queries #2941

Open

Swiddis added the v2.18.0 Issues targeting release v2.18.0 label Oct 9, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add pit for multi query #2753

Add pit for multi query #2753

rupal-bq commented Jun 14, 2024

dai-chen Jul 5, 2024

rupal-bq Jul 8, 2024 •

edited

Loading

dai-chen Jul 8, 2024

rupal-bq Jul 8, 2024

dai-chen Jul 8, 2024

dai-chen left a comment

Add pit for multi query #2753

Add pit for multi query #2753

Conversation

rupal-bq commented Jun 14, 2024

Description

Issues Resolved

Check List

dai-chen Jul 5, 2024

Choose a reason for hiding this comment

rupal-bq Jul 8, 2024 • edited Loading

Choose a reason for hiding this comment

dai-chen Jul 8, 2024

Choose a reason for hiding this comment

rupal-bq Jul 8, 2024

Choose a reason for hiding this comment

dai-chen Jul 8, 2024

Choose a reason for hiding this comment

dai-chen left a comment

Choose a reason for hiding this comment

rupal-bq Jul 8, 2024 •

edited

Loading