Pagination, phase 1: Add unit tests for `:core` module with coverage. #230

Yury-Fridlyand · 2023-02-22T21:36:14Z

Add unit tests for :core module with coverage. Uncovered: toCursor, because it is will be changed soon. Merge target: #226

Description

Completely pass :core:checkstyleTest. Partial pass for :core:test :core:jacocoTestReport :core:jacocoTestCoverageVerification:

* What went wrong:
Execution failed for task ':core:jacocoTestCoverageVerification'.
> Rule violated for class org.opensearch.sql.planner.physical.PhysicalPlan: lines covered ratio is 0.7, but expected minimum is 1.0
  Rule violated for class org.opensearch.sql.planner.physical.PhysicalPlan: branches covered ratio is 0.5, but expected minimum is 1.0
  Rule violated for class org.opensearch.sql.planner.physical.ProjectOperator: lines covered ratio is 0.9, but expected minimum is 1.0
  Rule violated for class org.opensearch.sql.planner.physical.ProjectOperator: branches covered ratio is 0.8, but expected minimum is 1.0
  Rule violated for class org.opensearch.sql.planner.PaginateOperator: branches covered ratio is 0.9, but expected minimum is 1.0

Issues Resolved

N/A

Check List

New functionality includes testing.
- All tests pass, including unit test, integration test and doctest
New functionality has been documented.
- New functionality has javadoc added
- New functionality has user manual doc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

codecov · 2023-02-22T21:40:00Z

Codecov Report

Merging #230 (16bdd77) into dev-pagination-v2_revC (2c1b2f9) will increase coverage by 5.16%.
The diff coverage is n/a.

@@                     Coverage Diff                      @@
##             dev-pagination-v2_revC     #230      +/-   ##
============================================================
+ Coverage                     94.66%   99.82%   +5.16%     
+ Complexity                     3270      185    -3085     
============================================================
  Files                           337       29     -308     
  Lines                          8339      582    -7757     
  Branches                        529       50     -479     
============================================================
- Hits                           7894      581    -7313     
+ Misses                          436        1     -435     
+ Partials                          9        0       -9

Flag	Coverage Δ
sql-engine	`99.82% <ø> (+5.16%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...ain/java/org/opensearch/sql/analysis/Analyzer.java
.../org/opensearch/sql/data/model/ExprValueUtils.java
...rg/opensearch/sql/executor/CanPaginateVisitor.java
...rg/opensearch/sql/executor/PaginatedPlanCache.java
...main/java/org/opensearch/sql/executor/QueryId.java
...java/org/opensearch/sql/executor/QueryService.java
.../sql/executor/execution/ContinuePaginatedPlan.java
...ensearch/sql/executor/execution/PaginatedPlan.java
.../sql/executor/execution/PaginatedQueryService.java
...earch/sql/executor/execution/QueryPlanFactory.java
... and 297 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Yury-Fridlyand · 2023-02-23T00:58:37Z

CI run for this PR: https://github.com/Bit-Quill/opensearch-project-sql/actions/runs/4247173319
No unit tests or checkstyle validations failed for :core:test

MaxKsyunz · 2023-02-23T21:35:17Z

core/src/main/java/org/opensearch/sql/executor/CanPaginateVisitor.java

@@ -44,13 +44,13 @@ public Boolean visitRelation(Relation node, Object context) {
    return Boolean.TRUE;
  }

+  /*


Is canPaginate not used anywhere?

It is used in commented code below only.

MaxKsyunz · 2023-02-23T21:37:02Z

core/src/main/java/org/opensearch/sql/executor/execution/PaginatedPlan.java

  public void explain(ResponseListener<ExecutionEngine.ExplainResponse> listener) {
-    listener.onFailure(new UnsupportedOperationException(
+    listener.onFailure(new NotImplementedException(


Why change from UnsupportedOperationException to NotImplementedException here but from NotImeplementedException to UnsuportedException ContinuePaginatedPlane.java?

Just to comply with nature of these exceptions. Explain for PaginatedPlan will be implemented soon (so throw NotImplementedException), but explain in ContinuePaginatedPlan won't be implemented ever (as I understand) - so UnsupportedOperationException is thrown.

core/src/test/java/org/opensearch/sql/executor/PaginatedPlanCacheTest.java

core/src/test/java/org/opensearch/sql/executor/CanPaginateVisitorTest.java

core/src/main/java/org/opensearch/sql/storage/read/TableScanBuilder.java

GabeFernandez310 · 2023-03-02T16:18:33Z

core/src/main/java/org/opensearch/sql/executor/execution/PaginatedQueryService.java

-    } catch (Exception e) {
-      listener.onFailure(e);
-    }
+    executionEngine.execute(plan, ExecutionContext.emptyExecutionContext(), listener);


Why was the try-catch block removed? Was something changed so that we don't expect this to throw an exception anymore?

executePlan called from execute function only which already has try-catch block.

…`, because it is will be changed soon. Signed-off-by: Yury-Fridlyand <[email protected]>

Signed-off-by: Yury-Fridlyand <[email protected]>

MaxKsyunz · 2023-03-08T01:08:23Z

@Yury-Fridlyand any reason this can't be merged?

Yury-Fridlyand · 2023-03-08T01:10:20Z

No reasons, just waited for more reviews. Going to merge, it is not the last step and last opportunity to review.

* Fixing integration tests broken during POC Signed-off-by: MaxKsyunz <[email protected]> * Comment to clarify an exception. Signed-off-by: MaxKsyunz <[email protected]> * Add support for paginated scroll request, first page. Implement PaginatedPlanCache.convertToPlan for second page to work. Signed-off-by: MaxKsyunz <[email protected]> * Progress on paginated scroll request, subsequent page. Signed-off-by: MaxKsyunz <[email protected]> * Move `ExpressionSerializer` from `opensearch` to `core`. Signed-off-by: Yury-Fridlyand <[email protected]> * Rename `Cursor` `asString` to `toString`. Signed-off-by: Yury-Fridlyand <[email protected]> * Disable scroll cleaning. Signed-off-by: Yury-Fridlyand <[email protected]> * Add full cursor serialization and deserialization. Signed-off-by: Yury-Fridlyand <[email protected]> * Misc fixes. Signed-off-by: Yury-Fridlyand <[email protected]> * Further work on pagination. * Added push down page size from `LogicalPaginate` to `LogicalRelation`. * Improved cursor encoding and decoding. * Added cursor compression. * Fixed issuing `SearchScrollRequest`. * Fixed returning last empty page. * Minor code grooming/commenting. Signed-off-by: Yury-Fridlyand <[email protected]> * Pagination fix for empty indices. Signed-off-by: Yury-Fridlyand <[email protected]> * Fix error reporting on wrong cursor. Signed-off-by: Yury-Fridlyand <[email protected]> * Minor comments and error reporting improvement. Signed-off-by: Yury-Fridlyand <[email protected]> * Add an end-to-end integration test. Signed-off-by: Yury-Fridlyand <[email protected]> * Add `explain` request handlers. Signed-off-by: Yury-Fridlyand <[email protected]> * Add IT for explain. Signed-off-by: Yury-Fridlyand <[email protected]> * Address issues flagged by checkstyle build step (#229) Signed-off-by: MaxKsyunz <[email protected]> * Pagination, phase 1: Add unit tests for `:core` module with coverage. (#230) * Add unit tests for `:core` module with coverage. Uncovered: `toCursor`, because it is will be changed soon. Signed-off-by: Yury-Fridlyand <[email protected]> * Pagination, phase 1: Add unit tests for SQL module with coverage. (#239) * Add unit tests for SQL module with coverage. Signed-off-by: Yury-Fridlyand <[email protected]> * Update sql/src/main/java/org/opensearch/sql/sql/domain/SQLQueryRequest.java Signed-off-by: Yury-Fridlyand <[email protected]> Co-authored-by: GabeFernandez310 <[email protected]> --------- Signed-off-by: Yury-Fridlyand <[email protected]> Co-authored-by: GabeFernandez310 <[email protected]> * Pagination, phase 1: Add unit tests for `:opensearch` module with coverage. (#233) * Add UT for `:opensearch` module with full coverage, except `toCursor`. Signed-off-by: Yury-Fridlyand <[email protected]> * Fix checkstyle. Signed-off-by: Yury-Fridlyand <[email protected]> --------- Signed-off-by: Yury-Fridlyand <[email protected]> * Fix the merges. Signed-off-by: Yury-Fridlyand <[email protected]> * Fix explain. Signed-off-by: Yury-Fridlyand <[email protected]> * Fix scroll cleaning. Signed-off-by: Yury-Fridlyand <[email protected]> * Store `TotalHits` and use it to report `total` in response. Signed-off-by: Yury-Fridlyand <[email protected]> * Add missing UT for `:protocol` module. Signed-off-by: Yury-Fridlyand <[email protected]> * Fix PPL UTs damaged in f4ea4ad. Signed-off-by: Yury-Fridlyand <[email protected]> * Minor checkstyle fixes. Signed-off-by: Yury-Fridlyand <[email protected]> * Fallback to v1 engine for pagination (#245) * Pagination fallback integration tests. Signed-off-by: MaxKsyunz <[email protected]> * Add UT with coverage for `toCursor` serialization. Signed-off-by: Yury-Fridlyand <[email protected]> * Fix broken tests in `legacy`. Signed-off-by: Yury-Fridlyand <[email protected]> * Fix getting `total` from non-paged requests and from queries without `FROM` clause. Signed-off-by: Yury-Fridlyand <[email protected]> * Fix scroll cleaning. Signed-off-by: Yury-Fridlyand <[email protected]> * Fix cursor request processing. Signed-off-by: Yury-Fridlyand <[email protected]> * Update ITs. Signed-off-by: Yury-Fridlyand <[email protected]> * Fix (again) TotalHits feature. Signed-off-by: Yury-Fridlyand <[email protected]> * Fix typo in prometheus config. Signed-off-by: Yury-Fridlyand <[email protected]> * Recover commented logging. Signed-off-by: Yury-Fridlyand <[email protected]> * Move `test_pagination_blackbox` to a separate class and add logging. Signed-off-by: Yury-Fridlyand <[email protected]> * Address some PR feedbacks: rename some classes and revert unnecessary whitespace changed. Signed-off-by: Yury-Fridlyand <[email protected]> * Minor commenting. Signed-off-by: Yury-Fridlyand <[email protected]> * Address PR comments. * Add javadocs * Renames * Cleaning up some comments * Remove unused code * Speed up IT Signed-off-by: Yury-Fridlyand <[email protected]> * Minor missing changes. Signed-off-by: Yury-Fridlyand <[email protected]> * Integration tests for fetch_size, max_result_window, and query.size_limit (#248) Signed-off-by: MaxKsyunz <[email protected]> * Remove `PaginatedQueryService`, extend `QueryService` to hold two planners and use them. Signed-off-by: Yury-Fridlyand <[email protected]> * Move push down functions from request builders to a new interface. Signed-off-by: Yury-Fridlyand <[email protected]> * Some file moves. Signed-off-by: Yury-Fridlyand <[email protected]> * Minor clean-up according to PR review. Signed-off-by: Yury-Fridlyand <[email protected]> --------- Signed-off-by: MaxKsyunz <[email protected]> Signed-off-by: Yury-Fridlyand <[email protected]> Co-authored-by: MaxKsyunz <[email protected]> Co-authored-by: GabeFernandez310 <[email protected]> Co-authored-by: Max Ksyunz <[email protected]>

* Support pagination in V2 engine, phase 1 (#226) * Fixing integration tests broken during POC Signed-off-by: MaxKsyunz <[email protected]> * Comment to clarify an exception. Signed-off-by: MaxKsyunz <[email protected]> * Add support for paginated scroll request, first page. Implement PaginatedPlanCache.convertToPlan for second page to work. Signed-off-by: MaxKsyunz <[email protected]> * Progress on paginated scroll request, subsequent page. Signed-off-by: MaxKsyunz <[email protected]> * Move `ExpressionSerializer` from `opensearch` to `core`. Signed-off-by: Yury-Fridlyand <[email protected]> * Rename `Cursor` `asString` to `toString`. Signed-off-by: Yury-Fridlyand <[email protected]> * Disable scroll cleaning. Signed-off-by: Yury-Fridlyand <[email protected]> * Add full cursor serialization and deserialization. Signed-off-by: Yury-Fridlyand <[email protected]> * Misc fixes. Signed-off-by: Yury-Fridlyand <[email protected]> * Further work on pagination. * Added push down page size from `LogicalPaginate` to `LogicalRelation`. * Improved cursor encoding and decoding. * Added cursor compression. * Fixed issuing `SearchScrollRequest`. * Fixed returning last empty page. * Minor code grooming/commenting. Signed-off-by: Yury-Fridlyand <[email protected]> * Pagination fix for empty indices. Signed-off-by: Yury-Fridlyand <[email protected]> * Fix error reporting on wrong cursor. Signed-off-by: Yury-Fridlyand <[email protected]> * Minor comments and error reporting improvement. Signed-off-by: Yury-Fridlyand <[email protected]> * Add an end-to-end integration test. Signed-off-by: Yury-Fridlyand <[email protected]> * Add `explain` request handlers. Signed-off-by: Yury-Fridlyand <[email protected]> * Add IT for explain. Signed-off-by: Yury-Fridlyand <[email protected]> * Address issues flagged by checkstyle build step (#229) Signed-off-by: MaxKsyunz <[email protected]> * Pagination, phase 1: Add unit tests for `:core` module with coverage. (#230) * Add unit tests for `:core` module with coverage. Uncovered: `toCursor`, because it is will be changed soon. Signed-off-by: Yury-Fridlyand <[email protected]> * Pagination, phase 1: Add unit tests for SQL module with coverage. (#239) * Add unit tests for SQL module with coverage. Signed-off-by: Yury-Fridlyand <[email protected]> * Update sql/src/main/java/org/opensearch/sql/sql/domain/SQLQueryRequest.java Signed-off-by: Yury-Fridlyand <[email protected]> Co-authored-by: GabeFernandez310 <[email protected]> --------- Signed-off-by: Yury-Fridlyand <[email protected]> Co-authored-by: GabeFernandez310 <[email protected]> * Pagination, phase 1: Add unit tests for `:opensearch` module with coverage. (#233) * Add UT for `:opensearch` module with full coverage, except `toCursor`. Signed-off-by: Yury-Fridlyand <[email protected]> * Fix checkstyle. Signed-off-by: Yury-Fridlyand <[email protected]> --------- Signed-off-by: Yury-Fridlyand <[email protected]> * Fix the merges. Signed-off-by: Yury-Fridlyand <[email protected]> * Fix explain. Signed-off-by: Yury-Fridlyand <[email protected]> * Fix scroll cleaning. Signed-off-by: Yury-Fridlyand <[email protected]> * Store `TotalHits` and use it to report `total` in response. Signed-off-by: Yury-Fridlyand <[email protected]> * Add missing UT for `:protocol` module. Signed-off-by: Yury-Fridlyand <[email protected]> * Fix PPL UTs damaged in f4ea4ad. Signed-off-by: Yury-Fridlyand <[email protected]> * Minor checkstyle fixes. Signed-off-by: Yury-Fridlyand <[email protected]> * Fallback to v1 engine for pagination (#245) * Pagination fallback integration tests. Signed-off-by: MaxKsyunz <[email protected]> * Add UT with coverage for `toCursor` serialization. Signed-off-by: Yury-Fridlyand <[email protected]> * Fix broken tests in `legacy`. Signed-off-by: Yury-Fridlyand <[email protected]> * Fix getting `total` from non-paged requests and from queries without `FROM` clause. Signed-off-by: Yury-Fridlyand <[email protected]> * Fix scroll cleaning. Signed-off-by: Yury-Fridlyand <[email protected]> * Fix cursor request processing. Signed-off-by: Yury-Fridlyand <[email protected]> * Update ITs. Signed-off-by: Yury-Fridlyand <[email protected]> * Fix (again) TotalHits feature. Signed-off-by: Yury-Fridlyand <[email protected]> * Fix typo in prometheus config. Signed-off-by: Yury-Fridlyand <[email protected]> * Recover commented logging. Signed-off-by: Yury-Fridlyand <[email protected]> * Move `test_pagination_blackbox` to a separate class and add logging. Signed-off-by: Yury-Fridlyand <[email protected]> * Address some PR feedbacks: rename some classes and revert unnecessary whitespace changed. Signed-off-by: Yury-Fridlyand <[email protected]> * Minor commenting. Signed-off-by: Yury-Fridlyand <[email protected]> * Address PR comments. * Add javadocs * Renames * Cleaning up some comments * Remove unused code * Speed up IT Signed-off-by: Yury-Fridlyand <[email protected]> * Minor missing changes. Signed-off-by: Yury-Fridlyand <[email protected]> * Integration tests for fetch_size, max_result_window, and query.size_limit (#248) Signed-off-by: MaxKsyunz <[email protected]> * Remove `PaginatedQueryService`, extend `QueryService` to hold two planners and use them. Signed-off-by: Yury-Fridlyand <[email protected]> * Move push down functions from request builders to a new interface. Signed-off-by: Yury-Fridlyand <[email protected]> * Some file moves. Signed-off-by: Yury-Fridlyand <[email protected]> * Minor clean-up according to PR review. Signed-off-by: Yury-Fridlyand <[email protected]> --------- Signed-off-by: MaxKsyunz <[email protected]> Signed-off-by: Yury-Fridlyand <[email protected]> Co-authored-by: MaxKsyunz <[email protected]> Co-authored-by: GabeFernandez310 <[email protected]> Co-authored-by: Max Ksyunz <[email protected]> * Make scroll timeout configurable. Signed-off-by: Yury-Fridlyand <[email protected]> * Fix IT to set cursor keep alive parameter. Signed-off-by: Yury-Fridlyand <[email protected]> * Remove `QueryId.None`. Signed-off-by: Yury-Fridlyand <[email protected]> * Rename according to PR feedback. Signed-off-by: Yury-Fridlyand <[email protected]> * Remove default implementations of `PushDownRequestBuilder`. Signed-off-by: Yury-Fridlyand <[email protected]> * Merge paginated plan optimizer into the regular optimizer. (opensearch-project#1516) Merge paginated plan optimizer into the regular optimizer. --------- Signed-off-by: MaxKsyunz <[email protected]> Co-authored-by: Yury-Fridlyand <[email protected]> * Complete rework on serialization and deserialization. (opensearch-project#1498) Signed-off-by: Yury-Fridlyand <[email protected]> * Resolve merge conflicts and fix tests. Signed-off-by: Yury-Fridlyand <[email protected]> * Minor cleanup. Signed-off-by: Yury-Fridlyand <[email protected]> * Minor cleanup - missing changes for the previous commit. Signed-off-by: Yury-Fridlyand <[email protected]> * Remove paginate operator (opensearch-project#1528) * Remove PaginateOperator class since it is no longer used. --------- Signed-off-by: MaxKsyunz <[email protected]> * Remove `PaginatedPlan` - move logic to `QueryPlan`. Signed-off-by: Yury-Fridlyand <[email protected]> * Remove default implementations from `SerializablePlan`. Signed-off-by: Yury-Fridlyand <[email protected]> * Add a doc. Signed-off-by: Yury-Fridlyand <[email protected]> * Update design graphs. Signed-off-by: Yury-Fridlyand <[email protected]> * More fixes for merge from upstream/main. Signed-off-by: MaxKsyunz <[email protected]> --------- Signed-off-by: MaxKsyunz <[email protected]> Signed-off-by: Yury-Fridlyand <[email protected]> Co-authored-by: MaxKsyunz <[email protected]> Co-authored-by: GabeFernandez310 <[email protected]> Co-authored-by: Max Ksyunz <[email protected]>

Yury-Fridlyand requested review from GumpacG, forestmvey, GabeFernandez310, MaxKsyunz and acarbonetto February 22, 2023 21:36

MaxKsyunz approved these changes Feb 23, 2023

View reviewed changes

Yury-Fridlyand commented Feb 23, 2023

View reviewed changes

core/src/test/java/org/opensearch/sql/executor/PaginatedPlanCacheTest.java Outdated Show resolved Hide resolved

Yury-Fridlyand commented Feb 23, 2023

View reviewed changes

core/src/test/java/org/opensearch/sql/executor/CanPaginateVisitorTest.java Outdated Show resolved Hide resolved

Yury-Fridlyand commented Feb 23, 2023

View reviewed changes

core/src/test/java/org/opensearch/sql/executor/CanPaginateVisitorTest.java Outdated Show resolved Hide resolved

Yury-Fridlyand commented Feb 23, 2023

View reviewed changes

core/src/main/java/org/opensearch/sql/storage/read/TableScanBuilder.java Outdated Show resolved Hide resolved

GabeFernandez310 reviewed Mar 2, 2023

View reviewed changes

GumpacG approved these changes Mar 2, 2023

View reviewed changes

Yury-Fridlyand force-pushed the dev-pagination-v2_revC branch from 11a18c1 to a3ef2bf Compare March 3, 2023 22:35

Yury-Fridlyand added 2 commits March 3, 2023 14:38

Add unit tests for :core module with coverage. Uncovered: `toCursor…

7b99390

…`, because it is will be changed soon. Signed-off-by: Yury-Fridlyand <[email protected]>

Minor fixes

16bdd77

Signed-off-by: Yury-Fridlyand <[email protected]>

Yury-Fridlyand force-pushed the dev-pagination-v2_revC-ut-coverate-for-core branch from a35f6ce to 16bdd77 Compare March 3, 2023 22:40

Yury-Fridlyand changed the title ~~Add unit tests for :core module with coverage.~~ Pagination, phase 1: Add unit tests for :core module with coverage. Mar 7, 2023

Yury-Fridlyand merged commit 2d29549 into dev-pagination-v2_revC Mar 8, 2023

Yury-Fridlyand deleted the dev-pagination-v2_revC-ut-coverate-for-core branch March 8, 2023 01:10

Yury-Fridlyand mentioned this pull request Mar 10, 2023

Support pagination in V2 engine, phase 1 #226

Merged

18 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pagination, phase 1: Add unit tests for `:core` module with coverage. #230

Pagination, phase 1: Add unit tests for `:core` module with coverage. #230

Yury-Fridlyand commented Feb 22, 2023

codecov bot commented Feb 22, 2023 •

edited

Loading

Yury-Fridlyand commented Feb 23, 2023

MaxKsyunz Feb 23, 2023

Yury-Fridlyand Feb 23, 2023

MaxKsyunz Feb 23, 2023

Yury-Fridlyand Feb 23, 2023

GabeFernandez310 Mar 2, 2023

Yury-Fridlyand Mar 2, 2023

MaxKsyunz commented Mar 8, 2023

Yury-Fridlyand commented Mar 8, 2023

Pagination, phase 1: Add unit tests for :core module with coverage. #230

Pagination, phase 1: Add unit tests for :core module with coverage. #230

Conversation

Yury-Fridlyand commented Feb 22, 2023

Description

Issues Resolved

Check List

codecov bot commented Feb 22, 2023 • edited Loading

Codecov Report

Yury-Fridlyand commented Feb 23, 2023

MaxKsyunz Feb 23, 2023

Choose a reason for hiding this comment

Yury-Fridlyand Feb 23, 2023

Choose a reason for hiding this comment

MaxKsyunz Feb 23, 2023

Choose a reason for hiding this comment

Yury-Fridlyand Feb 23, 2023

Choose a reason for hiding this comment

GabeFernandez310 Mar 2, 2023

Choose a reason for hiding this comment

Yury-Fridlyand Mar 2, 2023

Choose a reason for hiding this comment

MaxKsyunz commented Mar 8, 2023

Yury-Fridlyand commented Mar 8, 2023

Pagination, phase 1: Add unit tests for `:core` module with coverage. #230

Pagination, phase 1: Add unit tests for `:core` module with coverage. #230

codecov bot commented Feb 22, 2023 •

edited

Loading