Support pagination in V2 engine, phase 1 (#226) #1483

Yury-Fridlyand · 2023-03-28T23:38:54Z

Description

https://github.com/Bit-Quill/opensearch-project-sql/blob/61767f2b200b2a7f8c8b2df32de209b3c30caa61/docs/dev/Pagination-v2.md

PHASE 1
Pagination supports queries like

select * from <index>;

For example:

curl -s -XPOST http://localhost:9200/_plugins/_sql -H 'Content-Type: application/json' -d '{"query": "SELECT * from calcs", "fetch_size": 5}'

Cursor request:

curl -s -XPOST http://localhost:9200/_plugins/_sql -H 'Content-Type: application/json' -d '{"cursor": "...." }'

All credits to @MaxKsyunz.

You can use attached script for testing as well. Command line: ./cursor_test.sh <table> <page size>. Requires jq. Rename it before use.
cursor_test.sh.txt

Scroll API usage doc:
https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/java-search-scrolling.html

Please, see team review and discussion in Bit-Quill#226

Pagination demo: https://user-images.githubusercontent.com/88679692/224208630-8d38d833-abf8-4035-8d15-d5fb4382deca.mp4

Further improvements are coming

Support for nested expression serialization in #249
Rework on serialization and deserialization in #252
Support queries with LIMIT, WHERE, HAVING and ORDER BY and queries without FROM into pagination. in #253 - Phase 2
Rework ComparisonIT to use pagination. in #254 - Phase 2

Issues Resolved

#656

Check List

New functionality includes testing.
- All tests pass, including unit test, integration test and doctest
New functionality has been documented.
- New functionality has javadoc added
- New functionality has user manual doc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Yury-Fridlyand

I copied unresolved questions/issues from #226

Yury-Fridlyand · 2023-03-28T23:42:19Z

core/src/main/java/org/opensearch/sql/planner/optimizer/rule/PushPageSize.java

+import org.opensearch.sql.planner.logical.LogicalRelation;
+import org.opensearch.sql.planner.optimizer.Rule;
+
+public class PushPageSize


Bit-Quill#226 (comment) by @MaxKsyunz:

Could you implement this similar to PUSH_DOWN_* rules:

Add a pushDownPageSize method to TableScanBuilder

Add a PUSH_DOWN_PAGE_SIZE similar to others that matches LogicalPaginate of scanBuilder and calls pushDownPageSize.

Doing so would remove the need for PushPageSize and LogicalRelation.pageSize property, be less code, and follow a pattern already used in the code base, like PUSH_DOWN_LIMIT.

I tried to do this and faced #1470. I keep the draft, and I'll try to do thorough investigation and fix.

Yury-Fridlyand · 2023-03-28T23:46:31Z

core/src/main/java/org/opensearch/sql/storage/StorageEngine.java

@@ -30,4 +32,8 @@ default Collection<FunctionResolver> getFunctions() {
    return Collections.emptyList();
  }

+  default TableScanOperator getTableScan(String indexName, String scrollId) {


Bit-Quill#226 (comment) by @penghuo:

why get TableScanOperator in StorageEngine?

@Yury-Fridlyand:

It is used in cursor deserialization/parsing, this happens in :core module. All implementation of StorageEngine are defined outside of :core and unavailable and that point.

@MaxKsyunz:

@penghuo does this design make sense? Would love to know what you think.

opensearch/src/main/java/org/opensearch/sql/opensearch/request/OpenSearchScrollRequest.java

* Fixing integration tests broken during POC Signed-off-by: MaxKsyunz <[email protected]> * Comment to clarify an exception. Signed-off-by: MaxKsyunz <[email protected]> * Add support for paginated scroll request, first page. Implement PaginatedPlanCache.convertToPlan for second page to work. Signed-off-by: MaxKsyunz <[email protected]> * Progress on paginated scroll request, subsequent page. Signed-off-by: MaxKsyunz <[email protected]> * Move `ExpressionSerializer` from `opensearch` to `core`. Signed-off-by: Yury-Fridlyand <[email protected]> * Rename `Cursor` `asString` to `toString`. Signed-off-by: Yury-Fridlyand <[email protected]> * Disable scroll cleaning. Signed-off-by: Yury-Fridlyand <[email protected]> * Add full cursor serialization and deserialization. Signed-off-by: Yury-Fridlyand <[email protected]> * Misc fixes. Signed-off-by: Yury-Fridlyand <[email protected]> * Further work on pagination. * Added push down page size from `LogicalPaginate` to `LogicalRelation`. * Improved cursor encoding and decoding. * Added cursor compression. * Fixed issuing `SearchScrollRequest`. * Fixed returning last empty page. * Minor code grooming/commenting. Signed-off-by: Yury-Fridlyand <[email protected]> * Pagination fix for empty indices. Signed-off-by: Yury-Fridlyand <[email protected]> * Fix error reporting on wrong cursor. Signed-off-by: Yury-Fridlyand <[email protected]> * Minor comments and error reporting improvement. Signed-off-by: Yury-Fridlyand <[email protected]> * Add an end-to-end integration test. Signed-off-by: Yury-Fridlyand <[email protected]> * Add `explain` request handlers. Signed-off-by: Yury-Fridlyand <[email protected]> * Add IT for explain. Signed-off-by: Yury-Fridlyand <[email protected]> * Address issues flagged by checkstyle build step (#229) Signed-off-by: MaxKsyunz <[email protected]> * Pagination, phase 1: Add unit tests for `:core` module with coverage. (#230) * Add unit tests for `:core` module with coverage. Uncovered: `toCursor`, because it is will be changed soon. Signed-off-by: Yury-Fridlyand <[email protected]> * Pagination, phase 1: Add unit tests for SQL module with coverage. (#239) * Add unit tests for SQL module with coverage. Signed-off-by: Yury-Fridlyand <[email protected]> * Update sql/src/main/java/org/opensearch/sql/sql/domain/SQLQueryRequest.java Signed-off-by: Yury-Fridlyand <[email protected]> Co-authored-by: GabeFernandez310 <[email protected]> --------- Signed-off-by: Yury-Fridlyand <[email protected]> Co-authored-by: GabeFernandez310 <[email protected]> * Pagination, phase 1: Add unit tests for `:opensearch` module with coverage. (#233) * Add UT for `:opensearch` module with full coverage, except `toCursor`. Signed-off-by: Yury-Fridlyand <[email protected]> * Fix checkstyle. Signed-off-by: Yury-Fridlyand <[email protected]> --------- Signed-off-by: Yury-Fridlyand <[email protected]> * Fix the merges. Signed-off-by: Yury-Fridlyand <[email protected]> * Fix explain. Signed-off-by: Yury-Fridlyand <[email protected]> * Fix scroll cleaning. Signed-off-by: Yury-Fridlyand <[email protected]> * Store `TotalHits` and use it to report `total` in response. Signed-off-by: Yury-Fridlyand <[email protected]> * Add missing UT for `:protocol` module. Signed-off-by: Yury-Fridlyand <[email protected]> * Fix PPL UTs damaged in f4ea4ad. Signed-off-by: Yury-Fridlyand <[email protected]> * Minor checkstyle fixes. Signed-off-by: Yury-Fridlyand <[email protected]> * Fallback to v1 engine for pagination (#245) * Pagination fallback integration tests. Signed-off-by: MaxKsyunz <[email protected]> * Add UT with coverage for `toCursor` serialization. Signed-off-by: Yury-Fridlyand <[email protected]> * Fix broken tests in `legacy`. Signed-off-by: Yury-Fridlyand <[email protected]> * Fix getting `total` from non-paged requests and from queries without `FROM` clause. Signed-off-by: Yury-Fridlyand <[email protected]> * Fix scroll cleaning. Signed-off-by: Yury-Fridlyand <[email protected]> * Fix cursor request processing. Signed-off-by: Yury-Fridlyand <[email protected]> * Update ITs. Signed-off-by: Yury-Fridlyand <[email protected]> * Fix (again) TotalHits feature. Signed-off-by: Yury-Fridlyand <[email protected]> * Fix typo in prometheus config. Signed-off-by: Yury-Fridlyand <[email protected]> * Recover commented logging. Signed-off-by: Yury-Fridlyand <[email protected]> * Move `test_pagination_blackbox` to a separate class and add logging. Signed-off-by: Yury-Fridlyand <[email protected]> * Address some PR feedbacks: rename some classes and revert unnecessary whitespace changed. Signed-off-by: Yury-Fridlyand <[email protected]> * Minor commenting. Signed-off-by: Yury-Fridlyand <[email protected]> * Address PR comments. * Add javadocs * Renames * Cleaning up some comments * Remove unused code * Speed up IT Signed-off-by: Yury-Fridlyand <[email protected]> * Minor missing changes. Signed-off-by: Yury-Fridlyand <[email protected]> * Integration tests for fetch_size, max_result_window, and query.size_limit (#248) Signed-off-by: MaxKsyunz <[email protected]> * Remove `PaginatedQueryService`, extend `QueryService` to hold two planners and use them. Signed-off-by: Yury-Fridlyand <[email protected]> * Move push down functions from request builders to a new interface. Signed-off-by: Yury-Fridlyand <[email protected]> * Some file moves. Signed-off-by: Yury-Fridlyand <[email protected]> * Minor clean-up according to PR review. Signed-off-by: Yury-Fridlyand <[email protected]> --------- Signed-off-by: MaxKsyunz <[email protected]> Signed-off-by: Yury-Fridlyand <[email protected]> Co-authored-by: MaxKsyunz <[email protected]> Co-authored-by: GabeFernandez310 <[email protected]> Co-authored-by: Max Ksyunz <[email protected]>

Signed-off-by: Yury-Fridlyand <[email protected]>

GumpacG · 2023-04-04T21:17:36Z

Setting fetch_size to be greater than the total size of the query results gives a cursor when legacy does not. Then when making a request with that cursor, it returns an empty result. Should it omit the cursor when on the last page?

For example, the query:
{ "fetch_size": 30, "query": "SELECT * FROM calcs;" }

The result would be:

Result

{
    "schema": [
        {
            "name": "bool3",
            "type": "boolean"
        },
        {
            "name": "int0",
            "type": "integer"
        },
        {
            "name": "time1",
            "type": "timestamp"
        },
        {
            "name": "bool2",
            "type": "boolean"
        },
        {
            "name": "int2",
            "type": "integer"
        },
        {
            "name": "int1",
            "type": "integer"
        },
        {
            "name": "str3",
            "type": "keyword"
        },
        {
            "name": "int3",
            "type": "integer"
        },
        {
            "name": "str1",
            "type": "keyword"
        },
        {
            "name": "str2",
            "type": "keyword"
        },
        {
            "name": "time0",
            "type": "timestamp"
        },
        {
            "name": "datetime0",
            "type": "timestamp"
        },
        {
            "name": "num1",
            "type": "double"
        },
        {
            "name": "num0",
            "type": "double"
        },
        {
            "name": "datetime1",
            "type": "timestamp"
        },
        {
            "name": "num4",
            "type": "double"
        },
        {
            "name": "bool1",
            "type": "boolean"
        },
        {
            "name": "key",
            "type": "keyword"
        },
        {
            "name": "num3",
            "type": "double"
        },
        {
            "name": "bool0",
            "type": "boolean"
        },
        {
            "name": "num2",
            "type": "double"
        },
        {
            "name": "str0",
            "type": "keyword"
        },
        {
            "name": "date3",
            "type": "timestamp"
        },
        {
            "name": "date2",
            "type": "timestamp"
        },
        {
            "name": "date1",
            "type": "timestamp"
        },
        {
            "name": "date0",
            "type": "timestamp"
        },
        {
            "name": "zzz",
            "type": "keyword"
        }
    ],
    "datarows": [
        [
            true,
            1,
            "1970-01-01 19:36:22",
            false,
            5,
            -3,
            "e",
            8,
            "CLAMP ON LAMPS",
            "one",
            "1899-12-30 21:07:32",
            "2004-07-09 10:17:35",
            8.42,
            12.3,
            null,
            null,
            true,
            "key00",
            -11.52,
            true,
            17.86,
            "FURNITURE",
            "1986-03-20 00:00:00",
            "1977-04-20 00:00:00",
            "2004-04-01 00:00:00",
            "2004-04-15 00:00:00",
            "a"
        ],
        [
            null,
            null,
            "1970-01-01 02:05:25",
            false,
            -4,
            -6,
            "e",
            13,
            "CLOCKS",
            "two",
            "1900-01-01 13:48:48",
            "2004-07-26 12:30:34",
            6.71,
            -12.3,
            null,
            10.85,
            true,
            "key01",
            -9.31,
            false,
            16.73,
            "FURNITURE",
            null,
            "1995-09-03 00:00:00",
            "2004-04-02 00:00:00",
            "1972-07-04 00:00:00",
            "b"
        ],
        [
            null,
            null,
            "1970-01-01 09:33:31",
            false,
            5,
            null,
            "e",
            2,
            "AIR PURIFIERS",
            "three",
            "1900-01-01 18:21:08",
            "2004-08-02 07:59:23",
            9.78,
            15.7,
            null,
            -13.47,
            true,
            "key02",
            -12.17,
            null,
            null,
            "OFFICE SUPPLIES",
            "1997-02-02 00:00:00",
            "1997-09-19 00:00:00",
            "2004-04-03 00:00:00",
            "1975-11-12 00:00:00",
            "c"
        ],
        [
            null,
            null,
            "1970-01-01 22:50:16",
            false,
            -5,
            -4,
            "e",
            5,
            "BINDER ACCESSORIES",
            null,
            "1900-01-01 18:51:48",
            "2004-07-05 13:14:20",
            7.43,
            -15.7,
            null,
            -6.05,
            false,
            "key03",
            -7.25,
            true,
            8.51,
            "OFFICE SUPPLIES",
            null,
            "1980-07-26 00:00:00",
            "2004-04-04 00:00:00",
            "2004-06-04 00:00:00",
            "d"
        ],
        [
            true,
            7,
            null,
            true,
            3,
            null,
            null,
            9,
            "BINDER CLIPS",
            "five",
            "1900-01-01 15:01:19",
            "2004-07-28 23:30:22",
            9.05,
            3.5,
            null,
            8.32,
            false,
            "key04",
            12.93,
            false,
            6.46,
            "OFFICE SUPPLIES",
            "1996-03-07 00:00:00",
            "1997-05-30 00:00:00",
            "2004-04-05 00:00:00",
            "2004-06-19 00:00:00",
            "e"
        ],
        [
            false,
            3,
            "1970-01-01 19:57:33",
            true,
            2,
            null,
            null,
            7,
            "BINDING MACHINES",
            "six",
            "1900-01-01 08:59:39",
            "2004-07-22 00:30:23",
            9.38,
            -3.5,
            null,
            10.71,
            false,
            "key05",
            -19.96,
            null,
            8.98,
            "OFFICE SUPPLIES",
            "1979-04-01 00:00:00",
            "1980-11-07 00:00:00",
            "2004-04-06 00:00:00",
            null,
            "f"
        ],
        [
            null,
            8,
            null,
            false,
            9,
            null,
            "e",
            18,
            "BINDING SUPPLIES",
            null,
            "1900-01-01 07:37:48",
            "2004-07-28 06:54:50",
            16.42,
            0.0,
            null,
            null,
            null,
            "key06",
            10.93,
            true,
            11.69,
            "OFFICE SUPPLIES",
            null,
            "1977-02-08 00:00:00",
            "2004-04-07 00:00:00",
            null,
            "g"
        ],
        [
            false,
            null,
            "1970-01-01 19:48:23",
            true,
            0,
            2,
            "e",
            3,
            "BUSINESS ENVELOPES",
            "eight",
            "1900-01-01 19:45:54",
            "2004-07-12 17:30:16",
            11.38,
            null,
            null,
            -10.24,
            null,
            "key07",
            3.64,
            false,
            17.25,
            "OFFICE SUPPLIES",
            null,
            "1974-05-03 00:00:00",
            "2004-04-08 00:00:00",
            null,
            "h"
        ],
        [
            false,
            null,
            "1970-01-01 22:20:14",
            false,
            -6,
            3,
            null,
            17,
            "ANSWERING MACHINES",
            "nine",
            "1900-01-01 09:00:59",
            "2004-07-04 22:49:28",
            9.47,
            10.0,
            null,
            4.77,
            null,
            "key08",
            -13.38,
            null,
            null,
            "TECHNOLOGY",
            "1983-05-22 00:00:00",
            "1976-09-09 00:00:00",
            "2004-04-09 00:00:00",
            null,
            "i"
        ],
        [
            null,
            8,
            null,
            false,
            -9,
            3,
            "e",
            2,
            "BUSINESS COPIERS",
            "ten",
            "1900-01-01 20:36:00",
            "2004-07-23 21:13:37",
            12.4,
            null,
            null,
            null,
            true,
            "key09",
            -10.56,
            null,
            11.5,
            "TECHNOLOGY",
            null,
            "1998-08-12 00:00:00",
            "2004-04-10 00:00:00",
            null,
            "j"
        ],
        [
            null,
            4,
            "1970-01-01 00:05:57",
            false,
            -3,
            null,
            "e",
            11,
            "CD-R MEDIA",
            "eleven",
            "1900-01-01 01:31:32",
            "2004-07-14 08:16:44",
            10.32,
            null,
            null,
            19.39,
            true,
            "key10",
            -4.79,
            true,
            6.8,
            "TECHNOLOGY",
            "1999-08-20 00:00:00",
            "1974-03-17 00:00:00",
            "2004-04-11 00:00:00",
            null,
            "k"
        ],
        [
            null,
            10,
            "1970-01-01 04:40:49",
            true,
            -4,
            -8,
            null,
            2,
            "CONFERENCE PHONES",
            "twelve",
            "1899-12-30 22:15:40",
            "2004-07-25 15:22:26",
            2.47,
            null,
            null,
            3.82,
            true,
            "key11",
            -10.81,
            false,
            3.79,
            "TECHNOLOGY",
            null,
            "1994-04-20 00:00:00",
            "2004-04-12 00:00:00",
            null,
            "l"
        ],
        [
            true,
            null,
            "1970-01-01 04:48:07",
            true,
            0,
            null,
            null,
            11,
            "CORDED KEYBOARDS",
            null,
            "1900-01-01 13:53:46",
            "2004-07-17 14:01:56",
            12.05,
            null,
            null,
            3.38,
            false,
            "key12",
            -6.62,
            null,
            null,
            "TECHNOLOGY",
            null,
            "2001-02-04 00:00:00",
            "2004-04-13 00:00:00",
            null,
            "m"
        ],
        [
            true,
            4,
            null,
            true,
            4,
            null,
            null,
            18,
            "CORDLESS KEYBOARDS",
            "fourteen",
            "1900-01-01 04:57:51",
            "2004-07-19 22:21:31",
            10.37,
            null,
            null,
            null,
            false,
            "key13",
            -18.43,
            null,
            13.04,
            "TECHNOLOGY",
            "1996-05-13 00:00:00",
            "1988-01-05 00:00:00",
            "2004-04-14 00:00:00",
            null,
            "n"
        ],
        [
            null,
            11,
            "1970-01-01 18:58:41",
            true,
            -8,
            null,
            "e",
            18,
            "DOT MATRIX PRINTERS",
            "fifteen",
            "1899-12-30 22:42:43",
            "2004-07-31 11:57:52",
            7.1,
            null,
            null,
            -14.21,
            false,
            "key14",
            6.84,
            true,
            null,
            "TECHNOLOGY",
            "1986-11-08 00:00:00",
            "1972-07-12 00:00:00",
            "2004-04-15 00:00:00",
            null,
            "o"
        ],
        [
            true,
            4,
            null,
            false,
            -9,
            null,
            "e",
            11,
            "DVD",
            "sixteen",
            "1899-12-30 22:24:08",
            "2004-07-14 07:43:00",
            16.81,
            null,
            null,
            6.75,
            null,
            "key15",
            -10.98,
            false,
            10.98,
            "TECHNOLOGY",
            null,
            "1995-06-04 00:00:00",
            "2004-04-16 00:00:00",
            null,
            "p"
        ],
        [
            null,
            8,
            "1970-01-01 12:33:57",
            false,
            6,
            -9,
            null,
            0,
            "ERICSSON",
            null,
            "1900-01-01 11:58:29",
            "2004-07-28 12:34:28",
            7.12,
            null,
            null,
            null,
            null,
            "key16",
            -2.6,
            null,
            7.87,
            "TECHNOLOGY",
            "1992-01-18 00:00:00",
            "2002-04-27 00:00:00",
            "2004-04-17 00:00:00",
            null,
            "q"
        ]
    ],
    "total": 17,
    "size": 17,
    "status": 200,
    "cursor": "n:1f8b08000000000000ffed9b5b73aa3a1886ff4e3b75667328abdb8b751194832c454148803b08545a02a5222afcfa1da07bb774cddaf73271c6190f21dfe9f1f3358977bbf0f05284a764c6cf446e76b73bbebd26f834bb2bc23c8977e1b14a946b794caaeae5ada866f7b3bb7218b17ea94eb3e39603b2673660c19f718eeb4894495498c4f78cd77021b5185deac04b2f38872d16c8397a91b63ee249a0c047ac1b048b6619094f8f69fa0bfdf52ef896b50280bb0019a95588d4360660ef68651ae7ca39d2d43a68e64eac1b6594e327070023d06015a0388d3598d1b1ef8e366fe89833d6608d0598e2dc7c5b8be6355acc49a2cb4de0996d88e6f59afb6affb19b4b8972f514780a783e00998ebd60710536a89b4f1acf5774f349e3f98841821c3601925e03771c1bd872c72b3a6f32f0119ba67234862bd01f0170e9734fa6fea75dacd6bfb1c69e5d529f37a16772dbee1add96a80f1c58c49b48348ab530bf505b6d80d4c6170e672caad55ab0e93cf45a9d5c82fd5cedfc83dd63e744fdb1684e8d73245cda2e2e2defedd4831d49c6859126de26b33ca3f13dd2db1542ebef7325ace4d36a01801ad17b4a7d7410777df7bd205d0bd794d6e1ec08c67b80a89f627a89bd1550f8d326cc55c1df4b958fa4624d4c0ee7a40edafac4513a7ed8fc937da1b9d0d30be86e43ce57cfc50afcfa73be33dfb3d375d1e5c1aded8ea9cca4cc4137f164023e6f7b00e4477c00db910f19ac63c47d8e9207fb341eddcae63b4781b2530c7e30a86f096a8dd4f1526e69c1c38f82f3b146aa75a636385725dcd803dceeb5c4a21394cbe6a1f50f1a77a07ec30a00a4d2629bd1b77aecfcbc24be683d517b3c9dfb13fabc87be7645bb092958db1fe21ed6cfa5ee9d74700020d107bcb0253f00903ed06bcb3eb67c145bddc796f7b1d5436c521f9b25d0dac01ef2efb0360048231f6c24f191f5394ce9edcb94f2d87088adda1cdc0f7e1c7e32a86f09ea6e4ee45e59a71e756a032ac4b4a1e95a2e6f31b06f11ec4182340c6c2641a603752f410c26419804991ad42a839a413d21a8752a035e4d06f508eac081aa613859dcd763017e32a86f0aeabe5333a859a79e12d47da766f28375eac941cd7e2832a8a704f5b0567d614b7a80ad554f09ec05eade875c48f3bfd169cde47200b4e85ebbd41f50b78632746f47236d6ce6d96efd64be80684f3bac8ca82bbe677d81773e80c3cd5fc21cbec6cbce166cbe74f57aa89bf45fdd8eb181a4872c4e38bce821ec4953876ea94b9425f2679668ee28ac5c073385b5cbf392e683f471fed68d5760998f7c50a3029e7efb1074b981b6e1b8d0818a6aba3a5bb3be39b8658d7e1d3b4c5f7f93228ac3c385a34026456e176a761684413d21a84732c4633284c99009c13d746c9b756cd6b12704f570c8899dde1baf88b0434eb70c35104e2461f263243f6462ba6e46b6b6c880be39a007e9c1b6cd99f49812d483f4609b314c7a4c08eabe53b36d73d6a9a704757f1684ad557f839a9d05b969a8fbb56a97fdc37c2c3fd859908980cdfeb7c8c09e24d86cad9a813d49b0d94a08037b5a6003b1fc91146c2306fccf46ccfdec6e5b26c53e098f38dd8587245e157172dde3b098e190e06aa66aa4f6852b4fc97bf685794dc97b8cd5391f23922d3d950f3c43b29094d939e46865643583fb2d32f5884b3d97a73fb35b5b7739691fb424dcf0caa7e7761024797cb5a16d87aafd1e40c347851162eded1a66f3e2fefefe1f2c488778714d0000"
}

And when that cursor is requested it results in:

Result

{
    "schema": [
        {
            "name": "bool3",
            "type": "boolean"
        },
        {
            "name": "int0",
            "type": "integer"
        },
        {
            "name": "time1",
            "type": "timestamp"
        },
        {
            "name": "bool2",
            "type": "boolean"
        },
        {
            "name": "int2",
            "type": "integer"
        },
        {
            "name": "int1",
            "type": "integer"
        },
        {
            "name": "str3",
            "type": "keyword"
        },
        {
            "name": "int3",
            "type": "integer"
        },
        {
            "name": "str1",
            "type": "keyword"
        },
        {
            "name": "str2",
            "type": "keyword"
        },
        {
            "name": "time0",
            "type": "timestamp"
        },
        {
            "name": "datetime0",
            "type": "timestamp"
        },
        {
            "name": "num1",
            "type": "double"
        },
        {
            "name": "num0",
            "type": "double"
        },
        {
            "name": "datetime1",
            "type": "timestamp"
        },
        {
            "name": "num4",
            "type": "double"
        },
        {
            "name": "bool1",
            "type": "boolean"
        },
        {
            "name": "key",
            "type": "keyword"
        },
        {
            "name": "num3",
            "type": "double"
        },
        {
            "name": "bool0",
            "type": "boolean"
        },
        {
            "name": "num2",
            "type": "double"
        },
        {
            "name": "str0",
            "type": "keyword"
        },
        {
            "name": "date3",
            "type": "timestamp"
        },
        {
            "name": "date2",
            "type": "timestamp"
        },
        {
            "name": "date1",
            "type": "timestamp"
        },
        {
            "name": "date0",
            "type": "timestamp"
        },
        {
            "name": "zzz",
            "type": "keyword"
        }
    ],
    "datarows": [],
    "total": 0,
    "size": 0,
    "status": 200
}

acarbonetto · 2023-04-05T16:38:06Z

Setting fetch_size to be greater than the total size of the query results gives a cursor when legacy does not. Then when making a request with that cursor, it returns an empty result. Should it omit the cursor when on the last page?

We need to return the cursor so long as it is still 'active' - ie, the scroll API scroll id is still stored in memory and available.

Alternatively, we could call the scroll API to close the scroll id once all the results are hit, but that's out of scope.

Yury-Fridlyand · 2023-04-05T20:58:27Z

Superseded by #1497.

Yury-Fridlyand commented Mar 28, 2023

View reviewed changes

Yury-Fridlyand force-pushed the integ-pagination-p1 branch from 7a0490b to 79d507a Compare March 28, 2023 23:56

This comment was marked as spam.

Sign in to view

Yury-Fridlyand force-pushed the integ-pagination-p1 branch from 79d507a to e951192 Compare March 29, 2023 01:00

Yury-Fridlyand marked this pull request as ready for review March 29, 2023 01:34

Yury-Fridlyand requested review from anirudha, penghuo, ps48, kavithacm, derek-ho, joshuali925, dai-chen, YANG-DB, rupal-bq, mengweieric, vamsi-amazon and MaxKsyunz as code owners March 29, 2023 01:34

Yury-Fridlyand added enhancement New feature or request SQL new-engine-gap backport 2.x labels Mar 29, 2023

acarbonetto mentioned this pull request Mar 29, 2023

SQL ingestion and ops - standup #791

Open

Yury-Fridlyand mentioned this pull request Mar 31, 2023

Rework on serialization and deserialization Bit-Quill/opensearch-project-sql#252

Closed

6 tasks

Yury-Fridlyand added 2 commits March 31, 2023 18:27

Make scroll timeout configurable.

4948ac4

Signed-off-by: Yury-Fridlyand <[email protected]>

Fix IT to set cursor keep alive parameter.

37e7ebf

Signed-off-by: Yury-Fridlyand <[email protected]>

This was referenced Apr 5, 2023

Support pagination in V2 engine, phase 1 #1497

Merged

Rework on serialization and deserialization in pagination #1498

Merged

Yury-Fridlyand closed this Apr 5, 2023

Yury-Fridlyand added the pagination Pagination feature, ref #656 label May 1, 2023

Yury-Fridlyand deleted the integ-pagination-p1 branch June 24, 2023 00:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support pagination in V2 engine, phase 1 (#226) #1483

Support pagination in V2 engine, phase 1 (#226) #1483

Yury-Fridlyand commented Mar 28, 2023 •

edited

Loading

Yury-Fridlyand left a comment

Yury-Fridlyand Mar 28, 2023

Yury-Fridlyand Mar 28, 2023

Yury-Fridlyand Mar 28, 2023

This comment was marked as spam.

GumpacG commented Apr 4, 2023 •

edited by Yury-Fridlyand

Loading

acarbonetto commented Apr 5, 2023 •

edited

Loading

Yury-Fridlyand commented Apr 5, 2023

Support pagination in V2 engine, phase 1 (#226) #1483

Support pagination in V2 engine, phase 1 (#226) #1483

Conversation

Yury-Fridlyand commented Mar 28, 2023 • edited Loading

Description

Further improvements are coming

Issues Resolved

Check List

Yury-Fridlyand left a comment

Choose a reason for hiding this comment

Yury-Fridlyand Mar 28, 2023

Choose a reason for hiding this comment

Yury-Fridlyand Mar 28, 2023

Choose a reason for hiding this comment

Yury-Fridlyand Mar 28, 2023

Choose a reason for hiding this comment

This comment was marked as spam.

GumpacG commented Apr 4, 2023 • edited by Yury-Fridlyand Loading

acarbonetto commented Apr 5, 2023 • edited Loading

Yury-Fridlyand commented Apr 5, 2023

Yury-Fridlyand commented Mar 28, 2023 •

edited

Loading

GumpacG commented Apr 4, 2023 •

edited by Yury-Fridlyand

Loading

acarbonetto commented Apr 5, 2023 •

edited

Loading