[8.11] [fix] Using max_result_window to set up MVT size request leads…

… to not showing all results (#171344) (#171440)

# Backport

This will backport the following commits from `main` to `8.11`:
- [[fix] Using max_result_window to set up MVT size request leads to not
showing all results
(#171344)](#171344)

<!--- Backport version: 8.9.7 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Nathan
Reese","email":"[email protected]"},"sourceCommit":{"committedDate":"2023-11-16T19:11:13Z","message":"[fix]
Using max_result_window to set up MVT size request leads to not showing
all results (#171344)\n\nCloses
https://github.com/elastic/kibana/issues/170272\r\n\r\nThe cause of the
problem is that Maps is not
setting\r\n[_mvt](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-vector-tile-api.html)\r\n`size`
parameter.\r\n\r\nLets illustrate the problem with an example.\r\n*
index.max_result_window is set to 20,000.\r\n* There are 15,000 hits
matching the query for the tile, but because\r\n`size` is not provided,
the tile returns the default size value of\r\n10,000 hits.\r\n* Tile
request sets `track_total_hits` to 20,001. Tile meta
returns\r\n`\"hits.total.relation\": \"eq\"` and `\"hits.total.value\":
15000`\r\nproperties, indicating that all possible hits are returned in
the tile.\r\n* Maps UI does not display \"incomplete results\" even
though 5000 hits\r\nare not contained in the tile.\r\n* This happens
because `hits.total` properties from tile meta is used to\r\nindicate
\"incomplete results\" and the returned value indicates that
all\r\nresults are contained in the tile. We know this is not true
because the\r\ntile actually only returned 10,000 hits and not all
15,000 matching\r\nhits. Thus, without setting `size`, the tile is not
returning all of the\r\nhits indicated by `track_total_hits`.\r\n\r\n###
test instructions\r\n1. install web logs sample data set\r\n2. In dev
tools, run the following\r\n ```\r\n POST
kibana_sample_data_logs/_doc/\r\n {\r\n \"geo\": {\r\n \"coordinates\":
\"59,-106.5\"\r\n },\r\n \"@timestamp\": \"2023-08-01\"\r\n }\r\n\r\n
PUT /kibana_sample_data_logs/_settings\r\n {\r\n \"index\" : {\r\n
\"max_result_window\" : 20000\r\n }\r\n }\r\n ```\r\n3. Create new
map\r\n4. Set time range to `1 year ago to 1 year from now` to display
all web\r\nlogs data\r\n5. add documents layer. Verify point circled in
image is displayed.\r\nWithout changes, this point would not be returned
in the tile because\r\nthe tile would only return the first 10000
matches.\r\n<img width=\"500\" alt=\"Screenshot 2023-11-15 at 11 31 25
AM\"\r\nsrc=\"https://github.com/elastic/kibana/assets/373691/2249d571-24c4-41ad-975e-d44ee6d246ce\">","sha":"95a8d6a18f4f6e93ede4efdaaca880fc94378fb1","branchLabelMapping":{"^v8.12.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Team:Presentation","Feature:Maps","v8.12.0","v8.11.2"],"number":171344,"url":"https://github.com/elastic/kibana/pull/171344","mergeCommit":{"message":"[fix]
Using max_result_window to set up MVT size request leads to not showing
all results (#171344)\n\nCloses
https://github.com/elastic/kibana/issues/170272\r\n\r\nThe cause of the
problem is that Maps is not
setting\r\n[_mvt](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-vector-tile-api.html)\r\n`size`
parameter.\r\n\r\nLets illustrate the problem with an example.\r\n*
index.max_result_window is set to 20,000.\r\n* There are 15,000 hits
matching the query for the tile, but because\r\n`size` is not provided,
the tile returns the default size value of\r\n10,000 hits.\r\n* Tile
request sets `track_total_hits` to 20,001. Tile meta
returns\r\n`\"hits.total.relation\": \"eq\"` and `\"hits.total.value\":
15000`\r\nproperties, indicating that all possible hits are returned in
the tile.\r\n* Maps UI does not display \"incomplete results\" even
though 5000 hits\r\nare not contained in the tile.\r\n* This happens
because `hits.total` properties from tile meta is used to\r\nindicate
\"incomplete results\" and the returned value indicates that
all\r\nresults are contained in the tile. We know this is not true
because the\r\ntile actually only returned 10,000 hits and not all
15,000 matching\r\nhits. Thus, without setting `size`, the tile is not
returning all of the\r\nhits indicated by `track_total_hits`.\r\n\r\n###
test instructions\r\n1. install web logs sample data set\r\n2. In dev
tools, run the following\r\n ```\r\n POST
kibana_sample_data_logs/_doc/\r\n {\r\n \"geo\": {\r\n \"coordinates\":
\"59,-106.5\"\r\n },\r\n \"@timestamp\": \"2023-08-01\"\r\n }\r\n\r\n
PUT /kibana_sample_data_logs/_settings\r\n {\r\n \"index\" : {\r\n
\"max_result_window\" : 20000\r\n }\r\n }\r\n ```\r\n3. Create new
map\r\n4. Set time range to `1 year ago to 1 year from now` to display
all web\r\nlogs data\r\n5. add documents layer. Verify point circled in
image is displayed.\r\nWithout changes, this point would not be returned
in the tile because\r\nthe tile would only return the first 10000
matches.\r\n<img width=\"500\" alt=\"Screenshot 2023-11-15 at 11 31 25
AM\"\r\nsrc=\"https://github.com/elastic/kibana/assets/373691/2249d571-24c4-41ad-975e-d44ee6d246ce\">","sha":"95a8d6a18f4f6e93ede4efdaaca880fc94378fb1"}},"sourceBranch":"main","suggestedTargetBranches":["8.11"],"targetPullRequestStates":[{"branch":"main","label":"v8.12.0","labelRegex":"^v8.12.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/171344","number":171344,"mergeCommit":{"message":"[fix]
Using max_result_window to set up MVT size request leads to not showing
all results (#171344)\n\nCloses
https://github.com/elastic/kibana/issues/170272\r\n\r\nThe cause of the
problem is that Maps is not
setting\r\n[_mvt](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-vector-tile-api.html)\r\n`size`
parameter.\r\n\r\nLets illustrate the problem with an example.\r\n*
index.max_result_window is set to 20,000.\r\n* There are 15,000 hits
matching the query for the tile, but because\r\n`size` is not provided,
the tile returns the default size value of\r\n10,000 hits.\r\n* Tile
request sets `track_total_hits` to 20,001. Tile meta
returns\r\n`\"hits.total.relation\": \"eq\"` and `\"hits.total.value\":
15000`\r\nproperties, indicating that all possible hits are returned in
the tile.\r\n* Maps UI does not display \"incomplete results\" even
though 5000 hits\r\nare not contained in the tile.\r\n* This happens
because `hits.total` properties from tile meta is used to\r\nindicate
\"incomplete results\" and the returned value indicates that
all\r\nresults are contained in the tile. We know this is not true
because the\r\ntile actually only returned 10,000 hits and not all
15,000 matching\r\nhits. Thus, without setting `size`, the tile is not
returning all of the\r\nhits indicated by `track_total_hits`.\r\n\r\n###
test instructions\r\n1. install web logs sample data set\r\n2. In dev
tools, run the following\r\n ```\r\n POST
kibana_sample_data_logs/_doc/\r\n {\r\n \"geo\": {\r\n \"coordinates\":
\"59,-106.5\"\r\n },\r\n \"@timestamp\": \"2023-08-01\"\r\n }\r\n\r\n
PUT /kibana_sample_data_logs/_settings\r\n {\r\n \"index\" : {\r\n
\"max_result_window\" : 20000\r\n }\r\n }\r\n ```\r\n3. Create new
map\r\n4. Set time range to `1 year ago to 1 year from now` to display
all web\r\nlogs data\r\n5. add documents layer. Verify point circled in
image is displayed.\r\nWithout changes, this point would not be returned
in the tile because\r\nthe tile would only return the first 10000
matches.\r\n<img width=\"500\" alt=\"Screenshot 2023-11-15 at 11 31 25
AM\"\r\nsrc=\"https://github.com/elastic/kibana/assets/373691/2249d571-24c4-41ad-975e-d44ee6d246ce\">","sha":"95a8d6a18f4f6e93ede4efdaaca880fc94378fb1"}},{"branch":"8.11","label":"v8.11.2","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

Co-authored-by: Nathan Reese <[email protected]>

x-pack/plugins/maps/common/mvt_request_body.test.ts

@@ -52,6 +52,26 @@ describe('getHitsTileRequest', () => {
     expect(path).toEqual('/my%20index/_mvt/my%20location/0/0/0');
   });
 
+  test(`Should use requestBody.size to set both track_total_hits and size parameters`, () => {
+    const searchRequest = {
+      size: 20000,
+      runtime_mappings: {},
+      query: {},
+    };
+    const { body } = getHitsTileRequest({
+      buffer: 5,
+      risonRequestBody: rison.encode(searchRequest),
+      geometryFieldName: 'my location',
+      hasLabels: true,
+      index: 'my index',
+      x: 0,
+      y: 0,
+      z: 0,
+    });
+    expect(body?.track_total_hits).toEqual(20001);
+    expect(body?.size).toEqual(20000);
+  });
+
   describe('sort', () => {
     test(`Should include sort`, () => {
       const searchRequest = {

x-pack/plugins/maps/common/mvt_request_body.ts

@@ -83,14 +83,20 @@ export function getHitsTileRequest({
   if (!requestBody) {
     throw new Error('Required requestBody parameter not provided');
   }
+  const size = typeof requestBody.size === 'number' ? requestBody.size : 10000;
   const tileRequestBody = {
     buffer,
     grid_precision: 0, // no aggs
     exact_bounds: true,
     extent: 4096, // full resolution,
     query: requestBody.query,
     runtime_mappings: requestBody.runtime_mappings,
-    track_total_hits: typeof requestBody.size === 'number' ? requestBody.size + 1 : false,
+    // Number of hits matching the query to count accurately
+    // Used to notify users of truncated results
+    track_total_hits: size + 1,
+    // Maximum number of features to return in the hits layer
+    // Used to fetch number of hits that correspondes with track_total_hits
+    size,
     with_labels: hasLabels,
   } as SearchMvtRequest['body'];
   if (requestBody.fields) {

x-pack/plugins/maps/public/inspector/vector_tile_adapter/components/get_tile_request.test.ts

@@ -126,6 +126,7 @@ test('Should return elasticsearch vector tile request for hits tiles', () => {
           type: 'long',
         },
       },
+      size: 10000,
       sort: [
         {
           '@timestamp': {