Skip to content

Commit

Permalink
[fix] Using max_result_window to set up MVT size request leads to not…
Browse files Browse the repository at this point in the history
… showing all results (elastic#171344)

Closes elastic#170272

The cause of the problem is that Maps is not setting
[_mvt](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-vector-tile-api.html)
`size` parameter.

Lets illustrate the problem with an example.
* index.max_result_window is set to 20,000.
* There are 15,000 hits matching the query for the tile, but because
`size` is not provided, the tile returns the default size value of
10,000 hits.
* Tile request sets `track_total_hits` to 20,001. Tile meta returns
`"hits.total.relation": "eq"` and `"hits.total.value": 15000`
properties, indicating that all possible hits are returned in the tile.
* Maps UI does not display "incomplete results" even though 5000 hits
are not contained in the tile.
* This happens because `hits.total` properties from tile meta is used to
indicate "incomplete results" and the returned value indicates that all
results are contained in the tile. We know this is not true because the
tile actually only returned 10,000 hits and not all 15,000 matching
hits. Thus, without setting `size`, the tile is not returning all of the
hits indicated by `track_total_hits`.

### test instructions
1. install web logs sample data set
2. In dev tools, run the following
    ```
    POST kibana_sample_data_logs/_doc/
    {
      "geo": {
        "coordinates": "59,-106.5"
      },
      "@timestamp": "2023-08-01"
    }

    PUT /kibana_sample_data_logs/_settings
    {
      "index" : {
        "max_result_window" : 20000
      }
    }
    ```
3. Create new map
4. Set time range to `1 year ago to 1 year from now` to display all web
logs data
5. add documents layer. Verify point circled in image is displayed.
Without changes, this point would not be returned in the tile because
the tile would only return the first 10000 matches.
<img width="500" alt="Screenshot 2023-11-15 at 11 31 25 AM"
src="https://github.com/elastic/kibana/assets/373691/2249d571-24c4-41ad-975e-d44ee6d246ce">
  • Loading branch information
nreese authored Nov 16, 2023
1 parent 0b24e40 commit 95a8d6a
Show file tree
Hide file tree
Showing 3 changed files with 28 additions and 1 deletion.
20 changes: 20 additions & 0 deletions x-pack/plugins/maps/common/mvt_request_body.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,26 @@ describe('getHitsTileRequest', () => {
expect(path).toEqual('/my%20index/_mvt/my%20location/0/0/0');
});

test(`Should use requestBody.size to set both track_total_hits and size parameters`, () => {
const searchRequest = {
size: 20000,
runtime_mappings: {},
query: {},
};
const { body } = getHitsTileRequest({
buffer: 5,
risonRequestBody: rison.encode(searchRequest),
geometryFieldName: 'my location',
hasLabels: true,
index: 'my index',
x: 0,
y: 0,
z: 0,
});
expect(body?.track_total_hits).toEqual(20001);
expect(body?.size).toEqual(20000);
});

describe('sort', () => {
test(`Should include sort`, () => {
const searchRequest = {
Expand Down
8 changes: 7 additions & 1 deletion x-pack/plugins/maps/common/mvt_request_body.ts
Original file line number Diff line number Diff line change
Expand Up @@ -83,14 +83,20 @@ export function getHitsTileRequest({
if (!requestBody) {
throw new Error('Required requestBody parameter not provided');
}
const size = typeof requestBody.size === 'number' ? requestBody.size : 10000;
const tileRequestBody = {
buffer,
grid_precision: 0, // no aggs
exact_bounds: true,
extent: 4096, // full resolution,
query: requestBody.query,
runtime_mappings: requestBody.runtime_mappings,
track_total_hits: typeof requestBody.size === 'number' ? requestBody.size + 1 : false,
// Number of hits matching the query to count accurately
// Used to notify users of truncated results
track_total_hits: size + 1,
// Maximum number of features to return in the hits layer
// Used to fetch number of hits that correspondes with track_total_hits
size,
with_labels: hasLabels,
} as SearchMvtRequest['body'];
if (requestBody.fields) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,7 @@ test('Should return elasticsearch vector tile request for hits tiles', () => {
type: 'long',
},
},
size: 10000,
sort: [
{
'@timestamp': {
Expand Down

0 comments on commit 95a8d6a

Please sign in to comment.