Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Search Source] Fix retrieval of unmapped fields; Add field filters #89837

Merged
merged 20 commits into from
Feb 15, 2021
Merged

[Search Source] Fix retrieval of unmapped fields; Add field filters #89837

Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
e870118
[Search Source] Remove includes when retrieving fields from source
Feb 1, 2021
a691322
Merge branch 'master' into inclusion-filters
kibanamachine Feb 1, 2021
5321c06
Removing unusded imports
Feb 1, 2021
170cb50
Use exclusion filters to retrieve a list of fields
Feb 1, 2021
08a25e6
Exclude _source from fields list
Feb 3, 2021
892d18c
Merge branch 'master' into inclusion-filters
Feb 3, 2021
0788854
Fix small check in getting the field list
Feb 3, 2021
736d6a7
Fixing faulty import
Feb 4, 2021
0364594
Filter logic
Feb 4, 2021
ff9cd2b
Adding a unit test for maps use case
Feb 4, 2021
2872c36
Merge master
Feb 4, 2021
f7c845e
Updating maps unit & functional test
Feb 7, 2021
21b011e
Add unit test
Feb 7, 2021
0bca218
Merge branch 'master' into inclusion-filters
kibanamachine Feb 8, 2021
259cf6b
Move logic for requesting a field list inside search_source
Feb 10, 2021
4279ef8
Merge branch 'master' into inclusion-filters
kibanamachine Feb 10, 2021
c1aef7d
Remove unnecessary mock
Feb 10, 2021
a9c9389
Code cleanup as per PR comments
Feb 11, 2021
5b94504
Merge branch 'master' into inclusion-filters
kibanamachine Feb 12, 2021
01aa170
Merge branch 'master' into inclusion-filters
kibanamachine Feb 15, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions src/plugins/data/common/search/search_source/search_source.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -836,5 +836,27 @@ describe('SearchSource', () => {
expect(references[1].type).toEqual('index-pattern');
expect(JSON.parse(searchSourceJSON).filter[0].meta.indexRefName).toEqual(references[1].name);
});

test('mvt geoshape layer test', async () => {
// @ts-expect-error TS won't like using this field name, but technically it's possible.
searchSource.setField('docvalue_fields', ['prop1']);
searchSource.setField('source', ['geometry']);
searchSource.setField('fieldsFromSource', ['geometry', 'prop1']);
searchSource.setField('index', ({
...indexPattern,
getSourceFiltering: () => ({ excludes: [] }),
getComputedFields: () => ({
storedFields: ['*'],
scriptFields: {},
docvalueFields: [],
}),
} as unknown) as IndexPattern);
const request = await searchSource.getSearchRequestBody();
expect(request.stored_fields).toEqual(['geometry', 'prop1']);
expect(request.docvalue_fields).toEqual(['prop1']);
expect(request._source).toEqual({
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for moving this into its own test case.

After inspection, the behavior in master branch, where _source: { includes: ['geometry', 'prop1']} is not optimal and this PR's behavior is better. It is safe to update the expect statement to _source: { 'geometry'}. This PR does a better job of removing duplicate fields from _source which is great, thanks.

The existing behavior in master branch results in hits like the below. Notice how prop1's value is returned twice, once in _source and once in fields.

{
        "_index": "geo_shapes",
        "_id": "4",
        "_score": 0,
        "_source": {
          "prop1": 4,
          "geometry": { ...omitted }
        },
        "fields": {
          "prop1": [
            4
          ]
        }
      }

With this PR in place, the behavior is optimal and prop1's value is only returned a single time.

{
        "_index": "geo_shapes",
        "_id": "4",
        "_score": 0,
        "_source": {
          "geometry": {...omitted}
        },
        "fields": {
          "prop1": [
            4
          ]
        }
      }

includes: ['geometry', 'prop1'],
});
});
});
});
25 changes: 14 additions & 11 deletions src/plugins/data/common/search/search_source/search_source.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,9 +59,10 @@
*/

import { setWith } from '@elastic/safer-lodash-set';
majagrubic marked this conversation as resolved.
Show resolved Hide resolved
import { uniqueId, keyBy, pick, difference, omit, isObject, isFunction } from 'lodash';
import { uniqueId, keyBy, pick, difference, omit, isFunction, isEqual } from 'lodash';
import { map, switchMap, tap } from 'rxjs/operators';
import { defer, from } from 'rxjs';
import { isObject } from 'rxjs/internal-compatibility';
import { normalizeSortRequest } from './normalize_sort_request';
import { fieldWildcardFilter } from '../../../../kibana_utils/common';
import { IIndexPattern } from '../../index_patterns';
Expand Down Expand Up @@ -503,7 +504,6 @@ export class SearchSource {
private flatten() {
const { getConfig } = this.dependencies;
const searchRequest = this.mergeProps();

searchRequest.body = searchRequest.body || {};
const { body, index, query, filters, highlightAll } = searchRequest;
searchRequest.indexType = this.getIndexType(index);
Expand Down Expand Up @@ -539,7 +539,7 @@ export class SearchSource {
if (!body.hasOwnProperty('_source')) {
body._source = sourceFilters;
}
if (body._source.excludes) {
if (body._source) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will always evaluate to true and can probably be removed. (If body._source is undefined, we assign it on L596 above, and index.getSourceFiltering() will always be truthy).

const filter = fieldWildcardFilter(
body._source.excludes,
getConfig(UI_SETTINGS.META_FIELDS)
Expand Down Expand Up @@ -579,17 +579,20 @@ export class SearchSource {
const remainingFields = difference(uniqFieldNames, [
...Object.keys(body.script_fields),
...Object.keys(body.runtime_mappings),
]).filter(Boolean);
]).filter((remainingField) => {
if (!remainingField) return false;
if (!body._source || !body._source.excludes) return true;
return !body._source.excludes.includes(remainingField);
});

// only include unique values
body.stored_fields = [...new Set(remainingFields)];

// only include unique values
if (fieldsFromSource.length) {
// include remaining fields in _source
setWith(body, '_source.includes', remainingFields, (nsValue) =>
isObject(nsValue) ? {} : nsValue
);

if (!isEqual(remainingFields, fieldsFromSource)) {
setWith(body, '_source.includes', remainingFields, (nsValue) =>
isObject(nsValue) ? {} : nsValue
);
}
// if items that are in the docvalueFields are provided, we should
// make sure those are added to the fields API unless they are
// already set in docvalue_fields
Expand Down
36 changes: 36 additions & 0 deletions src/plugins/discover/public/application/helpers/get_field_list.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0 and the Server Side Public License, v 1; you may not use this file except
* in compliance with, at your election, the Elastic License 2.0 or the Server
* Side Public License, v 1.
*/

import { IndexPattern } from '../../../../data/common/index_patterns/index_patterns';

const META_FIELDS = ['_type', '_source'];

export function getFieldListFromIndexPattern(
indexPattern: IndexPattern,
showUnmappedFields?: boolean
) {
const { sourceFilters, fields } = indexPattern;
if (!sourceFilters || sourceFilters.length === 0) {
const allFields: Record<string, string> = { field: '*' };
if (showUnmappedFields) {
allFields.include_unmapped = 'true';
}
return [allFields];
}
const sourceFiltersValues = sourceFilters.map((sourceFilter) => sourceFilter.value);
const fieldsToInclude = fields.filter((field) => {
return !sourceFiltersValues.includes(field.name) && !META_FIELDS.includes(field.name);
});
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I may be overlooking something (and things might have changed since I last looked at this stuff), but I'm not sure if this part should be necessary.

The idea with search source is that you provide the index pattern, and this stuff happens internally so you don't need to worry about it. The filtering (and meta fields handling) should already happen inside search source via fieldsWildcardFilter.

As a consumer of search source, all you should be responsible for is providing a field list, and search source would ideally handle everything else.

But I might be oversimplifying: Is this addressing a case that is not already covered by search source?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this particular use case is supposed to take care of field filters when using the fields API. Unlike _source, fields API has no notion of excludes, but rather - includes. So we'd have to request a list of all fields, except the ones specified by the filters, which is what this code does. The logic doesn't exist (or doesn't work 😅 ) in search_source yet. TBF, I am not sure where the right place for it is. It felt cleaner to be in Discover, but happy to discuss alternatives.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fields API has no notion of excludes, but rather - includes. So we'd have to request a list of all fields, except the ones specified by the filters, which is what this code does.

Ah, this is the point I missed. Yeah this does ultimately feel like something search source should handle. I would expect that if you called setField('fields', ['*']), then search source would do this internally. But let's see what @ppisljar thinks

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After some discussions on date_nanos handling, we have reached the conclusion that Discover would explicitly specify the date format for any date field when requesting fields (#90415 (comment)). I believe this logic belongs to the same place as requesting fields to include, so if we could now decide where that would be, it would be appreciated :)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Current functionality was that when not setting the fields we would get you all of them. We changed this on the searchSource level, so that we would no longer return unmapped ones when using the fieldsAPI. I think there is place for this logic to live in the searchsource, thus get to the old behaviour. We could also make this optional by exposing `showUnmappedFields' on the searchsource.

it seems from the linked PR that we are handling date_nanos on searchsource level as well, which makes sense imo

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will change this then so that the logic lives in search source 👍

return fieldsToInclude.map((field) => {
const fieldToInclude: Record<string, string> = { field: field.name };
if (showUnmappedFields) {
fieldToInclude.include_unmapped = 'true';
}
return fieldToInclude;
});
}
8 changes: 3 additions & 5 deletions src/plugins/discover/public/application/helpers/update_search_source.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import { SAMPLE_SIZE_SETTING, SORT_DEFAULT_ORDER_SETTING } from '../../../common
import { IndexPattern, ISearchSource } from '../../../../data/common/';
import { SortOrder } from '../../saved_searches/types';
import { DiscoverServices } from '../../build_services';
import { getFieldListFromIndexPattern } from './get_field_list';

/**
* Helper function to update the given searchSource before fetching/sharing/persisting
Expand Down Expand Up @@ -48,11 +49,8 @@ export function updateSearchSource(
.setField('filter', data.query.filterManager.getFilters());
if (useNewFieldsApi) {
searchSource.removeField('fieldsFromSource');
const fields: Record<string, string> = { field: '*' };
if (showUnmappedFields) {
fields.include_unmapped = 'true';
}
searchSource.setField('fields', [fields]);
const fields = getFieldListFromIndexPattern(indexPattern, showUnmappedFields);
searchSource.setField('fields', fields);
} else {
searchSource.removeField('fields');
const fieldNames = indexPattern.fields.map((field) => field.name);
Expand Down