[Lens] show meta field data in Lens #77210

flash1293 · 2020-09-10T16:39:44Z

This PR makes sure meta fields are also handled correctly in terms of existence data and stats.

The biggest change here is to pass down the list of meta fields from advanced settings to the existence checker and look them up directly on the document instead of _source or fields.

Also, the meta flag is made part of the field type to introduce a new "Meta fields" section in the field list.

There are two related issues due to buggy handling of meta fields in Kibana in general which also show up here:

The id field is showing up but throws an error when trying to aggregate it. This is not a Lens bug and has to be solved on index pattern level (or maybe ES level) Index pattern: "_id" field marked as aggregatable #77206
_type is showing up as empty. This is not a Lens bug as well, IMHO the field shouldn't even be present in the index pattern in the first place. Again, this is either an index pattern bug or an ES bug (didn't dig into it): Discover: "_type" showing up as empty #77209 It should work fine on 7.x

cchaos · 2020-09-10T18:18:53Z

Should meta fields be indicated in the field list somehow?

Maybe a different section like the empty fields?

flash1293 · 2020-09-10T21:24:42Z

Maybe a different section like the empty fields?

I can definitely see a separate section in the very bottom (as they are an edge case anyway), with using the color to indicate emptiness (instead of having both “available meta fields” and “empty meta fields” sections, that seems like overkill). I will adjust the PR

elasticmachine · 2020-09-15T11:54:24Z

Pinging @elastic/kibana-app (Team:KibanaApp)

mbondyra

Tested on FF and all works well. The code looks good. Just one note that is not even related to this PR - the last element of the accordion is cut (you didn't introduce it though):

Removing the last 0 from these styles solves the problem, but as it's not related to this PR, feel free to ignore this comment :)

…dions

flash1293 · 2020-09-16T15:54:28Z

@mbondyra Refactored the different sections into a FieldList component if you want to take another look

cchaos

👍 To the separated section. This way the user can mostly just ignore those altogether.

A few questions/feedback:

Should we allow users to drag-n-drop these or select them to be aggregated at all? Some of them completely error and others are most likely not unique at all.
Can you also update the organization within the field selector combo box?

There's seems to be a discrepency between the messaging and the look of the _id field. It shows as if it has data (white bg) but the popover says it doesn't have any.

Can we update the information icon tooltips to have better messaging per category of field?

Empty fields

Tooltip:

This field doesn't have any data but you can still drag and drop to visualize.

Popover:

This field is empty because it doesn’t exist in the 500 sampled documents. Adding this field to the configuration may result in a blank chart.

Meta fields

The messaging will depend on if we decide they can still add these to the chart, but if we say they can't then:

Tooltip:

Meta fields cannot be visualized. Click for a field preview.

cchaos · 2020-09-16T17:00:20Z

x-pack/plugins/lens/public/indexpattern_datasource/field_list.tsx

+
+  return (
+    <div
+      className="lnsInnerIndexPatternDataPanel__listWrapper"


Don't forget now to update class names to align with the component name and move the SASS to be imported directly by the JS component.

flash1293 · 2020-09-17T11:38:25Z

Thanks for taking a look, @cchaos

Should we allow users to drag-n-drop these or select them to be aggregated at all? Some of them completely error and others are most likely not unique at all.

There are currently problems with _id and _type (see linked issues in the PR description, they are out of scope for this PR, but should get addressed), but in general aggregating meta fields, while being an edge case, can be useful and is supported in the rest of Kibana as well - I think we should support them (and we already do today, although in a buggy way).

Can you also update the organization within the field selector combo box?

Great point, fixed that.

There's seems to be a discrepency between the messaging and the look of the _id field. It shows as if it has data (white bg) but the popover says it doesn't have any.

_id is currently broken unrelated to this PR (see PR description)

Can we update the information icon tooltips to have better messaging per category of field?

Makes sense, done

The messaging will depend on if we decide they can still add these to the chart, but if we say they can't then:

As mentioned they should be addable to the chart, everything else is a bug. To give a bit of context here, there are also plugins adding additional meta fields which are supported in Kibana used from time to time - those might be more helpful than _id and _type (_index is already a good example of a useful meta field)

Don't forget now to update class names to align with the component name and move the SASS to be imported directly by the JS component.

Thanks, moved the SASS over

wylieconlon

Initial testing shows that there is a large performance regression with large index patterns when interacting with the filters, compared to the previous behavior.

flash1293 · 2020-09-22T17:12:57Z

Ah, I planned to check whether memoization and so on still works after the refactoring and forgot. I will take a look, thanks for taking performance into account 👍

wylieconlon · 2020-09-22T17:17:12Z

x-pack/plugins/lens/public/indexpattern_datasource/datapanel.tsx

        } else if (containsData(field)) {
          return 'availableFields';
        } else return 'emptyFields';
      }),
    };
-  }, [allFields, existingFields, currentIndexPattern, hasSyncedExistingFields]);


I traced the performance, and the main issue is that this first set of memoized code is expensive to run, but is now running on each keystroke.

cchaos

LGTM, thanks @flash1293

x-pack/plugins/lens/public/indexpattern_datasource/field_list.tsx

flash1293 · 2020-09-23T12:23:17Z

@wylieconlon Improved the performance of field group calculation by splitting out the filtering in a separate hook like it was done before.

flash1293 · 2020-09-23T12:44:33Z

While looking into performance, I noticed a pretty large performance bottleneck which was super easy to remove: 7ee7bbb

We were building a field index by name for the whole field list on each containsData check (which is running for each single field).

It looks like this would close #77898 as well

kibanamachine · 2020-09-23T14:14:27Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 7ee7bbb

Build metrics

@kbn/optimizer bundle module count

id	value	diff	baseline
lens	471	+10	461

page load bundle size

id	value	diff	baseline
lens	1.1MB	+7.1KB	1.1MB

History

💚 Build #76496 succeeded 3013d99
💚 Build #75818 succeeded 52fd78d
💚 Build #75150 succeeded fdc9d2b
💚 Build #74798 succeeded 08164ef
💔 Build #74776 failed 1592748

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

flash1293 added 2 commits September 10, 2020 18:24

show meta field data in Lens

cbb4db0

add comment

58ffcb4

flash1293 added release_note:fix v8.0.0 Feature:Lens v7.10.0 labels Sep 10, 2020

flash1293 added 4 commits September 14, 2020 15:20

Merge remote-tracking branch 'upstream/master' into lens/system-fields

3c0730e

add separate section for meta fields

0ad5d18

Merge remote-tracking branch 'upstream/master' into lens/system-fields

1aa9b94

fix api tests

eb19274

flash1293 marked this pull request as ready for review September 15, 2020 11:54

flash1293 requested a review from a team September 15, 2020 11:54

flash1293 added the Team:Visualizations Visualization editors, elastic-charts and infrastructure label Sep 15, 2020

mbondyra approved these changes Sep 15, 2020

View reviewed changes

flash1293 added 4 commits September 16, 2020 11:22

commit current state

dcc3fc3

Merge remote-tracking branch 'upstream/master' into lens/system-fields

4772717

create general fieldlist component handling arbitrary number of accor…

1592748

…dions

Merge remote-tracking branch 'upstream/master' into lens/system-fields

08164ef

cchaos suggested changes Sep 16, 2020

View reviewed changes

flash1293 added 2 commits September 17, 2020 13:20

Merge remote-tracking branch 'upstream/master' into lens/system-fields

01e29f7

review comments

fed77da

fix scss

fdc9d2b

flash1293 requested a review from a team as a code owner September 17, 2020 11:42

Merge remote-tracking branch 'upstream/master' into lens/system-fields

52fd78d

wylieconlon reviewed Sep 22, 2020

View reviewed changes

cchaos approved these changes Sep 22, 2020

View reviewed changes

x-pack/plugins/lens/public/indexpattern_datasource/field_list.tsx Outdated Show resolved Hide resolved

flash1293 added 2 commits September 23, 2020 14:01

Merge remote-tracking branch 'upstream/master' into lens/system-fields

a642ac9

memoize field groups

3013d99

improve performance

7ee7bbb

flash1293 merged commit 839817c into elastic:master Sep 24, 2020

flash1293 added a commit to flash1293/kibana that referenced this pull request Sep 24, 2020

[Lens] show meta field data in Lens (elastic#77210)

a69f2fb

This was referenced Sep 24, 2020

[7.x] [Lens] show meta field data in Lens (#77210) #78388

Merged

[Lens] Identify and fix performance bottlenecks with large index patterns #77898

Closed

flash1293 added a commit that referenced this pull request Sep 24, 2020

[7.x] [Lens] show meta field data in Lens (#77210) (#78388)

6a3ba80

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Lens] show meta field data in Lens #77210

[Lens] show meta field data in Lens #77210

flash1293 commented Sep 10, 2020 •

edited

Loading

cchaos commented Sep 10, 2020

flash1293 commented Sep 10, 2020 •

edited

Loading

elasticmachine commented Sep 15, 2020

mbondyra left a comment •

edited

Loading

flash1293 commented Sep 16, 2020

cchaos left a comment

cchaos Sep 16, 2020

flash1293 commented Sep 17, 2020 •

edited

Loading

wylieconlon left a comment

flash1293 commented Sep 22, 2020

wylieconlon Sep 22, 2020

cchaos left a comment

flash1293 commented Sep 23, 2020

flash1293 commented Sep 23, 2020

kibanamachine commented Sep 23, 2020

[Lens] show meta field data in Lens #77210

[Lens] show meta field data in Lens #77210

Conversation

flash1293 commented Sep 10, 2020 • edited Loading

cchaos commented Sep 10, 2020

flash1293 commented Sep 10, 2020 • edited Loading

elasticmachine commented Sep 15, 2020

mbondyra left a comment • edited Loading

Choose a reason for hiding this comment

flash1293 commented Sep 16, 2020

cchaos left a comment

Choose a reason for hiding this comment

A few questions/feedback:

cchaos Sep 16, 2020

Choose a reason for hiding this comment

flash1293 commented Sep 17, 2020 • edited Loading

wylieconlon left a comment

Choose a reason for hiding this comment

flash1293 commented Sep 22, 2020

wylieconlon Sep 22, 2020

Choose a reason for hiding this comment

cchaos left a comment

Choose a reason for hiding this comment

flash1293 commented Sep 23, 2020

flash1293 commented Sep 23, 2020

kibanamachine commented Sep 23, 2020

💚 Build Succeeded

Build metrics

@kbn/optimizer bundle module count

page load bundle size

History

flash1293 commented Sep 10, 2020 •

edited

Loading

flash1293 commented Sep 10, 2020 •

edited

Loading

mbondyra left a comment •

edited

Loading

flash1293 commented Sep 17, 2020 •

edited

Loading