[Discover] Use fields API to retrieve fields #83891

majagrubic · 2020-11-20T08:38:26Z

Summary

Release Notes: Support mapping defined runtime fields in Discover

This change implements the usage of the fields API in Discover. Changes are hidden behind a UI setting; if the value is off, everything is (supposed to be) working as previously.

Some notable changes:

fields column is formatted similarly to _source and in many ways behaves exactly like _source (ie. it's removed when a new column is added and cannot be removed if only column left)
support for multi-fields in the sidebar
using fields API will retrieve runtime fields 🎉 (follow the instructions here to create runtime fields for testing)

Current limitations of this PR:

expanded doc view hasn't been formatted when using the fields API (fields are not sorted and multifields are not grouped together) -> will be added in a follow-up PR
no support for unmapped fields, ie. Discover doesn't show unmapped fields -> will be added in a follow-up PR
tackling _source as a default column in the existing saved searches

Checklist

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
This was checked for cross-browser compatibility

For maintainers

This was checked for breaking API changes and was labeled appropriately

src/plugins/discover/public/application/angular/discover.js

kertal

Hurray for Clarice Salazar, who finally got a name also when using multi fields 🥳

src/plugins/discover/public/application/angular/discover.js

timroes · 2020-11-24T18:24:42Z

I just synced with Stacey about the backwardscompatibility mode again and wanted to leave some notes:

We should not use _source to query unmapped fields (until 8.0), but should use the [include_unmapped] option in the fields section of the API, which is developed in Support unmapped fields in search 'fields' option elasticsearch#65386 (issue: Support unmapped fields in search 'fields' option. elasticsearch#63690)
We should not set this for newly created saved searches, they should no longer show unmapped fields at all (the behavior we'll also enforce from 8.0 on for existing saved objects), meaning I think we don't want a global switch, but basically a "hidden" flag that will be used to set include_unmapped.

We still need to evaluate how include_unmapped works with * as a field, so can we simply set { field: '*', include_unmapped: true } or do we need to list them more specifically, since all the ES docs never show them used with a pure wildcard.

majagrubic · 2021-01-14T17:02:29Z

@elasticmachine merge upstream

kertal

Code LGTM, tested locally in Firefox, Chrome, Safari. Works! Great work 🥇!.keyword, runtime fields, alias fields they suddenly make sense. Could you open an issue to track remaining stuff that needs to be done? many thx!

majagrubic · 2021-01-15T07:58:24Z

@kertal I opened blocker issues and they're referenced above. will create another one for non-blocking issues

majagrubic · 2021-01-15T08:14:14Z

Tracking issue here: #83891

kibanamachine · 2021-01-19T14:54:39Z

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.

* [Discover] Use fields API to retrieve fields (#83891) * Add search source to example plugin. * Add uiSetting for fields API. * Update SearchSource to support fields API. * [PoC] reading from the fields API in Discover * Add N fields as a default column * Make fields column non-removeable * Do not add 'fields' to state * Remove fields from app state and read from source when needed * Remove fields column if a new column is added * Add search source to example plugin. * Add uiSetting for fields API. * Update SearchSource to support fields API. * Improve error handling in search examples plugin. * Add unit tests for legacy behavior. * Remove uiSettings feature flag; add fieldsFromSource config. * Rewrite flatten() based on final API design. * Update example app based on final API design. * Update maps app to use legacy fieldsFromSource. * Update Discover to use legacy fieldsFromSource. * Rename source filters to field filters. * Address feedback. * Update generated docs. * Update maps functional test. * Formatting fields column similar to _source * Moving logic for using search API to updating search source * Fix small merge error * Move useSource switch to Discover section of advanced settings * Do not use fields and source at the same time * Remove unmapped fields switch * Add basic support for grouping multifields * Remove output.txt * Fix some merge leftovers * Fix some merge leftovers * Fix merge errors * Fix typescript errors and update nested fields logic * Add a unit test * Fixing field formats * Fix multifield selection logic * Request all fields from source * Fix eslint * Fix default columns when switching between _source and fields * More unit tests * Update API changes * Add unit test for discover field details footer * Remove unused file * Remove fields formatting from index pattern * Remove unnecessary check * Addressing design comments * Fixing fields column display and renaming it to Document * Adding more unit tests * Adding a missing check for useNewFieldsAPI; minor fixes * Fixing typescript error * Remove unnecessary console statement * Add missing prop * Fixing import order * Adding functional test to test fields API * [Functional test] Clean up in after * Fixing context app * Addressing PR comments * Updating failed snapshot * Addressing PR comments * Fixing i18n translations, updating type * Addressing PR comments * Updating a functional test * Add a separate functional test for fields API * Read fields from source in a functional test * Skip buggy test * Use default behavior in functional tests * Fixing remaining failing tests * Fixing date-nanos test * Updating FLS test * Fixing yet another functional test * Skipping non-relevant tests * Fixing more tests * Update stub import in test * Fix import * Fix invalid import Co-authored-by: Luke Elmers <[email protected]> Co-authored-by: Kibana Machine <[email protected]> # Conflicts: # x-pack/test/functional/apps/security/doc_level_security_roles.js # x-pack/test/functional/apps/security/field_level_security.js * Fixing failing functional tests * Fixing failing functional test Co-authored-by: Kibana Machine <[email protected]>

kennethlwk · 2021-03-01T16:40:38Z

Hi, just wondering how do we get the fixed for this issue? We are using 7.3.1 Kibana, and we also see the same problem which is alias in Discover is not showing any value on the alias field, but only have value in actual text field.

Would be good if someone can tell us how do we fetch the changes, is it we just need to clone the latest code to the repo, then we will get the soluton.

kibanamachine · 2021-03-16T19:04:57Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: a37612a
Pipeline Steps (look for red circles / failed steps)
Interpreting CI Failures

Failed CI Steps

Metrics [docs]

‼️ ERROR: no builds found for mergeBase sha [82c30a3]

History

💚 Build #99047 succeeded a37612a
💚 Build #98934 succeeded f1bd9e7
💔 Build #98875 failed 0eec11a
💔 Build #98862 failed 02c37cf

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

lukeelmers and others added 4 commits November 2, 2020 10:05

Add search source to example plugin.

a232550

Add uiSetting for fields API.

ba796e0

Update SearchSource to support fields API.

9fbcc82

[PoC] reading from the fields API in Discover

5afd264

majagrubic changed the title ~~Discover ssource fields~~ [PoC] Discover uses fields API Nov 20, 2020

majagrubic commented Nov 20, 2020

View reviewed changes

src/plugins/discover/public/application/angular/discover.js Outdated Show resolved Hide resolved

majagrubic commented Nov 20, 2020

View reviewed changes

src/plugins/discover/public/application/angular/discover.js Show resolved Hide resolved

majagrubic commented Nov 20, 2020

View reviewed changes

src/plugins/discover/public/application/angular/discover.js Outdated Show resolved Hide resolved

majagrubic commented Nov 20, 2020

View reviewed changes

src/plugins/discover/public/application/angular/discover.js Outdated Show resolved Hide resolved

kertal reviewed Nov 20, 2020

View reviewed changes

Maja Grubic added 2 commits November 20, 2020 16:10

Add N fields as a default column

ac1e273

Make fields column non-removeable

e6f2a22

kertal reviewed Nov 23, 2020

View reviewed changes

src/plugins/discover/public/application/angular/discover.js Outdated Show resolved Hide resolved

Maja Grubic and others added 16 commits November 25, 2020 08:46

Merge master

c420e96

Do not add 'fields' to state

8ad5f99

Remove fields from app state and read from source when needed

64f0868

Remove fields column if a new column is added

b843d33

Add search source to example plugin.

1a3196d

Add uiSetting for fields API.

0cf84e4

Update SearchSource to support fields API.

ad23b7f

Improve error handling in search examples plugin.

3681311

Add unit tests for legacy behavior.

5ac33bb

Remove uiSettings feature flag; add fieldsFromSource config.

9e6c89f

Rewrite flatten() based on final API design.

d34ab66

Update example app based on final API design.

4773949

Update maps app to use legacy fieldsFromSource.

d272609

Update Discover to use legacy fieldsFromSource.

30c7c5f

Rename source filters to field filters.

24a5f9d

Address feedback.

9c7a967

Maja Grubic added 2 commits January 14, 2021 14:30

Fix import

02c37cf

Fix invalid import

0eec11a

Merge branch 'master' into discover-ssource-fields

f1bd9e7

kertal approved these changes Jan 15, 2021

View reviewed changes

Merge branch 'master' into discover-ssource-fields

a37612a

majagrubic merged commit 9b22789 into elastic:master Jan 15, 2021

majagrubic deleted the discover-ssource-fields branch January 15, 2021 14:51

majagrubic mentioned this pull request Jan 15, 2021

[7.x] [Discover] Use fields API to retrieve fields (#83891) #88493

Merged

kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Jan 19, 2021

kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Jan 19, 2021

kertal mentioned this pull request Feb 1, 2021

Discover doesn't show field aliases #21705

Closed

kertal mentioned this pull request Feb 1, 2021

[Discover] Adapt default column behavior #89826

Merged

1 task

timroes mentioned this pull request Feb 23, 2021

Discover shows whole json object #15951

Closed

majagrubic added the v8.0.0 label Feb 23, 2021

timroes mentioned this pull request Feb 23, 2021

Show runtime field values in Discover #77107

Closed

1 task

timroes added release_note:enhancement release_note:feature Makes this part of the condensed release notes and removed release_note:roadmap release_note:enhancement labels Mar 10, 2021

dmlemeshko mentioned this pull request May 12, 2021

Find objects API do not work properly for index patterns #97600

Closed

kertal mentioned this pull request Jun 30, 2021

[Discover] Fix multiline display in classic table #103499

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Discover] Use fields API to retrieve fields #83891

[Discover] Use fields API to retrieve fields #83891

majagrubic commented Nov 20, 2020 •

edited by timroes

Loading

kertal left a comment

timroes commented Nov 24, 2020

majagrubic commented Jan 14, 2021

kertal left a comment

majagrubic commented Jan 15, 2021

majagrubic commented Jan 15, 2021

kibanamachine commented Jan 19, 2021

kennethlwk commented Mar 1, 2021

kibanamachine commented Mar 16, 2021 •

edited

Loading

[Discover] Use fields API to retrieve fields #83891

[Discover] Use fields API to retrieve fields #83891

Conversation

majagrubic commented Nov 20, 2020 • edited by timroes Loading

Summary

Checklist

For maintainers

kertal left a comment

Choose a reason for hiding this comment

timroes commented Nov 24, 2020

majagrubic commented Jan 14, 2021

kertal left a comment

Choose a reason for hiding this comment

majagrubic commented Jan 15, 2021

majagrubic commented Jan 15, 2021

kibanamachine commented Jan 19, 2021

kennethlwk commented Mar 1, 2021

kibanamachine commented Mar 16, 2021 • edited Loading

💔 Build Failed

Failed CI Steps

Metrics [docs]

History

majagrubic commented Nov 20, 2020 •

edited by timroes

Loading

kibanamachine commented Mar 16, 2021 •

edited

Loading