[ML] Fix data visualizer grid failing if one of the fields failed and not updating when refreshed #115644

qn895 · 2021-10-19T20:13:42Z

Summary

Fix data viz failing in big batch if one of the fields failed. Previously, we batched the requests in groups, so if one the fields failed to aggregate for some reasons, ES will fail them all. This PR fixes it so that it will retry the fields in the failed batch individually, so even if one fail, it will fail gracefully.

Before:
Fix Field statistics in Discover did not accept searchSessionId and were not able to restore search sessions correctly.
Fix grid not fetching content when Refresh is clicked in Discover
Fix callout message for string fields to be smaller (also with smaller spacer) to match the rest of the table
Fix preview/field statistics not indicating that it's loading when being refreshed/updated
Fix Discover/Kibana keyword fields to have new tokenKeyword [EuiToken] Update tokenKeyword and rename old tokenKeyword as tokenTag eui#5251
Remove old routes under data_visualizer that are no longer in use with the migration to client side data.search requests

Reviewers hint

kibana-public team:
- Added new icons for keyword and histogram type (src/plugins/kibana_react/public/field_icon/field_icon.tsx).
Discover team:
- Telemetry is added to track when users toggle between the view mode
- Table is renamed to FieldStatisticsTable for clarity
- text and keyword types and labels are added to get_field_type_name
- Changed keyword field types to use EUI's new tokenKeyword.
ML team:
- Old routes under data_visualizer were deleted, along with API integration tests related the deleted routes
- Removed DataLoader

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
This was checked for cross-browser compatibility

Risk Matrix

Delete this section if it is not applicable to this PR.

Before closing this PR, invite QA, stakeholders, and other developers to identify risks that should be tested prior to the change/feature release.

When forming the risk matrix, consider some of the following examples and how they may potentially impact the change:

Risk	Probability	Severity	Mitigation/Notes
Multiple Spaces—unexpected behavior in non-default Kibana Space.	Low	High	Integration tests will verify that all features are still supported in non-default Kibana Space and when user switches between spaces.
Multiple nodes—Elasticsearch polling might have race conditions when multiple Kibana nodes are polling for the same tasks.	High	Low	Tasks are idempotent, so executing them multiple times will not result in logical error, but will degrade performance. To test for this case we add plenty of unit tests around this logic and document manual testing procedure.
Code should gracefully handle cases when feature X or plugin Y are disabled.	Medium	High	Unit tests will verify that any feature flag or plugin combination still results in our service operational.
See more potential risk examples

For maintainers

This was checked for breaking API changes and was labeled appropriately

…iscover

kertal · 2021-11-05T17:10:36Z

I have 2 question: shouldn't restoring the search session if field statistics are selected also show the field statistics? doesn't seem to work in the latest deployment I've tested

@kertal Thanks for catching that - it should indeed restore whether the session was using field statistics or not. I missed a small change after the switch to using Locator for session management #115681. It should be fixed here now cd65afc

thx confirmed it's fixed

If one request fails, what is displayed in this case?
Currently, it will just not display. For example, if the request to fetch the stats failed, it would just show the basic stats that we can gather. We can definitely add more transparency to what type of errors are happening in the background in a future follow up.

yes, I think this should be done, it's a good follow up

kertal

Data discovery team owned code LGTM, tested a-la-carte and works as expected.

Search session is now restored correctly
Adding error description for fields errors can be done in a follow up PR

I've got one last question to @Dosant, something I've observed . Watching restorable session from Nov 5th. makes me wonder why the number of searches differs so much (2 is the document view), everything else are field statistic. Can give you access to the instance if you want (next week)

…port

kibanamachine · 2021-11-08T20:01:57Z

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`dataVisualizer`	307	330	+23
`discover`	355	356	+1
total			+24

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`discover`	59	61	+2

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`dataVisualizer`	522.1KB	537.4KB	+15.2KB
`discover`	310.7KB	311.9KB	+1.2KB
`graph`	393.4KB	393.5KB	+102.0B
`lens`	953.7KB	953.8KB	+102.0B
`maps`	2.6MB	2.6MB	+102.0B
`osquery`	938.7KB	939.0KB	+306.0B
`securitySolution`	4.5MB	4.5MB	+102.0B
`stackAlerts`	159.7KB	159.8KB	+102.0B
total			+17.2KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`dataVisualizer`	10.2KB	9.8KB	-489.0B
`discover`	51.3KB	51.5KB	+168.0B
`presentationUtil`	68.4KB	68.5KB	+102.0B
total			-219.0B

Unknown metric groups

API count

id	before	after	diff
`discover`	85	89	+4

References to deprecated APIs

id	before	after	diff
`dataVisualizer`	252	237	-15

History

💚 Build #4869 succeeded 4370b44
💔 Build #4862 failed e8acc53e20f5194f7bc47103bf5bbb682eb032dc
💔 Build #4859 failed e48b539
💔 Build #4777 failed 8b3d4c3
💚 Build #4434 succeeded 41e7cb6
💔 Build #4074 failed 93016de

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @qn895

kibanamachine · 2021-11-08T20:38:53Z

💔 Backport failed

Status	Branch	Result
❌	8.0	Commit could not be cherrypicked due to conflicts

To backport manually run:
node scripts/backport --pr 115644

kibanamachine · 2021-11-10T20:39:49Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 115644 or prevent reminders by adding the backport:skip label.

… not updating when refreshed (elastic#115644) * [ML] Initial embed * [ML] Initial embed props * [ML] Add top nav link to data viz * Add visible fields * Add add data service to register links * Renames, refactor, use constants * Renames, refactor, use constants * Update tests and mocks * Embeddable * Update hook to update upon time udpate * Add filter support to query * Refactor filter utilities * Add filter support for embeddable * Fix saved search data undefined * Prototype aggregated view/document view switcher * Prototype flyout * Prototype save document view option in storage * Fix filter and query conflict with saved search * Minor styling edits * [ML] Initial embed * [ML] Initial embed props * Add embeddable 1 * Add visible fields * Embeddable 2 * Add filter support to query * Refactor filter utilities * Add filter support for embeddable * Fix saved search data undefined * Prototype aggregated view/document view switcher * Prototype flyout * Prototype save document view option in storage * Fix filter and query conflict with saved search * Minor styling edits * Fix missing code after conflicts * Remove dv locator and flyout * Make types happy * Fix types * Rename toggle option * Resolve conflicts * [ML] Reduce size of chart * [ML] Unbold name, switch icons of show distributions * [ML] Make size consistent * [ML] Make page size 25 * [ML] Switch to arrow right and down * [ML] Make legend font smaller * [ML] Add user setting * [ML] Add show preview by default setting * [ML] Match icon * Add panels around the subcontent * Add preference for aggregated vs doc * Fix types * Fix types, add constants for adv settings * Change to data view type * Temp fix for Kibana/EUI table overflow issue * Modify line height so text is not cut off, modify widths for varying screen sizes * Different width padders for different screens * Fix CI * Merge latest, move button to the right * Fix width for bar charts previews * Fix toggle buttons, fix maps * Delete unused file * Fix boolean styling * Change to enum, discover mode * Hide field stats * Hide field stats * Persist show mini preview/distribution settings * Remove window size, use size observer instead * Default to document view * Remove bold, switch icon * Set fixed width for top values, reduce font size in table * Fix custom url tests * Update width styling for panels * Fix missing flag for Discover sidebar, jest tests * Fix max width * Workaround for sorting * Fix import * Fix styling * Make height uniform, center alignment, fix map and keyword map not same size Move styling * Revert "Make height uniform, center alignment, fix map and keyword map not same size" This reverts commit 8fc42e2 * Revert "Make height uniform, center alignment, fix map and keyword map not same size" This reverts commit 8fc42e2 * Uniform height, left aligned, flex grid * Switch top values to have labels * Center content * Replace fixed widths with percentage * Fix table missing field types * Add dashboard embeddable and filter support * Fix file data viz styling and tests, lean up imports, remove hard coded pixels * Add search panel/kql filter bar * Temporarily fix scrolling * New kql filters for data visualizer * Set map height so it will fit the sampler shard size text * Use eui progress labels * Fix spacer * Add beta badge * Temporarily fix scrolling * Fix grow for Top Values for * [ML] Update functional tests to reflect new arrow icons * [ML] Add filter buttons and KQL bars * [ML] Update filter bar onChange behavior * [ML] Update top values filter onChange behavior * [ML] Update search filters when opening saved search * [ML] Clean up * [ML] Remove fit content for height * [ML] Fix boolean legend * [ML] Fix header section when browser width is small to large and when index pattern title is too large * [ML] Hide expander icon when dimension is xs or s & css fixes * [ML] Delete embeddables because they are not use * [ML] Rename view mode, refactor to separate hook, add error prompt if can't show, rename wrapper, clean up & fix tests * [ML] Make doc count 0 for empty fields, update t/f test * [ML] Add unit testing for search utils * Fix missing unsubscribe for embeddable output * Remove redundant onAddFilter for this PR, fix width * Rename Field Stats to Field stats to match convention * [ML] Fix expand all/collapse all behavior to override individual setting * [ML] Fix functional tests should be 0/0% * [ML] Fix docs content spacing, rename classnames, add filters to Discover, lens, and maps * [ML] Fix doc count for fields that exists but have no stats * [ML] Fix icon styling to match Discover but have text/keyword/histogram * [ML] Fix doc count for fields that exists but have no stats * [ML] Rename classnames to BEM style * Resolve latest changes * Add in place ss * Refactor helper functions * Refactor helper functions * Add error log * Migrate overall stats to data's search * Better handle errors * Fix url so restore session brings back correct view * Add progress bar * [ML] Add tests for data viz in Discover * [ML] Change to combinelatest * Update tests & dashboard behavior to reflect new advanced settings * Update telemetry * Remove workaround after eui bump fix * Remove dataloader * Snapshot * Migrate search to client side * Consolidate types * Change back to forkjoin instead of combinelatest for overallstats * Fix missing bool clause * Add login * Fix saved search attributes broken with latest changes * Update tests * Fix import * Match the no results found * Reset field stats so it reloads when query is refreshed * Reset field stats so it reloads when query is refreshed * Add doc stats * Merge to use hook completely * Merge to use hook completely * Fix doc chart doesn't show up when page is first mounted * Fix Discover auto refresh previously didn't update * Fix query util to return search source's results right away. Fix texts. * Refactor documentStats * Fix doc stats not showing upon page mount * Fix types * Delete old files * Update tests & i18n * Fix examples, tests * Remove old files & routes * Add telemetry, clean up, rename components for clarity * Fix size of callout message * Fix texts field * Consolidate field type * Consolidate field type, add count to top values * Clean up * Update tests * Remove progress on embedadble * Update snapshot * Clean up, consolidate searchOptions * Fix new es client types * Fix types * Fix loading state in Discover * Remove unused services, Change switchMap to map, mergeMap -> switchMap, update types * Fix missing filters * Fix message of table to show searching instead of no items found * Fix dashboard saved search source persisting time range * [ML] Fix table message state * [ML] Fix to not fetch field stats if cardinality is 0 * [ML] Fix locator missing view mode * [ML] Quit right away if field doesn't exist in docs * [ML] Change to use batch and only retry with individual field if failed * [ML] Batch requests for speed and retry failures for resiliency * No need to fetch field stats if overall stats haven't completed * Wait on overallStats to complete * Fix types after merge * Fix payload size too big 413, num of requests * Update field icon to using kbn/react-field package Co-authored-by: Kibana Machine <[email protected]>

… not updating when refreshed (#115644) (#118267) * [ML] Initial embed * [ML] Initial embed props * [ML] Add top nav link to data viz * Add visible fields * Add add data service to register links * Renames, refactor, use constants * Renames, refactor, use constants * Update tests and mocks * Embeddable * Update hook to update upon time udpate * Add filter support to query * Refactor filter utilities * Add filter support for embeddable * Fix saved search data undefined * Prototype aggregated view/document view switcher * Prototype flyout * Prototype save document view option in storage * Fix filter and query conflict with saved search * Minor styling edits * [ML] Initial embed * [ML] Initial embed props * Add embeddable 1 * Add visible fields * Embeddable 2 * Add filter support to query * Refactor filter utilities * Add filter support for embeddable * Fix saved search data undefined * Prototype aggregated view/document view switcher * Prototype flyout * Prototype save document view option in storage * Fix filter and query conflict with saved search * Minor styling edits * Fix missing code after conflicts * Remove dv locator and flyout * Make types happy * Fix types * Rename toggle option * Resolve conflicts * [ML] Reduce size of chart * [ML] Unbold name, switch icons of show distributions * [ML] Make size consistent * [ML] Make page size 25 * [ML] Switch to arrow right and down * [ML] Make legend font smaller * [ML] Add user setting * [ML] Add show preview by default setting * [ML] Match icon * Add panels around the subcontent * Add preference for aggregated vs doc * Fix types * Fix types, add constants for adv settings * Change to data view type * Temp fix for Kibana/EUI table overflow issue * Modify line height so text is not cut off, modify widths for varying screen sizes * Different width padders for different screens * Fix CI * Merge latest, move button to the right * Fix width for bar charts previews * Fix toggle buttons, fix maps * Delete unused file * Fix boolean styling * Change to enum, discover mode * Hide field stats * Hide field stats * Persist show mini preview/distribution settings * Remove window size, use size observer instead * Default to document view * Remove bold, switch icon * Set fixed width for top values, reduce font size in table * Fix custom url tests * Update width styling for panels * Fix missing flag for Discover sidebar, jest tests * Fix max width * Workaround for sorting * Fix import * Fix styling * Make height uniform, center alignment, fix map and keyword map not same size Move styling * Revert "Make height uniform, center alignment, fix map and keyword map not same size" This reverts commit 8fc42e2 * Revert "Make height uniform, center alignment, fix map and keyword map not same size" This reverts commit 8fc42e2 * Uniform height, left aligned, flex grid * Switch top values to have labels * Center content * Replace fixed widths with percentage * Fix table missing field types * Add dashboard embeddable and filter support * Fix file data viz styling and tests, lean up imports, remove hard coded pixels * Add search panel/kql filter bar * Temporarily fix scrolling * New kql filters for data visualizer * Set map height so it will fit the sampler shard size text * Use eui progress labels * Fix spacer * Add beta badge * Temporarily fix scrolling * Fix grow for Top Values for * [ML] Update functional tests to reflect new arrow icons * [ML] Add filter buttons and KQL bars * [ML] Update filter bar onChange behavior * [ML] Update top values filter onChange behavior * [ML] Update search filters when opening saved search * [ML] Clean up * [ML] Remove fit content for height * [ML] Fix boolean legend * [ML] Fix header section when browser width is small to large and when index pattern title is too large * [ML] Hide expander icon when dimension is xs or s & css fixes * [ML] Delete embeddables because they are not use * [ML] Rename view mode, refactor to separate hook, add error prompt if can't show, rename wrapper, clean up & fix tests * [ML] Make doc count 0 for empty fields, update t/f test * [ML] Add unit testing for search utils * Fix missing unsubscribe for embeddable output * Remove redundant onAddFilter for this PR, fix width * Rename Field Stats to Field stats to match convention * [ML] Fix expand all/collapse all behavior to override individual setting * [ML] Fix functional tests should be 0/0% * [ML] Fix docs content spacing, rename classnames, add filters to Discover, lens, and maps * [ML] Fix doc count for fields that exists but have no stats * [ML] Fix icon styling to match Discover but have text/keyword/histogram * [ML] Fix doc count for fields that exists but have no stats * [ML] Rename classnames to BEM style * Resolve latest changes * Add in place ss * Refactor helper functions * Refactor helper functions * Add error log * Migrate overall stats to data's search * Better handle errors * Fix url so restore session brings back correct view * Add progress bar * [ML] Add tests for data viz in Discover * [ML] Change to combinelatest * Update tests & dashboard behavior to reflect new advanced settings * Update telemetry * Remove workaround after eui bump fix * Remove dataloader * Snapshot * Migrate search to client side * Consolidate types * Change back to forkjoin instead of combinelatest for overallstats * Fix missing bool clause * Add login * Fix saved search attributes broken with latest changes * Update tests * Fix import * Match the no results found * Reset field stats so it reloads when query is refreshed * Reset field stats so it reloads when query is refreshed * Add doc stats * Merge to use hook completely * Merge to use hook completely * Fix doc chart doesn't show up when page is first mounted * Fix Discover auto refresh previously didn't update * Fix query util to return search source's results right away. Fix texts. * Refactor documentStats * Fix doc stats not showing upon page mount * Fix types * Delete old files * Update tests & i18n * Fix examples, tests * Remove old files & routes * Add telemetry, clean up, rename components for clarity * Fix size of callout message * Fix texts field * Consolidate field type * Consolidate field type, add count to top values * Clean up * Update tests * Remove progress on embedadble * Update snapshot * Clean up, consolidate searchOptions * Fix new es client types * Fix types * Fix loading state in Discover * Remove unused services, Change switchMap to map, mergeMap -> switchMap, update types * Fix missing filters * Fix message of table to show searching instead of no items found * Fix dashboard saved search source persisting time range * [ML] Fix table message state * [ML] Fix to not fetch field stats if cardinality is 0 * [ML] Fix locator missing view mode * [ML] Quit right away if field doesn't exist in docs * [ML] Change to use batch and only retry with individual field if failed * [ML] Batch requests for speed and retry failures for resiliency * No need to fetch field stats if overall stats haven't completed * Wait on overallStats to complete * Fix types after merge * Fix payload size too big 413, num of requests * Update field icon to using kbn/react-field package Co-authored-by: Kibana Machine <[email protected]> Co-authored-by: Kibana Machine <[email protected]>

qn895 added 30 commits July 19, 2021 14:31

[ML] Initial embed

c7a35e6

[ML] Initial embed props

5b48b41

[ML] Add top nav link to data viz

a080199

Add visible fields

c570d42

Add add data service to register links

b2dd8cb

Renames, refactor, use constants

28d54d0

Renames, refactor, use constants

4128150

Update tests and mocks

9fbffbf

Merge remote-tracking branch 'upstream/master' into ml-dv-in-discover

71833e0

Embeddable

eaa2563

Update hook to update upon time udpate

df58def

Merge remote-tracking branch 'upstream/master' into ml-dv-link-from-d…

66760b0

…iscover

Add filter support to query

e9e18b7

Refactor filter utilities

60aaa39

Merge in embeddable

95ee436

Add filter support for embeddable

e65edcf

Fix saved search data undefined

19eaa69

Prototype aggregated view/document view switcher

9215bcc

Prototype flyout

76f9ad6

Prototype save document view option in storage

0e4bacb

Fix filter and query conflict with saved search

c40ec0a

Minor styling edits

8f88ffd

[ML] Initial embed

cd4baee

[ML] Initial embed props

dedbbec

Add embeddable 1

f5a563f

Add visible fields

215a983

Embeddable 2

e76781c

Add filter support to query

eb827fa

Refactor filter utilities

b39328d

Add filter support for embeddable

51b43bd

qn895 added 2 commits November 4, 2021 10:41

[ML] Change to use batch and only retry with individual field if failed

817fe2d

[ML] Batch requests for speed and retry failures for resiliency

41e7cb6

kertal self-requested a review November 5, 2021 16:52

kertal approved these changes Nov 5, 2021

View reviewed changes

qn895 added 6 commits November 5, 2021 13:16

No need to fetch field stats if overall stats haven't completed

ccb0469

Merge upstream/main into branch

f88e1f4

Wait on overallStats to complete

8b3d4c3

Fix types after merge

3d2d8b8

Merge remote-tracking branch 'upstream/main' into dv-async-search-sup…

e48b539

…port

Fix payload size too big 413, num of requests

4370b44

qn895 force-pushed the dv-async-search-support branch from e8acc53 to 4370b44 Compare November 8, 2021 03:55

qn895 added 3 commits November 8, 2021 10:52

Merge upstream/main into branch

0ca94c6

Merge remote-tracking branch 'upstream/main' into dv-async-search-sup…

c811d4d

…port

Update field icon to using kbn/react-field package

9ea2537

qn895 added auto-backport Deprecated - use backport:version if exact versions are needed v8.1.0 labels Nov 8, 2021

qn895 merged commit 6e18f3f into elastic:main Nov 8, 2021

kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Nov 10, 2021

qn895 mentioned this pull request Nov 10, 2021

[8.0][ML] Fix data visualizer grid failing if one of the fields failed and not updating when refreshed #118267

Merged

kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Nov 11, 2021

qn895 deleted the dv-async-search-support branch January 20, 2022 17:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ML] Fix data visualizer grid failing if one of the fields failed and not updating when refreshed #115644

[ML] Fix data visualizer grid failing if one of the fields failed and not updating when refreshed #115644

qn895 commented Oct 19, 2021 •

edited

Loading

kertal commented Nov 5, 2021

kertal left a comment

kibanamachine commented Nov 8, 2021

API count

References to deprecated APIs

kibanamachine commented Nov 8, 2021

kibanamachine commented Nov 10, 2021

[ML] Fix data visualizer grid failing if one of the fields failed and not updating when refreshed #115644

[ML] Fix data visualizer grid failing if one of the fields failed and not updating when refreshed #115644

Conversation

qn895 commented Oct 19, 2021 • edited Loading

Summary

Reviewers hint

Checklist

Risk Matrix

For maintainers

kertal commented Nov 5, 2021

kertal left a comment

Choose a reason for hiding this comment

kibanamachine commented Nov 8, 2021

💚 Build Succeeded

Metrics [docs]

Module Count

Public APIs missing comments

Async chunks

Page load bundle

API count

References to deprecated APIs

History

kibanamachine commented Nov 8, 2021

💔 Backport failed

kibanamachine commented Nov 10, 2021

qn895 commented Oct 19, 2021 •

edited

Loading