[data views] Provide method of excluding data tiers when getting field list #167946

mattkime · 2023-10-04T02:32:33Z

Summary

This PR implements an Advanced setting (data_views:fields_excluded_data_tiers) that allows the exclusion of listed data tiers when getting a field list. The expected common use case would be excluding frozen indices to speed up slow field caps calls.

There is no serverless functionality since serverless doesn't have data tiers.

Additional information

This exclusion applies to all uses of DataViews in Kibana (including Discover and Alerting Rules).
There's one trade-off that fields which only exist in frozen tiers will appear unmapped in Discover and Lens
In case there are many historical fields in a frozen tier , the exclusion will lead to faster response time and a reduction of network traffic

resolves #167741

src/plugins/data_views/common/constants.ts

src/plugins/data_views/server/ui_settings.ts

mattkime · 2024-01-16T20:07:39Z

/ci

kertal · 2024-01-22T12:27:23Z

src/plugins/data_views/server/ui_settings.ts

+export const dataTiersUiSettingsConfig: Record<string, UiSettingsParams> = {
+  [DATA_VIEWS_FIELDS_EXCLUDED_TIERS]: {
+    name: i18n.translate('dataViews.advancedSettings.dataTiersName', {
+      defaultMessage: 'Data tiers excluded from field requests',
+    }),
+    value: '',
+    type: 'string',
+    description: i18n.translate('dataViews.advancedSettings.dataTiersText', {
+      defaultMessage:
+        'Exclude fields from specified tiers (such as data_frozen) for faster performance.',
+    }),
+    schema: schema.string(),
+  },
+};
+


@ninoslavmiskovic FYI, this is the advanced setting

kertal · 2024-01-22T22:12:30Z

@mattkime qq: apart from tests, this is done, right?

mattkime · 2024-01-22T22:22:11Z

@kertal Exactly!

elasticmachine · 2024-01-22T22:22:23Z

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

kertal · 2024-01-22T22:27:20Z

great! 👍
Well, ideally we could test for different tiers. but what should already work is, exclude data_content, which then should return no fields at all, exclude data_frozen which should then return all fields... not very granular, but a possible start

davismcphee

Code-only review. The code changes look good to me, just one minor piece of feedback so far. Including functional tests once we've figured out how would be great, although I'm also not opposed to merging with just unit tests and the tests @kertal suggested for now with additional functional tests as a followup if we think it could be a bit before we're able to add them.

src/plugins/data_views/server/rest_api_routes/internal/fields_for.ts

src/plugins/data_views/server/ui_settings.ts

davismcphee

Tested locally using the following sample data (thanks @kertal), and confirmed that adding data_cold to the exclusion list properly excludes cold_field from both the fields and fields_for_wildcard results:


PUT /test-cold
{
  "settings": {
    "index.routing.allocation.include._tier_preference": "data_cold"
  }
}

PUT /test-cold/_doc/1
{
  "cold_field": "test"
}

PUT /test-normal/_doc/1
{
  "normal_field": "test"
}

LGTM 👍 Would you mind adding a followup issue to add functional tests once the necessary QA functionality is merged?

davismcphee · 2024-01-31T01:43:33Z

src/plugins/data_views/server/ui_settings.ts

+    value: '',
+    type: 'string',
+    description: i18n.translate('dataViews.advancedSettings.dataTiersText', {
+      defaultMessage:


Is it worth indicating the expected comma-separated format, or do we think users will know what to expect for this?

kibana-ci · 2024-01-31T03:30:28Z

💚 Build Succeeded

Buildkite Build
Commit: 2e1bbe0

Metrics [docs]

✅ unchanged

History

💚 Build #190016 succeeded b23aafe
💚 Build #189947 succeeded 520ef68
💚 Build #188642 succeeded 6447460
💛 Build #187762 was flaky 5ea87bd
💚 Build #187087 succeeded 19cd661
💔 Build #187039 failed 318a2f6

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @mattkime

kertal

Tested a-la-carte, excluding data_cold, work as expected, fields are showing up unmapped 👍 thx for bringing this over the finishing line in no time

teresaalvarezsoler · 2024-02-01T08:39:13Z

Is this something we are also considering for Unified Search when users fetch fields for filtering, queriying, etc.?

kertal · 2024-02-01T08:59:10Z

@teresaalvarezsoler yes, since AFAIK these fields are provided by data views, so this also applies to these fields (there might be exceptions)

* main: use build hash in FTR tests [Security Solution] Fix moderate typo (elastic#175883) [Fleet] Fix conflicting dynamic template mappings for intermediate objects (elastic#175970) [Visualize] Prevent overwriting managed content (elastic#175274) [SLO] Add/edit form mark optional fields (elastic#175807) skip failing test suite (elastic#175984) [data views] Provide method of excluding data tiers when getting field list (elastic#167946) [Dataset quality] State management (elastic#174906) [Cloud Security] add is_internal config option for outputs (elastic#175546)

…d list (elastic#167946) ## Summary This PR implements an advanced setting that allows the exclusion of listed data tiers when getting a field list. The expected common use case would be excluding frozen indices to speed up slow field caps calls. There is no serverless functionality since serverless doesn't have data tiers.

first pass at excluding fields from frozen indices.

d6d9b4a

kertal reviewed Dec 28, 2023

View reviewed changes

src/plugins/data_views/common/constants.ts Outdated Show resolved Hide resolved

mattkime added 5 commits January 9, 2024 23:30

Merge branch 'main' into data_views_exclude_frozen

9499fb0

change to string, use for filter

7fc1679

Merge branch 'main' into data_views_exclude_frozen

4a82039

implement excluded tiers for field caps

c9cacdf

implement excluded tiers for field caps

5e51b9d

mattkime changed the title ~~first pass at excluding fields from frozen indices.~~ [data views] Provide method of excluding data teirs when getting field list Jan 11, 2024

kertal reviewed Jan 11, 2024

View reviewed changes

src/plugins/data_views/server/ui_settings.ts Outdated Show resolved Hide resolved

kertal added the Team:DataDiscovery Discover, search (e.g. data plugin and KQL), data views, saved searches. For ES|QL, use Team:ES|QL. label Jan 11, 2024

kertal assigned mattkime Jan 11, 2024

mattkime added 2 commits January 11, 2024 10:39

fix tier names, include in setting description

318a2f6

update setting telemetry

19cd661

mattkime changed the title ~~[data views] Provide method of excluding data teirs when getting field list~~ [data views] Provide method of excluding data tiers when getting field list Jan 11, 2024

Merge branch 'main' into data_views_exclude_frozen

5ea87bd

Merge branch 'main' into data_views_exclude_frozen

06a1171

kertal reviewed Jan 22, 2024

View reviewed changes

mattkime marked this pull request as ready for review January 22, 2024 22:22

mattkime requested review from a team as code owners January 22, 2024 22:22

Merge branch 'main' into data_views_exclude_frozen

6447460

pgayvallet approved these changes Jan 23, 2024

View reviewed changes

davismcphee reviewed Jan 24, 2024

View reviewed changes

src/plugins/data_views/server/rest_api_routes/internal/fields_for.ts Outdated Show resolved Hide resolved

kertal reviewed Jan 29, 2024

View reviewed changes

src/plugins/data_views/server/ui_settings.ts Outdated Show resolved Hide resolved

mattkime added 3 commits January 29, 2024 10:17

Update ui_settings.ts

520ef68

Merge branch 'main' into data_views_exclude_frozen

b98deaa

apply tier filter to new fields endpoint

b23aafe

mattkime added Feature:Data Views Data Views code and UI - index patterns before 8.0 release_note:feature Makes this part of the condensed release notes labels Jan 30, 2024

davismcphee approved these changes Jan 31, 2024

View reviewed changes

Update ui_settings.ts

2e1bbe0

mattkime mentioned this pull request Jan 31, 2024

[data views] Add funtional test for data tier filtering #175947

Closed

kertal self-requested a review January 31, 2024 09:07

kertal approved these changes Jan 31, 2024

View reviewed changes

mattkime merged commit ae52d87 into elastic:main Jan 31, 2024
19 checks passed

kibanamachine added v8.13.0 backport:skip This commit does not require backporting labels Jan 31, 2024

kertal mentioned this pull request Feb 1, 2024

[DataViews] Provide functionality to exclude configured data tiers in Advanced Settings #167741

Closed

kertal mentioned this pull request Apr 12, 2024

[DataViews] Excluding data tiers for field fetching doesn't work on data view rule execution #180675

Closed

mattkime added backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) and removed backport:skip This commit does not require backporting labels Apr 16, 2024

kibanamachine added the v8.14.0 label Apr 16, 2024

mattkime added backport:skip This commit does not require backporting and removed backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) v8.14.0 labels Apr 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[data views] Provide method of excluding data tiers when getting field list #167946

[data views] Provide method of excluding data tiers when getting field list #167946

mattkime commented Oct 4, 2023 •

edited by kertal

Loading

mattkime commented Jan 16, 2024

kertal Jan 22, 2024

kertal commented Jan 22, 2024

mattkime commented Jan 22, 2024

elasticmachine commented Jan 22, 2024

kertal commented Jan 22, 2024

davismcphee left a comment •

edited

Loading

davismcphee left a comment

davismcphee Jan 31, 2024

kibana-ci commented Jan 31, 2024

kertal left a comment

teresaalvarezsoler commented Feb 1, 2024

kertal commented Feb 1, 2024

[data views] Provide method of excluding data tiers when getting field list #167946

[data views] Provide method of excluding data tiers when getting field list #167946

Conversation

mattkime commented Oct 4, 2023 • edited by kertal Loading

Summary

Additional information

mattkime commented Jan 16, 2024

kertal Jan 22, 2024

Choose a reason for hiding this comment

kertal commented Jan 22, 2024

mattkime commented Jan 22, 2024

elasticmachine commented Jan 22, 2024

kertal commented Jan 22, 2024

davismcphee left a comment • edited Loading

Choose a reason for hiding this comment

davismcphee left a comment

Choose a reason for hiding this comment

davismcphee Jan 31, 2024

Choose a reason for hiding this comment

kibana-ci commented Jan 31, 2024

💚 Build Succeeded

Metrics [docs]

History

kertal left a comment

Choose a reason for hiding this comment

teresaalvarezsoler commented Feb 1, 2024

kertal commented Feb 1, 2024

mattkime commented Oct 4, 2023 •

edited by kertal

Loading

davismcphee left a comment •

edited

Loading