[dot-kibana-split] Split ingest types into `.kibana_ingest` #155130

pgayvallet · 2023-04-18T10:22:11Z

This PR is targeting the `dot-kibana-split` feature branch and not `main`

Summary

Part of #154888
Move the fleet/ingest saved object types to the .kibana_ingest index

Types moved to .kibana_ingest:

epm-packages
epm-packages-assets
fleet-fleet-server-host
fleet-message-signing-keys
fleet-preconfiguration-deletion-record
fleet-proxy
ingest_manager_settings
ingest-agent-policies
ingest-download-sources
ingest-outputs
ingest-package-policies

…ot-kibana-ingest

…et/kibana into split-dot-kibana-ingest

kibana-ci · 2023-04-19T14:09:49Z

💔 Build Failed

Failed CI Steps

Test Failures

[job] [logs] FTR Configs #11 / detection engine api security and spaces enabled - Group 1 install_prebuilt_rules_from_real_package should install prebuilt rules from the package storage
[job] [logs] FTR Configs #12 / detection engine api security and spaces enabled - Group 10 migrate_legacy_actions migrates legacy actions for rule with action run daily
[job] [logs] FTR Configs #12 / detection engine api security and spaces enabled - Group 10 migrate_legacy_actions migrates legacy actions for rule with action run daily
[job] [logs] FTR Configs #12 / detection engine api security and spaces enabled - Group 10 migrate_legacy_actions migrates legacy actions for rule with action run hourly
[job] [logs] FTR Configs #12 / detection engine api security and spaces enabled - Group 10 migrate_legacy_actions migrates legacy actions for rule with action run hourly
[job] [logs] FTR Configs #12 / detection engine api security and spaces enabled - Group 10 migrate_legacy_actions migrates legacy actions for rule with action run on every run
[job] [logs] FTR Configs #12 / detection engine api security and spaces enabled - Group 10 migrate_legacy_actions migrates legacy actions for rule with action run on every run
[job] [logs] FTR Configs #12 / detection engine api security and spaces enabled - Group 10 migrate_legacy_actions migrates legacy actions for rule with action run weekly
[job] [logs] FTR Configs #12 / detection engine api security and spaces enabled - Group 10 migrate_legacy_actions migrates legacy actions for rule with action run weekly
[job] [logs] FTR Configs #12 / detection engine api security and spaces enabled - Group 10 migrate_legacy_actions migrates legacy actions for rule with no actions
[job] [logs] FTR Configs #12 / detection engine api security and spaces enabled - Group 10 migrate_legacy_actions migrates legacy actions for rule with no actions

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`fleet`	798	799	+1

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/core-saved-objects-server`	101	102	+1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`fleet`	945.5KB	945.5KB	+1.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`fleet`	126.7KB	126.8KB	+51.0B

Unknown metric groups

API count

id	before	after	diff
`@kbn/core-saved-objects-server`	504	505	+1

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

elasticmachine · 2023-04-19T15:55:56Z

Pinging @elastic/kibana-core (Team:Core)

…ot-kibana-ingest

elasticmachine · 2023-04-19T16:01:36Z

Pinging @elastic/fleet (Team:Fleet)

@gsoldevila

## Description Fix #104081 This PR move some of the SO types from the `.kibana` index into the following ones: - `.kibana_alerting_cases` - `.kibana_analytics` - `.kibana_security_solution` - `.kibana_ingest` This split/reallocation will occur during the `8.8.0` Kibana upgrade (*meaning: from any version older than `8.8.0` to any version greater or equal to `8.8.0`*) **This PR main changes are:** - implement the changes required in the SO migration algorithm to support this reallocation - update the FTR tools (looking at you esArchiver) to support these new indices - update hardcoded references to `.kibana` and usage of the `core.savedObjects.getKibanaIndex()` to use new APIs to target the correct index/indices - update FTR datasets, tests and utility accordingly ## To reviewers **Overall estimated risk of regressions: low** But, still, please take the time to review changes in your code. The parts of the production code that were the most impacted are the telemetry collectors, as most of them were performing direct requests against the `.kibana` index, so we had to adapt them. Most other contributor-owned changes are in FTR tests and datasets. If you think a type is misplaced (either we missed some types that should be moved to a specific index, or some types were moved and shouldn't have been) please tell us, and we'll fix the reallocation either in this PR or in a follow-up. ## .Kibana split The following new indices are introduced by this PR, with the following SO types being moved to it. (any SO type not listed here will be staying in its current index) Note: The complete **_type => index_** breakdown is available in [this spreadsheet](https://docs.google.com/spreadsheets/d/1b_MG_E_aBksZ4Vkd9cVayij1oBpdhvH4XC8NVlChiio/edit#gid=145920788). #### `.kibana_alerting_cases` - action - action_task_params - alert - api_key_pending_invalidation - cases - cases-comments - cases-configure - cases-connector-mappings - cases-telemetry - cases-user-actions - connector_token - rules-settings - maintenance-window #### `.kibana_security_solution` - csp-rule-template - endpoint:user-artifact - endpoint:user-artifact-manifest - exception-list - exception-list-agnostic - osquery-manager-usage-metric - osquery-pack - osquery-pack-asset - osquery-saved-query - security-rule - security-solution-signals-migration - siem-detection-engine-rule-actions - siem-ui-timeline - siem-ui-timeline-note - siem-ui-timeline-pinned-event #### `.kibana_analytics` - canvas-element - canvas-workpad-template - canvas-workpad - dashboard - graph-workspace - index-pattern - kql-telemetry - lens - lens-ui-telemetry - map - search - search-session - search-telemetry - visualization #### `.kibana_ingest` - epm-packages - epm-packages-assets - fleet-fleet-server-host - fleet-message-signing-keys - fleet-preconfiguration-deletion-record - fleet-proxy - ingest_manager_settings - ingest-agent-policies - ingest-download-sources - ingest-outputs - ingest-package-policies ## Tasks / PRs ### Sub-PRs **Implementation** - 🟣 #154846 - 🟣 #154892 - 🟣 #154882 - 🟣 #154884 - 🟣 #155155 **Individual index split** - 🟣 #154897 - 🟣 #155129 - 🟣 #155140 - 🟣 #155130 ### Improvements / follow-ups - 👷🏼 Extract logic into [runV2Migration](#154151 (comment)) @gsoldevila - Make `getCurrentIndexTypesMap` resillient to intermittent failures #154151 (comment) - 🚧 Build a more structured [MigratorSynchronizer](#154151 (comment)) - 🟣 #155035 - 🟣 #155116 - 🟣 #155366 ## Reallocation tweaks Tweaks to the reallocation can be done after the initial merge, as long as it's done before the public release of 8.8 - `url` should get back to `.kibana` (see [comment](#154888 (comment))) ## Release Note For performance purposes, Kibana is now using more system indices to store its internal data. The following system indices will be created when upgrading to `8.8.0`: - `.kibana_alerting_cases` - `.kibana_analytics` - `.kibana_security_solution` - `.kibana_ingest` --------- Co-authored-by: pgayvallet <[email protected]> Co-authored-by: Christos Nasikas <[email protected]> Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Georgii Gorbachev <[email protected]>

pgayvallet added 2 commits April 18, 2023 12:09

create const for ingest SO index

b9b10d6

update index pattern for fleet types

3763a32

pgayvallet added Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc Feature:Saved Objects release_note:skip Skip the PR/issue when compiling release notes v8.8.0 labels Apr 18, 2023

pgayvallet mentioned this pull request Apr 18, 2023

Split the .kibana saved objects index into multiple indices #154888

Merged

pgayvallet added 2 commits April 18, 2023 12:25

updqte reference of .kibana

c01b154

fix bundle size

66afe2b

pgayvallet force-pushed the split-dot-kibana-ingest branch from 29c726c to 66afe2b Compare April 18, 2023 11:38

pgayvallet and others added 2 commits April 19, 2023 08:28

Merge remote-tracking branch 'upstream/dot-kibana-split' into split-d…

f7f9f7c

…ot-kibana-ingest

Update ES archiver dynamic mappings

f5cca20

pgayvallet changed the title ~~[dot-kibana-split] split ingest types into .kibana_ingest~~ [dot-kibana-split] Split ingest types into .kibana_ingest Apr 19, 2023

pgayvallet and others added 7 commits April 19, 2023 11:46

updating snapshot

fc43533

fix some FTR direct calls to .kibana

f5b5cef

Merge remote-tracking branch 'upstream/dot-kibana-split' into split-d…

9f47a90

…ot-kibana-ingest

Merge remote-tracking branch 'upstream/dot-kibana-split' into split-d…

3ed9be4

…ot-kibana-ingest

fix test

5618ce6

Fix data.json which doesn't have a mappings.json

623f9b7

Merge branch 'split-dot-kibana-ingest' of https://github.com/pgayvall…

aa90ed4

…et/kibana into split-dot-kibana-ingest

pgayvallet marked this pull request as ready for review April 19, 2023 15:55

pgayvallet requested review from a team as code owners April 19, 2023 15:55

Merge remote-tracking branch 'upstream/dot-kibana-split' into split-d…

69c5050

…ot-kibana-ingest

botelastic bot added the Team:Fleet Team label for Observability Data Collection Fleet team label Apr 19, 2023

pgayvallet merged commit bc094dc into elastic:dot-kibana-split Apr 19, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[dot-kibana-split] Split ingest types into `.kibana_ingest` #155130

[dot-kibana-split] Split ingest types into `.kibana_ingest` #155130

pgayvallet commented Apr 18, 2023 •

edited

Loading

kibana-ci commented Apr 19, 2023 •

edited

Loading

API count

elasticmachine commented Apr 19, 2023

elasticmachine commented Apr 19, 2023

[dot-kibana-split] Split ingest types into .kibana_ingest #155130

[dot-kibana-split] Split ingest types into .kibana_ingest #155130

Conversation

pgayvallet commented Apr 18, 2023 • edited Loading

This PR is targeting the dot-kibana-split feature branch and not main

Summary

kibana-ci commented Apr 19, 2023 • edited Loading

💔 Build Failed

Failed CI Steps

Test Failures

Metrics [docs]

Module Count

Public APIs missing comments

Async chunks

Page load bundle

API count

History

elasticmachine commented Apr 19, 2023

elasticmachine commented Apr 19, 2023

[dot-kibana-split] Split ingest types into `.kibana_ingest` #155130

[dot-kibana-split] Split ingest types into `.kibana_ingest` #155130

pgayvallet commented Apr 18, 2023 •

edited

Loading

This PR is targeting the `dot-kibana-split` feature branch and not `main`

kibana-ci commented Apr 19, 2023 •

edited

Loading