[Response Ops][Alerting] Migrate installation of context-specific component templates, index templates and concrete write index to framework for alerts-as-data #151792

ymao1 · 2023-02-21T20:49:35Z

Summary

In a previous PR we started installing a context-specific component templates, index templates and concrete write indices for framework alerts as data when the xpack.alerting.enableFrameworkAlerts config flag is set to true. In that PR we used a different naming pattern than what is used by the rule registry for those resources. In this PR, we are aligning the naming of these resources with the rule registry and installing these resources on alerting plugin setup when enableFrameworkAlerts: true. If the flag is set to false, the rule registry will continue to handle this resource installation.

In this PR we are doing the following:

Registering all rules currently registered with the rule registry with the alerting framework. This registration allows the alerting framework to build context specific component templates. Because this PR only addresses resource installation, rules will continue to be registered with the rule registry.
When enableFrameworkAlerts: true:
- The framework installs the context specific component template with the following naming convention: .alerts-{context}.alerts-mappings. This matches what the rule registry currently installs so the transition should be seamless
- The framework installs the context specific index template for the default space with the following name: .alerts-{context}.alerts-default-index-template. Space awareness will be addressed in a followup PR. This matches the current rule registry naming.This index template will reference
  (1) ECS component template (if useEcs: true),
  (2) context-specific component template,
  (3) legacy alert component template and
  (4) framework component template
  where the legacy alert component template + framework component template = technical component template (from the rule registry).
- The framework creates or updates the concrete write index for the default space with the naming convention: .internal.alerts-{context}.alerts-default-000001. Space awareness will be addressed in a followup PR. This matches the current rule registry naming.
- The installation of the index template & write index differs from the rule registry in that it occurs on alerting plugin start vs the first rule run.
We modified the rule registry resource installer to skip installation of these resources when enableFrameworkAlerts: true. In addition, it will wait for the alerting resource installation promise so if a rule runs before its resources are fully initialized, it will wait for initialization to complete before writing.

To Verify

The following rule registry contexts are affected:
observability.apm
observability.logs
observability.metrics
observability.slo
observability.uptime
security

For each context, we should verify the following:

Note that if your rule context references the ECS mappings, there may be differences in those mappings between main and this branch depending on whether you're running main with enableFrameworkAlerts true or false. These differences are explained in the summary of this prior PR: https://github.com/elastic/kibana/pull/150384 but essentially we're aligning with the latest ECS fields. In the instructions, I suggest running main with enableFrameworkAlerts: true to minimize the differences caused by ECS changes

While running main with enableFrameworkAlerts: true:

Get the context specific component template GET _component_template/.alerts-{context}.alerts-mappings
Create rule for this context that creates an alert and then
Get the index template GET _index_template/.alerts-{context}.alerts-default-index-template
Get the index mapping for the concrete index: GET .internal.alerts-{context}.alerts-default-000001/_mapping

While running this branch with xpack.alerting.enableFrameworkAlerts: true (with a fresh ES instance):
5. Get the context specific component template GET _component_template/.alerts-{context}.alerts-mappings
6. Get the index template GET _index_template/.alerts-{context}.alerts-default-index-template
7. Get the index mapping for the concrete index: GET .internal.alerts-{context}.alerts-default-000001/_mapping
Note that you should not have to create a rule that generates alerts before seeing these resources installed.

Compare the component templates
Compare 1 and 5. The difference should be:

component template from this branch should have _meta.managed: true. This is a flag indicating to the user that these templates are system managed and should not be manually modified.

Compare the index templates
Compare 3 and 6. The differences should be:

index template from this branch should have managed: true in the _meta fields
index template from this branch should not have a priority field. This will be addressed in a followup PR
index template from this branch should be composed of .alerts-legacy-alert-mappings and .alerts-framework-mappings instead of .alerts-technical-mappings but under the hood, these mappings are equivalent.

Compare the index mappings
Compare 4 and 7. The difference should be:

index mappings from this branch should have _meta.managed: true.

Verify that installed resources templates work as expected

Run this branch on a fresh ES install with xpack.alerting.enableFrameworkAlerts: true.
Create a rule in your context that generates alerts.
Verify that there are no errors during rule execution.
Verify that the alerts show up in your alerts table as expected.
(For detection rules only): Run this branch with xpack.alerting.enableFrameworkAlerts: true and verify rules in a non-default space continue to create resources on first rule run and run as expected.
(For detection rules only): Run this branch with xpack.alerting.enableFrameworkAlerts: true and verify rule preview continue to work as expected

Verify that installed resources templates work with existing rule registry resources.

Run main or a previous version and create a rule in your context that generates alerts.
Using the same ES data, switch to this branch with xpack.alerting.enableFrameworkAlerts: false and verify Kibana starts with no rule registry errors and the rule continues to run as expected.
Using the same ES data, switch to this branch with xpack.alerting.enableFrameworkAlerts: true and verify Kibana starts with no alerting or rule registry errors and the rule continues to run as expected.
Verify the alerts show up on the alerts table as expected.
(For detection rules only): Run this branch with xpack.alerting.enableFrameworkAlerts: true and verify rules in a non-default space continue to create resources on first rule run and run as expected.
(For detection rules only): Run this branch with xpack.alerting.enableFrameworkAlerts: true and verify rule preview continue to work as expected

ymao1 · 2023-02-27T19:37:08Z

packages/kbn-alerts-as-data-utils/src/field_maps/legacy_experimental_field_map.ts

@@ -0,0 +1,20 @@
+/*


Moved from x-pack/plugins/rule_registry/common/assets/field_maps/experimental_rule_field_map.ts

x-pack/plugins/rule_registry/server/rule_data_plugin_service/resource_installer.ts

…mework install context specific resources

x-pack/plugins/alerting/common/alert_schema/field_maps/component_template_from_field_map.ts

ymao1 · 2023-02-27T20:38:08Z

x-pack/plugins/alerting/server/alerts_service/alerts_service.ts

-      if (!isEqual(fieldMap, registeredFieldMap)) {
-        throw new Error(`${context} has already been registered with a different mapping`);
+      const registeredOptions = this.registeredContexts.get(context);
+      if (!isEqual(opts, registeredOptions)) {


Testing equality of all options to catch the possibility of rule types registering with the same context but different useEcs or useLegacyAlerts flags.

ymao1 · 2023-02-27T20:38:35Z

x-pack/plugins/alerting/server/alerts_service/alerts_service.ts

@@ -326,6 +361,14 @@ export class AlertsService implements IAlertsService {
  ) {
    this.options.logger.info(`Installing index template ${indexPatterns.template}`);

+    const indexMetadata: Metadata = {


Added this to align with metadata that rule registry is outputting

ymao1 · 2023-02-27T20:40:56Z

x-pack/plugins/alerting/server/types.ts

+  /**
+   * Optional secondary alias to use. This alias should not include the namespace.
+   */
+  secondaryAlias?: string;


Added to allow detection alerts to specify siem.signals alias

…esources-component

mikecote

Great work!! I finished reviewing the code and have the comments below from our discussion. I will move onto testing the PR 👍 🎉

x-pack/plugins/alerting/common/alert_schema/field_maps/component_template_from_field_map.ts

x-pack/plugins/alerting/server/alerts_service/alerts_service.ts

mikecote · 2023-03-01T16:27:42Z

x-pack/plugins/rule_registry/server/rule_data_plugin_service/resource_installer.ts

@@ -16,6 +16,7 @@ import {
  DEFAULT_ALERTS_ILM_POLICY,


I'm not sure if this is a concern given we can do a diff down the line but:

question: do we have something in place to prevent registrationContext from differentiating to what is provided to the rule type via alerts object? (thinking in case a new rule type is in the works somewhere or soemthing.. until we remove the need for registrationContext)

If a new rule type is added to the rule registry without also being registered with the framework and the framework handles the installation the context specific resources, any run of the rule type that generates an alert will throw an error on execution. I believe this should be caught during development but if we want an additional check, I think we could add a check to the initializeIndex function in the rule registry RuleDataService that verifies any context registered with the rule data service is also registered in the framework.

I think adding the function within the rule registry's initializeIndex or doing a manual check when switching the framework alerts feature flag will be ok. If it's trivial to do the manual check, maybe we just do that when validating the feature flag PR?

x-pack/plugins/rule_registry/server/rule_data_plugin_service/resource_installer.ts

x-pack/plugins/alerting/server/plugin.ts

x-pack/plugins/alerting/server/alerts_service/alerts_service.ts

x-pack/plugins/alerting/server/alerts_service/create_resource_installation_helper.ts

mikecote

Completed my testing this morning, everything looks great! I used observability.metrics for my testing and all the differences were as outlined in the PR. Great work!

ymao1 · 2023-03-06T13:52:58Z

@elasticmachine merge upstream

kdelemme

LGTM! Tested locally with an existing SLO burn rate rule running.

dominiqueclarke

Uptime changes LGTM

marshallmain

Detection alerts changes LGTM

crespocarlos

LGTM on behalf of Infra Observability UI

ymao1 · 2023-03-08T12:55:20Z

@elasticmachine merge upstream

kibana-ci · 2023-03-08T14:41:31Z

💚 Build Succeeded

Buildkite Build
Commit: c071f5d

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`apm`	1410	1411	+1
`observability`	550	549	-1
`securitySolution`	3751	3750	-1
`triggersActionsUi`	547	548	+1
total			-0

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/alerts-as-data-utils`	12	14	+2
`alerting`	486	487	+1
total			+3

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`apm`	3.8MB	3.8MB	+8.0B
`observability`	1.1MB	1.1MB	+158.0B
`triggersActionsUi`	1.4MB	1.4MB	+10.0B
total			+176.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`alerting`	41.8KB	41.8KB	+66.0B
`triggersActionsUi`	80.6KB	80.8KB	+130.0B
total			+196.0B

Unknown metric groups

API count

id	before	after	diff
`@kbn/alerts-as-data-utils`	12	14	+2
`alerting`	497	503	+6
total			+8

ESLint disabled line counts

id	before	after	diff
`alerting`	77	78	+1
`securitySolution`	431	433	+2
total			+3

Total ESLint disabled count

id	before	after	diff
`alerting`	78	79	+1
`securitySolution`	508	510	+2
total			+3

History

💛 Build #112048 was flaky 7f3149e
💛 Build #111883 was flaky 05db78c
💔 Build #111881 failed 87cecce
💛 Build #111687 was flaky 41522c1
💔 Build #111489 failed 1c6dc24

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @ymao1

…ponent templates, index templates and concrete write index to framework for alerts-as-data (elastic#151792) Resolves elastic#151697 ## Summary In a previous [PR](elastic#145581) we started installing a context-specific component templates, index templates and concrete write indices for framework alerts as data when the `xpack.alerting.enableFrameworkAlerts` config flag is set to true. In that PR we used a different naming pattern than what is used by the rule registry for those resources. In this PR, we are aligning the naming of these resources with the rule registry and installing these resources on alerting plugin setup when `enableFrameworkAlerts: true`. If the flag is set to false, the rule registry will continue to handle this resource installation. In this PR we are doing the following: * Registering all rules currently registered with the rule registry with the alerting framework. This registration allows the alerting framework to build context specific component templates. Because this PR only addresses resource installation, rules will continue to be registered with the rule registry. * When `enableFrameworkAlerts: true`: * The framework installs the context specific component template with the following naming convention: `.alerts-{context}.alerts-mappings`. This matches what the rule registry currently installs so the transition should be seamless * The framework installs the context specific index template for the `default` space with the following name: `.alerts-{context}.alerts-default-index-template`. Space awareness will be addressed in a followup PR. This matches the current rule registry naming.This index template will reference (1) ECS component template (if `useEcs: true`), (2) context-specific component template, (3) legacy alert component template and (4) framework component template where the legacy alert component template + framework component template = technical component template (from the rule registry). * The framework creates or updates the concrete write index for the `default` space with the naming convention: `.internal.alerts-{context}.alerts-default-000001`. Space awareness will be addressed in a followup PR. This matches the current rule registry naming. * The installation of the index template & write index differs from the rule registry in that it occurs on alerting plugin start vs the first rule run. * We modified the rule registry resource installer to skip installation of these resources when `enableFrameworkAlerts: true`. In addition, it will wait for the alerting resource installation promise so if a rule runs before its resources are fully initialized, it will wait for initialization to complete before writing. ## To Verify The following rule registry contexts are affected: `observability.apm` `observability.logs` `observability.metrics` `observability.slo` `observability.uptime` `security` For each context, we should verify the following: `Note that if your rule context references the ECS mappings, there may be differences in those mappings between main and this branch depending on whether you're running main with enableFrameworkAlerts true or false. These differences are explained in the summary of this prior PR: elastic#150384 but essentially we're aligning with the latest ECS fields. In the instructions, I suggest running main with enableFrameworkAlerts: true to minimize the differences caused by ECS changes` **While running `main` with `enableFrameworkAlerts: true`:** 1. Get the context specific component template `GET _component_template/.alerts-{context}.alerts-mappings` 2. Create rule for this context that creates an alert and then 3. Get the index template `GET _index_template/.alerts-{context}.alerts-default-index-template` 4. Get the index mapping for the concrete index: `GET .internal.alerts-{context}.alerts-default-000001/_mapping` **While running this branch with `xpack.alerting.enableFrameworkAlerts: true` (with a fresh ES instance):** 5. Get the context specific component template `GET _component_template/.alerts-{context}.alerts-mappings` 6. Get the index template `GET _index_template/.alerts-{context}.alerts-default-index-template` 7. Get the index mapping for the concrete index: `GET .internal.alerts-{context}.alerts-default-000001/_mapping` Note that you should not have to create a rule that generates alerts before seeing these resources installed. **Compare the component templates** Compare 1 and 5. The difference should be: * component template from this branch should have `_meta.managed: true`. This is a flag indicating to the user that these templates are system managed and should not be manually modified. **Compare the index templates** Compare 3 and 6. The differences should be: * index template from this branch should have `managed: true` in the `_meta` fields * index template from this branch should not have a `priority` field. This will be addressed in a followup PR * index template from this branch should be composed of `.alerts-legacy-alert-mappings` and `.alerts-framework-mappings` instead of `.alerts-technical-mappings` but under the hood, these mappings are equivalent. **Compare the index mappings** Compare 4 and 7. The difference should be: * index mappings from this branch should have `_meta.managed: true`. ### Verify that installed resources templates work as expected 1. Run this branch on a fresh ES install with `xpack.alerting.enableFrameworkAlerts: true`. 2. Create a rule in your context that generates alerts. 3. Verify that there are no errors during rule execution. 4. Verify that the alerts show up in your alerts table as expected. 5. (For detection rules only): Run this branch with `xpack.alerting.enableFrameworkAlerts: true` and verify rules in a non-default space continue to create resources on first rule run and run as expected. 6. (For detection rules only): Run this branch with `xpack.alerting.enableFrameworkAlerts: true` and verify rule preview continue to work as expected ### Verify that installed resources templates work with existing rule registry resources. 1. Run `main` or a previous version and create a rule in your context that generates alerts. 2. Using the same ES data, switch to this branch with `xpack.alerting.enableFrameworkAlerts: false` and verify Kibana starts with no rule registry errors and the rule continues to run as expected. 3. Using the same ES data, switch to this branch with `xpack.alerting.enableFrameworkAlerts: true` and verify Kibana starts with no alerting or rule registry errors and the rule continues to run as expected. 4. Verify the alerts show up on the alerts table as expected. 5. (For detection rules only): Run this branch with `xpack.alerting.enableFrameworkAlerts: true` and verify rules in a non-default space continue to create resources on first rule run and run as expected. 6. (For detection rules only): Run this branch with `xpack.alerting.enableFrameworkAlerts: true` and verify rule preview continue to work as expected --------- Co-authored-by: Kibana Machine <[email protected]>

ymao1 force-pushed the alerting/faad-resources-component branch 2 times, most recently from c24a423 to 9ea206c Compare February 24, 2023 20:03

ymao1 commented Feb 27, 2023

View reviewed changes

x-pack/plugins/rule_registry/server/rule_data_plugin_service/resource_installer.ts Show resolved Hide resolved

Registering all rule types with alerting and conditionally having fra…

cc24f4d

…mework install context specific resources

ymao1 force-pushed the alerting/faad-resources-component branch from 22a7324 to cc24f4d Compare February 27, 2023 19:51

ymao1 changed the title ~~Alerting/faad resources component~~ [Response Ops][Alerting] Installing context-specific component templates, index templates and concrete write index for framework alerts-as-data Feb 27, 2023

ymao1 commented Feb 27, 2023

View reviewed changes

x-pack/plugins/alerting/common/alert_schema/field_maps/component_template_from_field_map.ts Outdated Show resolved Hide resolved

ymao1 commented Feb 27, 2023

View reviewed changes

ymao1 added 3 commits February 27, 2023 15:50

Cleanup

47e3cbf

Fixing race condition between existing rule running and framework setup

004ada0

Merge branch 'main' of github.com:elastic/kibana into alerting/faad-r…

e04ee96

…esources-component

ymao1 self-assigned this Feb 28, 2023

ymao1 added Feature:Alerting Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) Feature:Alerting/Alerts-as-Data Issues related to Alerts-as-data and RuleRegistry v8.8.0 release_note:skip Skip the PR/issue when compiling release notes labels Feb 28, 2023

ymao1 marked this pull request as ready for review February 28, 2023 21:01

ymao1 requested review from a team as code owners February 28, 2023 21:01

ymao1 added 3 commits March 1, 2023 14:10

Rule registry should continue installing preview stuff

788c990

Merging in main

1c6dc24

Merging in main

41522c1

mikecote reviewed Mar 2, 2023

View reviewed changes

ymao1 added 6 commits March 3, 2023 08:49

Refactoring resource installation helper

9bd797d

Renaming isContextInitialized fn

662a40d

PR feedback

d3dcbab

Returning error messages with init result

f18446a

Merging in main

87cecce

Fixing types

05db78c

mikecote approved these changes Mar 6, 2023

View reviewed changes

Merge branch 'main' into alerting/faad-resources-component

7f3149e

dominiqueclarke self-requested a review March 6, 2023 20:45

kdelemme approved these changes Mar 6, 2023

View reviewed changes

dominiqueclarke approved these changes Mar 6, 2023

View reviewed changes

marshallmain approved these changes Mar 6, 2023

View reviewed changes

crespocarlos self-requested a review March 7, 2023 09:10

crespocarlos approved these changes Mar 7, 2023

View reviewed changes

andrew-goldstein approved these changes Mar 7, 2023

View reviewed changes

dgieselaar approved these changes Mar 8, 2023

View reviewed changes

Merge branch 'main' into alerting/faad-resources-component

c071f5d

ymao1 merged commit 64a7df7 into elastic:main Mar 8, 2023

kibanamachine added the backport:skip This commit does not require backporting label Mar 8, 2023

ymao1 deleted the alerting/faad-resources-component branch March 8, 2023 14:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Response Ops][Alerting] Migrate installation of context-specific component templates, index templates and concrete write index to framework for alerts-as-data #151792

[Response Ops][Alerting] Migrate installation of context-specific component templates, index templates and concrete write index to framework for alerts-as-data #151792

ymao1 commented Feb 21, 2023 •

edited

Loading

ymao1 Feb 27, 2023

ymao1 Feb 27, 2023

ymao1 Feb 27, 2023

ymao1 Feb 27, 2023

mikecote left a comment

mikecote Mar 1, 2023

ymao1 Mar 3, 2023

mikecote Mar 6, 2023

mikecote left a comment

ymao1 commented Mar 6, 2023

kdelemme left a comment

dominiqueclarke left a comment

marshallmain left a comment

crespocarlos left a comment

ymao1 commented Mar 8, 2023

kibana-ci commented Mar 8, 2023

API count

ESLint disabled line counts

Total ESLint disabled count

[Response Ops][Alerting] Migrate installation of context-specific component templates, index templates and concrete write index to framework for alerts-as-data #151792

[Response Ops][Alerting] Migrate installation of context-specific component templates, index templates and concrete write index to framework for alerts-as-data #151792

Conversation

ymao1 commented Feb 21, 2023 • edited Loading

Summary

To Verify

Verify that installed resources templates work as expected

Verify that installed resources templates work with existing rule registry resources.

ymao1 Feb 27, 2023

Choose a reason for hiding this comment

ymao1 Feb 27, 2023

Choose a reason for hiding this comment

ymao1 Feb 27, 2023

Choose a reason for hiding this comment

ymao1 Feb 27, 2023

Choose a reason for hiding this comment

mikecote left a comment

Choose a reason for hiding this comment

mikecote Mar 1, 2023

Choose a reason for hiding this comment

ymao1 Mar 3, 2023

Choose a reason for hiding this comment

mikecote Mar 6, 2023

Choose a reason for hiding this comment

mikecote left a comment

Choose a reason for hiding this comment

ymao1 commented Mar 6, 2023

kdelemme left a comment

Choose a reason for hiding this comment

dominiqueclarke left a comment

Choose a reason for hiding this comment

marshallmain left a comment

Choose a reason for hiding this comment

crespocarlos left a comment

Choose a reason for hiding this comment

ymao1 commented Mar 8, 2023

kibana-ci commented Mar 8, 2023

💚 Build Succeeded

Metrics [docs]

Module Count

Public APIs missing comments

Async chunks

Page load bundle

API count

ESLint disabled line counts

Total ESLint disabled count

History

ymao1 commented Feb 21, 2023 •

edited

Loading