[Response Ops][Alerting] Migrate installation of context-specific component templates, index templates & indices to alerting framework #151697
Assignees
Labels
Feature:Alerting/Alerts-as-Data
Issues related to Alerts-as-data and RuleRegistry
Feature:Alerting
Team:ResponseOps
Label for the ResponseOps team (formerly the Cases and Alerting teams)
Comments
ymao1
added
Feature:Alerting
Team:ResponseOps
|
Pinging @elastic/response-ops (Team:ResponseOps) |
github-project-automation
bot
moved this to Awaiting Triage
in AppEx: ResponseOps - Execution & Connectors
ymao1
moved this from Awaiting Triage
to Todo
in AppEx: ResponseOps - Execution & Connectors
ymao1
moved this from Todo
to In Progress
in AppEx: ResponseOps - Execution & Connectors
ymao1
moved this from In Progress
to In Review
in AppEx: ResponseOps - Execution & Connectors
ymao1
changed the title
ymao1
added a commit
that referenced
this issue
Mar 8, 2023
…ponent templates, index templates and concrete write index to framework for alerts-as-data (#151792) Resolves #151697 ## Summary In a previous [PR](#145581) we started installing a context-specific component templates, index templates and concrete write indices for framework alerts as data when the `xpack.alerting.enableFrameworkAlerts` config flag is set to true. In that PR we used a different naming pattern than what is used by the rule registry for those resources. In this PR, we are aligning the naming of these resources with the rule registry and installing these resources on alerting plugin setup when `enableFrameworkAlerts: true`. If the flag is set to false, the rule registry will continue to handle this resource installation. In this PR we are doing the following: * Registering all rules currently registered with the rule registry with the alerting framework. This registration allows the alerting framework to build context specific component templates. Because this PR only addresses resource installation, rules will continue to be registered with the rule registry. * When `enableFrameworkAlerts: true`: * The framework installs the context specific component template with the following naming convention: `.alerts-{context}.alerts-mappings`. This matches what the rule registry currently installs so the transition should be seamless * The framework installs the context specific index template for the `default` space with the following name: `.alerts-{context}.alerts-default-index-template`. Space awareness will be addressed in a followup PR. This matches the current rule registry naming.This index template will reference (1) ECS component template (if `useEcs: true`), (2) context-specific component template, (3) legacy alert component template and (4) framework component template where the legacy alert component template + framework component template = technical component template (from the rule registry). * The framework creates or updates the concrete write index for the `default` space with the naming convention: `.internal.alerts-{context}.alerts-default-000001`. Space awareness will be addressed in a followup PR. This matches the current rule registry naming. * The installation of the index template & write index differs from the rule registry in that it occurs on alerting plugin start vs the first rule run. * We modified the rule registry resource installer to skip installation of these resources when `enableFrameworkAlerts: true`. In addition, it will wait for the alerting resource installation promise so if a rule runs before its resources are fully initialized, it will wait for initialization to complete before writing. ## To Verify The following rule registry contexts are affected: `observability.apm` `observability.logs` `observability.metrics` `observability.slo` `observability.uptime` `security` For each context, we should verify the following: `Note that if your rule context references the ECS mappings, there may be differences in those mappings between main and this branch depending on whether you're running main with enableFrameworkAlerts true or false. These differences are explained in the summary of this prior PR: #150384 but essentially we're aligning with the latest ECS fields. In the instructions, I suggest running main with enableFrameworkAlerts: true to minimize the differences caused by ECS changes` **While running `main` with `enableFrameworkAlerts: true`:** 1. Get the context specific component template `GET _component_template/.alerts-{context}.alerts-mappings` 2. Create rule for this context that creates an alert and then 3. Get the index template `GET _index_template/.alerts-{context}.alerts-default-index-template` 4. Get the index mapping for the concrete index: `GET .internal.alerts-{context}.alerts-default-000001/_mapping` **While running this branch with `xpack.alerting.enableFrameworkAlerts: true` (with a fresh ES instance):** 5. Get the context specific component template `GET _component_template/.alerts-{context}.alerts-mappings` 6. Get the index template `GET _index_template/.alerts-{context}.alerts-default-index-template` 7. Get the index mapping for the concrete index: `GET .internal.alerts-{context}.alerts-default-000001/_mapping` Note that you should not have to create a rule that generates alerts before seeing these resources installed. **Compare the component templates** Compare 1 and 5. The difference should be: * component template from this branch should have `_meta.managed: true`. This is a flag indicating to the user that these templates are system managed and should not be manually modified. **Compare the index templates** Compare 3 and 6. The differences should be: * index template from this branch should have `managed: true` in the `_meta` fields * index template from this branch should not have a `priority` field. This will be addressed in a followup PR * index template from this branch should be composed of `.alerts-legacy-alert-mappings` and `.alerts-framework-mappings` instead of `.alerts-technical-mappings` but under the hood, these mappings are equivalent. **Compare the index mappings** Compare 4 and 7. The difference should be: * index mappings from this branch should have `_meta.managed: true`. ### Verify that installed resources templates work as expected 1. Run this branch on a fresh ES install with `xpack.alerting.enableFrameworkAlerts: true`. 2. Create a rule in your context that generates alerts. 3. Verify that there are no errors during rule execution. 4. Verify that the alerts show up in your alerts table as expected. 5. (For detection rules only): Run this branch with `xpack.alerting.enableFrameworkAlerts: true` and verify rules in a non-default space continue to create resources on first rule run and run as expected. 6. (For detection rules only): Run this branch with `xpack.alerting.enableFrameworkAlerts: true` and verify rule preview continue to work as expected ### Verify that installed resources templates work with existing rule registry resources. 1. Run `main` or a previous version and create a rule in your context that generates alerts. 2. Using the same ES data, switch to this branch with `xpack.alerting.enableFrameworkAlerts: false` and verify Kibana starts with no rule registry errors and the rule continues to run as expected. 3. Using the same ES data, switch to this branch with `xpack.alerting.enableFrameworkAlerts: true` and verify Kibana starts with no alerting or rule registry errors and the rule continues to run as expected. 4. Verify the alerts show up on the alerts table as expected. 5. (For detection rules only): Run this branch with `xpack.alerting.enableFrameworkAlerts: true` and verify rules in a non-default space continue to create resources on first rule run and run as expected. 6. (For detection rules only): Run this branch with `xpack.alerting.enableFrameworkAlerts: true` and verify rule preview continue to work as expected --------- Co-authored-by: Kibana Machine <[email protected]>
github-project-automation
bot
moved this from In Review
to Done
in AppEx: ResponseOps - Execution & Connectors
bmorelli25
pushed a commit
to bmorelli25/kibana
that referenced
this issue
Mar 10, 2023
…ponent templates, index templates and concrete write index to framework for alerts-as-data (elastic#151792) Resolves elastic#151697 ## Summary In a previous [PR](elastic#145581) we started installing a context-specific component templates, index templates and concrete write indices for framework alerts as data when the `xpack.alerting.enableFrameworkAlerts` config flag is set to true. In that PR we used a different naming pattern than what is used by the rule registry for those resources. In this PR, we are aligning the naming of these resources with the rule registry and installing these resources on alerting plugin setup when `enableFrameworkAlerts: true`. If the flag is set to false, the rule registry will continue to handle this resource installation. In this PR we are doing the following: * Registering all rules currently registered with the rule registry with the alerting framework. This registration allows the alerting framework to build context specific component templates. Because this PR only addresses resource installation, rules will continue to be registered with the rule registry. * When `enableFrameworkAlerts: true`: * The framework installs the context specific component template with the following naming convention: `.alerts-{context}.alerts-mappings`. This matches what the rule registry currently installs so the transition should be seamless * The framework installs the context specific index template for the `default` space with the following name: `.alerts-{context}.alerts-default-index-template`. Space awareness will be addressed in a followup PR. This matches the current rule registry naming.This index template will reference (1) ECS component template (if `useEcs: true`), (2) context-specific component template, (3) legacy alert component template and (4) framework component template where the legacy alert component template + framework component template = technical component template (from the rule registry). * The framework creates or updates the concrete write index for the `default` space with the naming convention: `.internal.alerts-{context}.alerts-default-000001`. Space awareness will be addressed in a followup PR. This matches the current rule registry naming. * The installation of the index template & write index differs from the rule registry in that it occurs on alerting plugin start vs the first rule run. * We modified the rule registry resource installer to skip installation of these resources when `enableFrameworkAlerts: true`. In addition, it will wait for the alerting resource installation promise so if a rule runs before its resources are fully initialized, it will wait for initialization to complete before writing. ## To Verify The following rule registry contexts are affected: `observability.apm` `observability.logs` `observability.metrics` `observability.slo` `observability.uptime` `security` For each context, we should verify the following: `Note that if your rule context references the ECS mappings, there may be differences in those mappings between main and this branch depending on whether you're running main with enableFrameworkAlerts true or false. These differences are explained in the summary of this prior PR: elastic#150384 but essentially we're aligning with the latest ECS fields. In the instructions, I suggest running main with enableFrameworkAlerts: true to minimize the differences caused by ECS changes` **While running `main` with `enableFrameworkAlerts: true`:** 1. Get the context specific component template `GET _component_template/.alerts-{context}.alerts-mappings` 2. Create rule for this context that creates an alert and then 3. Get the index template `GET _index_template/.alerts-{context}.alerts-default-index-template` 4. Get the index mapping for the concrete index: `GET .internal.alerts-{context}.alerts-default-000001/_mapping` **While running this branch with `xpack.alerting.enableFrameworkAlerts: true` (with a fresh ES instance):** 5. Get the context specific component template `GET _component_template/.alerts-{context}.alerts-mappings` 6. Get the index template `GET _index_template/.alerts-{context}.alerts-default-index-template` 7. Get the index mapping for the concrete index: `GET .internal.alerts-{context}.alerts-default-000001/_mapping` Note that you should not have to create a rule that generates alerts before seeing these resources installed. **Compare the component templates** Compare 1 and 5. The difference should be: * component template from this branch should have `_meta.managed: true`. This is a flag indicating to the user that these templates are system managed and should not be manually modified. **Compare the index templates** Compare 3 and 6. The differences should be: * index template from this branch should have `managed: true` in the `_meta` fields * index template from this branch should not have a `priority` field. This will be addressed in a followup PR * index template from this branch should be composed of `.alerts-legacy-alert-mappings` and `.alerts-framework-mappings` instead of `.alerts-technical-mappings` but under the hood, these mappings are equivalent. **Compare the index mappings** Compare 4 and 7. The difference should be: * index mappings from this branch should have `_meta.managed: true`. ### Verify that installed resources templates work as expected 1. Run this branch on a fresh ES install with `xpack.alerting.enableFrameworkAlerts: true`. 2. Create a rule in your context that generates alerts. 3. Verify that there are no errors during rule execution. 4. Verify that the alerts show up in your alerts table as expected. 5. (For detection rules only): Run this branch with `xpack.alerting.enableFrameworkAlerts: true` and verify rules in a non-default space continue to create resources on first rule run and run as expected. 6. (For detection rules only): Run this branch with `xpack.alerting.enableFrameworkAlerts: true` and verify rule preview continue to work as expected ### Verify that installed resources templates work with existing rule registry resources. 1. Run `main` or a previous version and create a rule in your context that generates alerts. 2. Using the same ES data, switch to this branch with `xpack.alerting.enableFrameworkAlerts: false` and verify Kibana starts with no rule registry errors and the rule continues to run as expected. 3. Using the same ES data, switch to this branch with `xpack.alerting.enableFrameworkAlerts: true` and verify Kibana starts with no alerting or rule registry errors and the rule continues to run as expected. 4. Verify the alerts show up on the alerts table as expected. 5. (For detection rules only): Run this branch with `xpack.alerting.enableFrameworkAlerts: true` and verify rules in a non-default space continue to create resources on first rule run and run as expected. 6. (For detection rules only): Run this branch with `xpack.alerting.enableFrameworkAlerts: true` and verify rule preview continue to work as expected --------- Co-authored-by: Kibana Machine <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
In #145581 we started installing FAAD resources (behind a feature flag). We also chose a slightly different naming convention than what is used by the rule registry but have since decided that we should use the same convention.
This issue is for:
alerts-{context}-component-templateto.alerts-{context}.alerts-mappingsenableFrameworkAlerts: trueETA: While working on this issue, it was easier to roll in the installation of index templates & concrete write indices in addition to the component templates, so that is also included in the scope of this.
The text was updated successfully, but these errors were encountered: