[Response Ops][Alerting] Migrate installation of context-specific com…

…ponent templates, index templates and concrete write index to framework for alerts-as-data (elastic#151792)

Resolves elastic#151697

## Summary

In a previous [PR](elastic#145581) we
started installing a context-specific component templates, index
templates and concrete write indices for framework alerts as data when
the `xpack.alerting.enableFrameworkAlerts` config flag is set to true.
In that PR we used a different naming pattern than what is used by the
rule registry for those resources. In this PR, we are aligning the
naming of these resources with the rule registry and installing these
resources on alerting plugin setup when `enableFrameworkAlerts: true`.
If the flag is set to false, the rule registry will continue to handle
this resource installation.

In this PR we are doing the following:
* Registering all rules currently registered with the rule registry with
the alerting framework. This registration allows the alerting framework
to build context specific component templates. Because this PR only
addresses resource installation, rules will continue to be registered
with the rule registry.
* When `enableFrameworkAlerts: true`:
* The framework installs the context specific component template with
the following naming convention: `.alerts-{context}.alerts-mappings`.
This matches what the rule registry currently installs so the transition
should be seamless
* The framework installs the context specific index template for the
`default` space with the following name:
`.alerts-{context}.alerts-default-index-template`. Space awareness will
be addressed in a followup PR. This matches the current rule registry
naming.This index template will reference
    (1) ECS component template (if `useEcs: true`), 
    (2) context-specific component template, 
    (3) legacy alert component template and 
    (4) framework component template
where the legacy alert component template + framework component template
= technical component template (from the rule registry).
* The framework creates or updates the concrete write index for the
`default` space with the naming convention:
`.internal.alerts-{context}.alerts-default-000001`. Space awareness will
be addressed in a followup PR. This matches the current rule registry
naming.
* The installation of the index template & write index differs from the
rule registry in that it occurs on alerting plugin start vs the first
rule run.
* We modified the rule registry resource installer to skip installation
of these resources when `enableFrameworkAlerts: true`. In addition, it
will wait for the alerting resource installation promise so if a rule
runs before its resources are fully initialized, it will wait for
initialization to complete before writing.

## To Verify

The following rule registry contexts are affected:
`observability.apm`
`observability.logs`
`observability.metrics`
`observability.slo`
`observability.uptime`
`security`

For each context, we should verify the following:

`Note that if your rule context references the ECS mappings, there may
be differences in those mappings between main and this branch depending
on whether you're running main with enableFrameworkAlerts true or false.
These differences are explained in the summary of this prior PR:
elastic#150384 but essentially we're
aligning with the latest ECS fields. In the instructions, I suggest
running main with enableFrameworkAlerts: true to minimize the
differences caused by ECS changes`

**While running `main` with `enableFrameworkAlerts: true`:**
1. Get the context specific component template `GET
_component_template/.alerts-{context}.alerts-mappings`
2. Create rule for this context that creates an alert and then
3. Get the index template `GET
_index_template/.alerts-{context}.alerts-default-index-template`
4. Get the index mapping for the concrete index: `GET
.internal.alerts-{context}.alerts-default-000001/_mapping`

**While running this branch with `xpack.alerting.enableFrameworkAlerts:
true` (with a fresh ES instance):**
5. Get the context specific component template `GET
_component_template/.alerts-{context}.alerts-mappings`
6. Get the index template `GET
_index_template/.alerts-{context}.alerts-default-index-template`
7. Get the index mapping for the concrete index: `GET
.internal.alerts-{context}.alerts-default-000001/_mapping`
Note that you should not have to create a rule that generates alerts
before seeing these resources installed.

**Compare the component templates**
Compare 1 and 5. The difference should be:
* component template from this branch should have `_meta.managed: true`.
This is a flag indicating to the user that these templates are system
managed and should not be manually modified.

**Compare the index templates**
Compare 3 and 6. The differences should be:
* index template from this branch should have `managed: true` in the
`_meta` fields
* index template from this branch should not have a `priority` field.
This will be addressed in a followup PR
* index template from this branch should be composed of
`.alerts-legacy-alert-mappings` and `.alerts-framework-mappings` instead
of `.alerts-technical-mappings` but under the hood, these mappings are
equivalent.

**Compare the index mappings**
Compare 4 and 7. The difference should be:
* index mappings from this branch should have `_meta.managed: true`.

### Verify that installed resources templates work as expected
1. Run this branch on a fresh ES install with
`xpack.alerting.enableFrameworkAlerts: true`.
2. Create a rule in your context that generates alerts.
3. Verify that there are no errors during rule execution.
4. Verify that the alerts show up in your alerts table as expected.
5. (For detection rules only): Run this branch with
`xpack.alerting.enableFrameworkAlerts: true` and verify rules in a
non-default space continue to create resources on first rule run and run
as expected.
6. (For detection rules only): Run this branch with
`xpack.alerting.enableFrameworkAlerts: true` and verify rule preview
continue to work as expected

### Verify that installed resources templates work with existing rule
registry resources.

1. Run `main` or a previous version and create a rule in your context
that generates alerts.
2. Using the same ES data, switch to this branch with
`xpack.alerting.enableFrameworkAlerts: false` and verify Kibana starts
with no rule registry errors and the rule continues to run as expected.
3. Using the same ES data, switch to this branch with
`xpack.alerting.enableFrameworkAlerts: true` and verify Kibana starts
with no alerting or rule registry errors and the rule continues to run
as expected.
4. Verify the alerts show up on the alerts table as expected.
5. (For detection rules only): Run this branch with
`xpack.alerting.enableFrameworkAlerts: true` and verify rules in a
non-default space continue to create resources on first rule run and run
as expected.
6. (For detection rules only): Run this branch with
`xpack.alerting.enableFrameworkAlerts: true` and verify rule preview
continue to work as expected

---------

Co-authored-by: Kibana Machine <[email protected]>

packages/kbn-alerts-as-data-utils/src/field_maps/index.ts

@@ -9,4 +9,5 @@
 export * from './alert_field_map';
 export * from './ecs_field_map';
 export * from './legacy_alert_field_map';
+export * from './legacy_experimental_field_map';
 export type { FieldMap, MultiField } from './types';

packages/kbn-alerts-as-data-utils/src/field_maps/legacy_experimental_field_map.ts

@@ -0,0 +1,20 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+import { ALERT_EVALUATION_THRESHOLD, ALERT_EVALUATION_VALUE } from '@kbn/rule-data-utils';
+
+export const legacyExperimentalFieldMap = {
+  [ALERT_EVALUATION_THRESHOLD]: {
+    type: 'scaled_float',
+    scaling_factor: 100,
+    required: false,
+  },
+  [ALERT_EVALUATION_VALUE]: { type: 'scaled_float', scaling_factor: 100, required: false },
+} as const;
+
+export type ExperimentalRuleFieldMap = typeof legacyExperimentalFieldMap;

x-pack/plugins/alerting/common/alert_schema/field_maps/component_template_from_field_map.test.ts

@@ -0,0 +1,79 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+import { getComponentTemplateFromFieldMap } from './component_template_from_field_map';
+import { testFieldMap, expectedTestMapping } from './mapping_from_field_map.test';
+
+describe('getComponentTemplateFromFieldMap', () => {
+  it('correctly creates component template from field map', () => {
+    expect(
+      getComponentTemplateFromFieldMap({ name: 'test-mappings', fieldMap: testFieldMap })
+    ).toEqual({
+      name: 'test-mappings',
+      _meta: {
+        managed: true,
+      },
+      template: {
+        settings: {},
+        mappings: {
+          dynamic: 'strict',
+          ...expectedTestMapping,
+        },
+      },
+    });
+  });
+
+  it('correctly creates component template with settings when includeSettings = true', () => {
+    expect(
+      getComponentTemplateFromFieldMap({
+        name: 'test-mappings',
+        fieldMap: testFieldMap,
+        includeSettings: true,
+      })
+    ).toEqual({
+      name: 'test-mappings',
+      _meta: {
+        managed: true,
+      },
+      template: {
+        settings: {
+          number_of_shards: 1,
+          'index.mapping.total_fields.limit': 1500,
+        },
+        mappings: {
+          dynamic: 'strict',
+          ...expectedTestMapping,
+        },
+      },
+    });
+  });
+
+  it('correctly creates component template with dynamic setting when defined', () => {
+    expect(
+      getComponentTemplateFromFieldMap({
+        name: 'test-mappings',
+        fieldMap: testFieldMap,
+        includeSettings: true,
+        dynamic: false,
+      })
+    ).toEqual({
+      name: 'test-mappings',
+      _meta: {
+        managed: true,
+      },
+      template: {
+        settings: {
+          number_of_shards: 1,
+          'index.mapping.total_fields.limit': 1500,
+        },
+        mappings: {
+          dynamic: false,
+          ...expectedTestMapping,
+        },
+      },
+    });
+  });
+});

x-pack/plugins/alerting/common/alert_schema/field_maps/component_template_from_field_map.ts

@@ -11,13 +11,15 @@ import { mappingFromFieldMap } from './mapping_from_field_map';
 
 export interface GetComponentTemplateFromFieldMapOpts {
   name: string;
-  fieldLimit?: number;
   fieldMap: FieldMap;
+  includeSettings?: boolean;
+  dynamic?: 'strict' | false;
 }
 export const getComponentTemplateFromFieldMap = ({
   name,
   fieldMap,
-  fieldLimit,
+  dynamic,
+  includeSettings,
 }: GetComponentTemplateFromFieldMapOpts): ClusterPutComponentTemplateRequest => {
   return {
     name,
@@ -26,10 +28,16 @@ export const getComponentTemplateFromFieldMap = ({
     },
     template: {
       settings: {
-        number_of_shards: 1,
-        'index.mapping.total_fields.limit': fieldLimit ?? 1000,
+        ...(includeSettings
+          ? {
+              number_of_shards: 1,
+              'index.mapping.total_fields.limit':
+                Math.ceil(Object.keys(fieldMap).length / 1000) * 1000 + 500,
+            }
+          : {}),
       },
-      mappings: mappingFromFieldMap(fieldMap, 'strict'),
+
+      mappings: mappingFromFieldMap(fieldMap, dynamic ?? 'strict'),
     },
   };
 };