[Response Ops][Alerting] Initial implementation of FAAD AlertsClient for writing generic AAD documents

[ymao1]

Resolves #156442

Summary

1. Adds shouldWriteAlerts flag to rule type registration which defaults to false if not set. This prevents duplicate AAD documents from being written for the rule registry rule types that had to register with the framework in order to get their resources installed on startup.
2. Initial implementation of AlertsClient which primarily functions as a proxy to the LegacyAlertsClient. It does 2 additional thing:
   a. When initialized with the active & recovered alerts from the previous execution (de-serialized from the task manager state), it queries the AAD index for the corresponding alert document.
   b. When returning the alerts to serialize into the task manager state, it builds the alert document and bulk upserts into the AAD index.

This PR does not opt any rule types into writing these generic docs but adds an example functional test that does. To test it out with the ES query rule type, add the following

diff --git a/x-pack/plugins/stack_alerts/server/rule_types/es_query/rule_type.ts b/x-pack/plugins/stack_alerts/server/rule_types/es_query/rule_type.ts
index 214d2ee4b76..0439a576b03 100644
--- a/x-pack/plugins/stack_alerts/server/rule_types/es_query/rule_type.ts
+++ b/x-pack/plugins/stack_alerts/server/rule_types/es_query/rule_type.ts
@@ -187,5 +187,12 @@ export function getRuleType(
     },
     producer: STACK_ALERTS_FEATURE_ID,
     doesSetRecoveryContext: true,
+    alerts: {
+      context: 'stack',
+      shouldWrite: true,
+      mappings: {
+        fieldMap: {},
+      },
+    },
   };
 }

To Verify

• Verify that rule registry rule types still work as expected
• Verify that non rule-registry rule types still work as expected
• Modify a rule type to register with FAAD and write alerts and verify that the alert documents look as expected.

[ymao1]

When we call processAlerts here, it is just to get the map of recovered alert IDs to return to the executors so we don't need to know the maintenance window IDs.

[ymao1]

Added a new generic for rule types to specify their rule type specific alert schema. This probably isn't strictly needed until we implement the followup PR for #156443

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[ymao1]

@elasticmachine merge upstream

[mikecote]

Code LGTM and ran some tests locally and everything worked great 🎉 I left a bunch of questions. The main question I have is about the bulk call to update alerts and OCC.

[mikecote]

question: should we also update action_group field to recovered when building a recovered alert?

[ymao1]

Do you have an opinion one way or the other? I was undecided on whether to update the action_group here since we would be losing the history of which action group the alert was active in, but I guess in the case of a rule type with multiple action groups, if an alert remained active but switched between groups, we would lose that history then anyway?

[mikecote]

@ymao1 and I discussed offline and we'll update the action_group on recovery. Which will solve the concern from #156946 (comment).

[ymao1]

Updated in 1f376bd

[mikecote]

question: What happens if the bulk call happens at the same time a user updates the same alert document? Will this overwrite the user's action and/or vice versa?

[ymao1]

That's a good question! I think that's something that could happen currently with the existing flow right? I think we could create a followup issue to discuss the correct behavior (like which update gets priority) unless you feel like it's something necessary to address in this PR?

[mikecote]

I think follow up issue is good for this one, we can take the time to think it through, especially if it's an edge case today and we haven't encountered it yet (I can see this being unlikely).

[ymao1]

Issue created here: #158403

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: ad9304f

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
alerting	594	595	+1

Unknown metric groups

API count

id	before	after	diff
alerting	615	617	+2

ESLint disabled line counts

id	before	after	diff
enterpriseSearch	19	21	+2
securitySolution	399	403	+4
total			+6

Total ESLint disabled count

id	before	after	diff
enterpriseSearch	20	22	+2
securitySolution	479	483	+4
total			+6

History

• 💚 Build #129667 succeeded cfa1339
• 💔 Build #129545 failed bae41de
• 💛 Build #128980 was flaky d14d39a
• 💛 Build #128868 was flaky 89b25ac
• 💔 Build #128812 failed 58f6254
• 💔 Build #128805 failed f65aa1cb960171c84af65c31e19824eeb1fd8b3a

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @ymao1