[RAC][Rule Registry] Rule registry executors skip writing AAD when execution is cancelled

[ymao1]

As part of this investigation, we have identified improvements in the way the alerting framework can respond to being cancelled. With this issue, we are planning to (configurably) skip scheduling actions for rules that have been cancelled due to task timeout. With this issue, we are planning to pass a helper function executionIsCancelled() that allows executors to check whether or not they should continue execution.

In order to be consistent with these framework changes, the rule registry executors should be checking whether or not rule execution is cancelled before writing AAD, otherwise this will lead to inconsistencies. For example, if the rule registry continues to write out AAD documents when the rule execution status is in error, with reason timeout and no notifications are sent.

[ymao1]

It looks like there are currently two rule registry executors types: lifecycle_rule_type and persistence_rule_type.

Inside the lifecycle_rule_type, we could add a check to this existing check:

kibana/x-pack/plugins/rule_registry/server/utils/create_lifecycle_executor.ts

Lines 284 to 295 in bbb2e96

	if (allEventsToIndex.length > 0 && ruleDataClient.isWriteEnabled()) {
	logger.debug(`Preparing to index ${allEventsToIndex.length} alerts.`);
	await ruleDataClient.getWriter().bulk({
	body: allEventsToIndex.flatMap(({ event, indexName }) => [
	indexName
	? { index: { _id: event[ALERT_UUID]!, _index: indexName, require_alias: false } }
	: { index: { _id: event[ALERT_UUID]! } },
	event,
	]),
	});
	}

to ensure !executionIsCancelled() AND rule type is not skipping actions

Inside the persistence_rule_type, we could add a check to this existing check:

kibana/x-pack/plugins/rule_registry/server/utils/create_persistence_rule_type_factory.ts

Lines 26 to 42 in bbb2e96

	if (ruleDataClient.isWriteEnabled() && numAlerts) {
	const commonRuleFields = getCommonAlertFields(options);
	const response = await ruleDataClient.getWriter().bulk({
	body: alerts.flatMap((alert) => [
	{ index: {} },
	{
	[ALERT_INSTANCE_ID]: alert.id,
	[VERSION]: ruleDataClient.kibanaVersion,
	...commonRuleFields,
	...alert.fields,
	},
	]),
	refresh,
	});
	return response;
	} else {

[ymao1]

Blocked by #113459 and #113462

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[gmmorris]

Blocked on #113804