[ResponseOps] muted alert still fires when using notifyWhen: onActionGroupChange

[pmuellr]

We had a user report that an alert that they had muted was still firing it's actions. Note - the rule itself is not muted, just one specific alert instance. Looking at the event log, you can see that actions are being invoked for that alert, despite it be listed in the muted instances of the saved object. The rule is set to notifyWhen: onActionGroupChange. It seems like the easiest way to test this, given it's on an action group change, is to disable the rule, which will set the instances to "recovered", and the re-enable it, so that the next execution will fire all the instances (muted and not) because of the change from recovered -> active.

In the code below, you can see that the "false" path (second half) checks for throttled and muted cases, but the "true" path does not. So basically, throttling and muting are ignored when notifyWhen == onActionGroupChange.

kibana/x-pack/plugins/alerting/server/task_runner/task_runner.ts

Lines 461 to 488 in 73db13f

	const alertsToExecute =
	notifyWhen === 'onActionGroupChange'
	? Object.entries(alertsWithScheduledActions).filter(
	([alertName, alert]: [string, AlertInstance<InstanceState, InstanceContext>]) => {
	const shouldExecuteAction = alert.scheduledActionGroupOrSubgroupHasChanged();
	if (!shouldExecuteAction) {
	this.logger.debug(
	`skipping scheduling of actions for '${alertName}' in rule ${ruleLabel}: alert is active but action group has not changed`
	);
	}
	return shouldExecuteAction;
	}
	)
	: Object.entries(alertsWithScheduledActions).filter(
	([alertName, alert]: [string, AlertInstance<InstanceState, InstanceContext>]) => {
	const throttled = alert.isThrottled(throttle);
	const muted = mutedAlertIdsSet.has(alertName);
	const shouldExecuteAction = !throttled && !muted;
	if (!shouldExecuteAction) {
	this.logger.debug(
	`skipping scheduling of actions for '${alertName}' in rule ${ruleLabel}: rule is ${
	muted ? 'muted' : 'throttled'
	}`
	);
	}
	return shouldExecuteAction;
	}
	);

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[pmuellr]

I'm a little surprised by this one, since I believe notifyWhen: onActionGroupChange is the default, so you'd think this bug would show up more often. But thinking maybe it only really started happening when we started "recovering" the instances when the rule is disabled - from this PR #111671 merged in 7.16.0?