[alerting] stop deleting / re-creating tasks when rules are disabled / re-enabled

[pmuellr]

Today, when a rule is disabled, we delete the associated task manager document; and then re-create it when the rule is re-enabled.

This has a couple of problems:

• race conditions which can cause > 1 task manager documents to exist for a single rule; see [alerting][discuss] decrease the number of scenarios where we regenerate API keys #106876 (comment)
• we lose all the existing instance state, which then causes problems with recovered alerts; see Active alerts do not recover after re-enabling a rule #110080

I think one thing we could do is add a new capability to task manager to "disable/enable" a task. From a brief look a while back, this seems pretty easy. The claim call would add a new filter to ignore task docs that have enabled: false on them. And we'd add that new field to the task SO, as well as some way to query/set it. Then when we disable a rule, we set the enabled:false flag on the task, and when we enable the rule, we set it to enabled:true.

One downside of this is keeping the stored state of the rule in the TM index, whether the rule is disabled or not. Today, you can save disk space by disabling rules; if we do something like what's suggested above, we wouldn't.

There are some older issues that were raised about "losing state when rules are disabled" that we should look into to see if there are other downsides with maintain this state longer than we do today.

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[mikecote]

We should check with solution teams if this would become breaking behaviour.

[pmuellr]

I believe the problem "race conditions can cause > 1 task manager documents to exist for a single rule" has been resolved in this PR: #117397 (shipped in 8.1.0). Now that tasks use the same id as the rule, it's not possible for a rule to have multiple tasks associated with it, for newly created rules.

[ymao1]

It is straightforward to add an enabled field to the task document and update the claim query to only claim enabled tasks. This seems like a positive addition to the task manager scheduling API.

For alerting rules specifically, if update enable/disable to just update this flag on the task document instead of deleting and recreating the task document, we will see this change in behavior:

Previously, if a rule had active alerts A,B,C and gets disabled, the task document containing state about these alerts is deleted.

When the rule gets re-enabled, if it generates alerts A and B, actions would be scheduled because A and B are considered new alerts, so regardless of the notification config (onActiveAlert, onActionGroupChange or onThrottleInterval) a notification would be sent out for A and B.

By not deleting the task document, when the rule gets re-enabled, if it generates alerts A and B, a notification will only be sent out if the notification config is set to onActiveAlert or if the throttle interval has passed for onThrottleInterval. Notifications will not be issued if the config is onActionGroupChange because the task state prior to disable is now accessible so the framework would not consider these alerts new or changed.

@mikecote @kobelb Would you consider this a breaking change?

ETA or do we consider the prior behavior to be a bug and this would fix the bug?

[kobelb]

@mikecote @kobelb Would you consider this a breaking change?

I would treat this change in behavior as a bug-fix, and not a breaking change.

[mikecote]

@mikecote @kobelb Would you consider this a breaking change?

I would treat this change in behavior as a bug-fix, and not a breaking change.

+1 from me