[Response Ops][Alerting] Discuss how to handle possible conflicts when updating alerts-as-data documents

[ymao1]

From #156946 (comment):

What happens if the bulk call happens at the same time a user updates the same alert document? Will this overwrite the user's action and/or vice versa?

For alerts as data, alert documents are created or updated each rule execution. Users can also perform certain updates on these documents (attaching cases, closing an alert, etc). It is possible that these two updates could occur at the same time and override each other.

As an example:

1. Rule execution begins. Alert "A" is queried from the alert index and has status active
2. User updates alert "A" to status closed
3. Rule reports alert "A" as active again
4. Rule execution ends. Alert "A" is updated by the executor using the data from (1) and status is reset to active.

A few options:

• Since we only allow certain fields to be updated by the user, we ensure that we're only performing partial updates during rule executions and we don't include those user-updatable fields in the update doc
• We add OCC to the bulk index call to retry on version conflicts
• Others?

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[pmuellr]

I actually just ran into this issue as part of working on converting the alerts indices to data streams (PR #160572). In that PR, updates to existing documents use OCC, because they have to when you do updates in data streams.

The scenario for me was a metric threshold rule running, with an active alert, every 3 seconds. I then added that alert to a case. And hit a 409 conflict on the next rule run. Somehow, the case update raced in during the rule run, so the OCC info in the rule run was stale.

I'd have to look a bit more to see if we could delay reading the alert doc OCC info to later, to hopefully avoid a race condition, or similar re-arrangement scenarios.

But feels to me we need to come up with a safer way to do this. I think Ying's suggestions above will work:

• every update to an alert doc is a partial update
• every update is done with OCC, and retried some fixed set of times (like rule docs)

We still run the risk of race conditions, but only for fields updated by multiple independent update calls. If every update caller is updating fields no other caller is updating, we shouldn't see any overwritten data. Right?

[pmuellr]

Now that we're finishing up with #160572, I want to start thinking about this issue a bit more. A couple of things:

• I think the chance of conflicts is going to be worse now, as I believe the only code doing overwriting of alerts is the alerting rule runner. It fetches the alerts at the beginning of a run, along with seq_no and primary_term, and then uses those to update existing alert docs when the rule completes. This means the window of opportunity for a conflict is the length of a rule run.

• Any other update to an alert doc during a rule run - closing, adding a case, etc - will cause the rule run update to the alert doc to fail. Previously, this race condition would cause the update (closing, adding a case, etc), to be lost.

• I think the only way around this situation is that all writes beyond create, to alert documents, should use partial updates. It appears simple updates like closing an alert already use partial updates, so it's the updating of alert docs by the rule runner that will need to be converted to a partial update. The danger is that these updates may no longer "delete" fields that aren't passed in on the update, instead leaving the previous value.

[pmuellr]

I've got an experiment open in PR #165275 where I changed all the alert update docs from occ-based index to partial update. Seeing what that breaks.