Auto case creation when alerts are detected

[shanisagiv1]

Describe the feature:
Allow users to create cases automatically based on alerting rules when alerts are detected.
This feature should help users to reduce manual work that happens today when users create cases and assign alerts manually.

Business Values:

• Increase case adoption by reducing manual work - "one-stop shop for alert management"
• Group alerts under the same context
• By improving the case experience we'll make it more attractive for users

Describe a specific use case for the feature:

• The user will be able to define case creation logic for each alerting rule with the following inputs: (we should consider the naming)
  • grouping fields - this field will group alerts with the same value for the selected field/fields
  • time window - this field will assign alerts to the same case if the defined time wasn't elapsed (for example, all the alerts within 5 days will be assigned to the same case, and on the 6th day a new case will be created)
  • reopen when close - when it's marked, this field will reopen the case if it was closed (for any reason) and a new alert with the same grouping field within the defined time window is detected.

integrate with existing grouping/suppressing capabilities:

• Currently, there are pre-processing functionalities for "alert grouping" mechanism in some o11y rule types (e.g: Metric threshold) and a "suppressed by" mechanism in security rule types (e.g: Custom query).
  So whatever is defined by the pre-processing, the alerts that are generated will be assigned to the case based on the case creation grouping field. All the generated alerts by the same rule will be grouped by the defined values (if any) within a defined window and will be assigned to the same case. when a pre-processing is defined, means just a single alert is created so using the same “value” (e.g. host for pre & post processing) we’ll get a single case with a single alert (because the other alerts with the same host were suppressed by the pre-processing).

Desired flow (draft UX below):

• The user defines the new case action with the grouping logic ( User cant define more than a single case creation (like other actions)
• The alert table presents the related case
• The case view presents the creation reason
  • Case title says that it was auto-generated (can be renamed by the user
  • Reporter should be "Auto-generated by Alerting rule"
  • First case activity says "Case was created by <rule name & link to the rule> and assign X alert"
  • For each new alert assignment a new alert assignment activity should be added.

ER:

• https://github.com/elastic/enhancements/issues/14779
• https://github.com/elastic/enhancements/issues/14333
• https://elastic.slack.com/archives/C01PCU5P60L/p1675223947526709
• https://elasticstack.slack.com/archives/CNKF2LEMP/p1674660002635719

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[elasticmachine]

Pinging @elastic/response-ops-cases (Feature:Cases)

[MikePaquette]

thanks @shanisagiv1

A few questions, please:

1. When the auto-create-case action is configured for a rule which has either alert-grouping or alert-suppression settings already configured, does it make sense to automatically populate the alerts-to-case grouping setting with this value? For example if we think this is the most common case, it could help reduce accidental misalignment.

It's true that we have a Server log or Index but it was wrong to add them as regular actions in the first place. we want to introduce a new type of action to the framework called "system actions" (case action will be the first and Server & index will be migrated in the future.. Nothing should be changed for the user in this release)

2. Will the single "system actions" still appear in the connector list?
3. How might our security response actions fit in? Today, we have dedicated response actions, but our vision is to make all security response actions operate on a common framework, so regardless of whether the response is to to sent, the ability to trigger it as a rule action is required. I want to make sure we don't rule out this use case by our decision.

[shanisagiv1]

Thanks for the questions @MikePaquette

1. yes, we're considering this default value, if you also think it's valuable, we definitely can support that.
2. yes (except the case creation), maybe we'll upgrade the ux in the future rule flyout that we're working on, but nothing is changed in Sec flow
3. Supporting system action ("response action". osquery like) is part of it, and we'll kick off this RFC soon - see more here

cc @cnasikas

[katrin-freihofner]

Thank you @shanisagiv1. I have a few questions:

1. For Observability it would be very interesting to not only group by a field but also group by dependencies (both, services and infrastructure assets). We are currently listing the dependencies and underlying infrastructure in APM. cc @alex-fedotyev

2. Are these cases and any updates to them automatically pushed to a 3rd party (if set up in the settings)?
3. Is there a plan to add features to this automatic case creation like letting users add tags and choose an assignee?

About

Reporter should be "Auto-generated by Alerting rule"

I suggest using "rule" instead of "Alerting rule" to be consistent with the rest of the UI.

[shanisagiv1]

1. Can you elaborate? what are these dependencies? is it part of the alert? if so, the user will be able to group by that
2. yes, but it's done manually for now. We're planning to allow auto-ticket creation when case is created.
3. we can do it if needed in a later phase
   cc @katrin-freihofner