[Request][8.15 & Serverless] Alert suppression for ES|QL and ML rules (…

…#5568)

* First draft

* More changes

* Fixing styles

* Fixed attribute

* Style update

* Vitalii's input

* ML rule update

* Update docs/detections/rules-ui-create.asciidoc

Co-authored-by: Ryland Herrick <[email protected]>

* Update docs/serverless/rules/rules-ui-create.mdx

* Update docs/detections/alert-suppression.asciidoc

Co-authored-by: natasha-moore-elastic <[email protected]>

* Update docs/serverless/alerts/alert-suppression.mdx

---------

Co-authored-by: Ryland Herrick <[email protected]>
Co-authored-by: natasha-moore-elastic <[email protected]>

docs/detections/alert-suppression.asciidoc

@@ -16,6 +16,8 @@ Alert suppression allows you to reduce the number of repeated or duplicate detec
 * <<create-indicator-rule,Indicator match>>
 * <<create-eql-rule,Event correlation>> (non-sequence queries only)
 * <<create-new-terms-rule,New terms>>  
+* <<create-esql-rule,{esql}>>
+* <<create-ml-rule,{ml-app}>>
 
 Normally, when a rule meets its criteria repeatedly, it creates multiple alerts, one for each time the rule's criteria are met. When alert suppression is configured, duplicate qualifying events are grouped, and only one alert is created for each group. Depending on the rule type, you can configure alert suppression to create alerts each time the rule runs, or once within a specified time window. You can also specify multiple fields to group events by unique combinations of values.
 
@@ -25,31 +27,28 @@ NOTE: Alert suppression is not available for Elastic prebuilt rules. However, if
 
 === Configure alert suppression
 
-You can configure alert suppression when you create or edit a supported rule type. Refer to documentation for creating <<create-custom-rule,custom query>>, <<create-threshold-rule, threshold>>, <<create-eql-rule,event correlation>>, or <<create-new-terms-rule,new terms>> rules for detailed instructions.
+You can configure alert suppression when you create or edit a supported rule type. Refer to documentation for creating <<create-custom-rule,custom query>>, <<create-threshold-rule, threshold>>, <<create-eql-rule,event correlation>>, <<create-new-terms-rule,new terms>>, <<create-esql-rule,{esql}>>, or <<create-ml-rule,{ml}>> rules for detailed instructions.
 
 . When configuring the rule type (the *Define rule* step for a new rule, or the *Definition* tab for an existing rule), specify how you want to group events for alert suppression:
 +
 --
-* Custom query rule: In *Suppress alerts by*, enter 1-3 field names to group events by the fields' values. 
-* Threshold rule: In *Group by*, enter up to 3 field names to group events by the fields' values, or leave the setting empty to group all qualifying events together. 
-* Indicator match rule: In *Suppress alerts by*, enter 1-3 field names to group events by the fields' values. 
-* Event correlation rule (non-sequence queries only): In *Suppress alerts by*, enter 1-3 field names to group events by the fields' values. 
-* New terms rule: In *Suppress alerts by*, enter 1-3 field names to group events by the fields' values.
+* **Custom query, indicator match, threshold, event correlation (non-sequence queries only), new terms, {ml}, and {esql} rules:** In *Suppress alerts by*, enter 1-3 field names to group events by the fields' values. 
+* **Threshold rule:** In *Group by*, enter up to 3 field names to group events by the fields' values, or leave the setting empty to group all qualifying events together. 
 
 --
 +
 [NOTE]
 ======
 If you specify a field with multiple values, alerts with that field are handled as follows:
 
-* **Custom query or threshold rules** - A group of alerts is created for each value. For example, if you suppress alerts by `destination.ip` of `[127.0.0.1, 127.0.0.2, 127.0.0.3]`, alerts will be suppressed separately for each value of `127.0.0.1`, `127.0.0.2`, and `127.0.0.3`. 
-* **Indicator match rule, event correlation (non-sequence queries only), or new terms rule** - Alerts with the specified field name and identical array values are grouped together. For example, if you suppress alerts by `destination.ip` of `[127.0.0.1, 127.0.0.2, 127.0.0.3]`, alerts with the entire array are grouped and only one alert is created for the group.
+* **Custom query or threshold rules:** A group of alerts is created for each value. For example, if you suppress alerts by `destination.ip` of `[127.0.0.1, 127.0.0.2, 127.0.0.3]`, alerts will be suppressed separately for each value of `127.0.0.1`, `127.0.0.2`, and `127.0.0.3`. 
+* **Indicator match, event correlation (non-sequence queries only), new terms, {esql}, or {ml} rules:** Alerts with the specified field name and identical array values are grouped together. For example, if you suppress alerts by `destination.ip` of `[127.0.0.1, 127.0.0.2, 127.0.0.3]`, alerts with the entire array are grouped and only one alert is created for the group.
 
 ======
 
 . If available, select how often to create alerts for duplicate events:
 +
-NOTE: Both options are available for custom query, indicator match, event correlation, and new terms rules. Threshold rules only have the *Per time period* option.
+NOTE: Both options are available for custom query, indicator match, event correlation, new terms, {esql}, and {ml} rules. Threshold rules only have the *Per time period* option.
 +
 --
 * *Per rule execution*: Create an alert each time the rule runs and meets its criteria.
@@ -113,5 +112,5 @@ image::images/timeline-button.png[Investigate in timeline button, 200]
 
 Some rule types have a maximum number of alerts that can be suppressed (custom query rules don't have a suppression limit):
 
-* **Threshold and event correlation (non-sequence queries only)** - The maximum number of alerts is the value you choose for the rule's **Max alerts per run** <<rule-ui-advanced-params,advanced setting>>, which is `100` by default.
-* **Indicator match and new terms** - The maximum number is five times the value you choose for the rule's **Max alerts per run** <<rule-ui-advanced-params,advanced setting>>. The default value is `100`, which means the default maximum limit for indicator match rules and new term rules is `500`.
+* **Threshold, event correlation (non-sequence queries only), {esql}, and {ml}:** The maximum number of alerts is the value you choose for the rule's **Max alerts per run** <<rule-ui-advanced-params,advanced setting>>, which is `100` by default.
+* **Indicator match and new terms:** The maximum number is five times the value you choose for the rule's **Max alerts per run** <<rule-ui-advanced-params,advanced setting>>. The default value is `100`, which means the default maximum limit for indicator match rules and new term rules is `500`.

docs/detections/api/rules/rules-api-create.asciidoc

@@ -503,11 +503,11 @@ a detection rule exception (`detection`) or an endpoint exception (`endpoint`).
 |==============================================
 
 [[opt-fields-alert-suppression-create]]
-===== Optional alert suppression fields for query, indicator match, threshold, event correlation (non-sequence queries only), and new terms rules
+===== Optional alert suppression fields for query, indicator match, threshold, event correlation (non-sequence queries only), new terms, {esql}, and {ml} rules
 
-preview::["Alert suppression is in technical preview for threshold, indicator match, event correlation, and new terms rules. The functionality may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features."]
+preview::["Alert suppression is in technical preview for threshold, indicator match, event correlation, new terms, {ml}, and {esql} rules. The functionality may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features."]
 
-====== Query, indicator match, event correlation (non-sequence queries only), and new terms rules
+====== Query, indicator match, event correlation (non-sequence queries only), new terms, {esql}, and {ml} rules
 
 [width="100%",options="header"]
 |==============================================

docs/detections/api/rules/rules-api-update.asciidoc

@@ -532,11 +532,11 @@ in the UI (*Rules* -> *Detection rules (SIEM)* -> *_Rule name_*).
 
 
 [[opt-fields-alert-suppression-update]]
-===== Optional alert suppression fields for query, indicator match, threshold, event correlation (non-sequence queries only), and new terms rules
+===== Optional alert suppression fields for query, indicator match, threshold, event correlation (non-sequence queries only), new terms, {esql}, and {ml} rules
 
-preview::["Alert suppression is in technical preview for threshold, indicator match, event correlation, and new terms rules. The functionality may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features."]
+preview::["Alert suppression is in technical preview for threshold, indicator match, event correlation, new terms, {ml}, and {esql} rules. The functionality may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features."]
 
-====== Query, indicator match, event correlation (non-sequence queries only), and new terms rules
+====== Query, indicator match, event correlation (non-sequence queries only), new terms, {esql}, and {ml} rules
 
 [width="100%",options="header"]
 |==============================================

docs/detections/rules-ui-create.asciidoc

@@ -46,6 +46,10 @@ then select:
 NOTE: If a required job isn't currently running, it will automatically start when you finish configuring and enable the rule.
 .. The anomaly score threshold above which alerts are created.
 +
+. preview:[] (Optional, https://www.elastic.co/pricing[Platinum or higher subscription] required) Use *Suppress alerts by* to reduce the number of repeated or duplicate alerts created by the rule. Refer to <<alert-suppression>> for more information.
++
+NOTE: Because {ml} rules generate alerts from anomalies, which don't contain source event fields, you can only use anomaly fields when configuring alert suppression.
++
 
 ////
 The following step is repeated across all rule types. If you change anything 
@@ -353,6 +357,9 @@ NOTE: Refer to the sections below to learn more about <<esql-rule-query-types,{e
 TIP: Click the help icon (image:images/esql-help-ref-button.png[Click the ES|QL help icon,20,20]) to open the in-product reference documentation for all {esql} commands and functions.
 +
 
+. preview:[] (Optional, https://www.elastic.co/pricing[Platinum or higher subscription] required) Use *Suppress alerts by* to reduce the number of repeated or duplicate alerts created by the rule. Refer to <<alert-suppression>> for more information.
++
+
 ////
 The following steps are repeated across multiple rule types. If you change anything 
 in these steps or sub-steps, apply the change to the other rule types, too.
@@ -466,6 +473,8 @@ If the `LIMIT` value and **Max alerts per run** value are different, the rule us
 
 - When writing an aggregating query, use the {ref}/esql-commands.html#esql-stats-by[`STATS...BY`] command with fields that you want to search and filter for after alerts are created. For example, using the `host.name`, `user.name`, `process.name` fields with the `BY` operator of the `STATS...BY` command returns these fields in alert documents, and allows you to search and filter for them from the Alerts table. 
 
+- When configuring alert suppression on a non-aggregating query, we recommend sorting results by ascending `@timestamp` order. Doing so ensures that alerts are properly suppressed, especially if the number of alerts generated is higher than the **Max alerts per run** value.
+
 [float]
 [[esql-rule-limitations]]
 ==== {esql} rule limitations 

docs/serverless/alerts/alert-suppression.mdx

@@ -20,6 +20,8 @@ Alert suppression allows you to reduce the number of repeated or duplicate detec
 * <DocLink slug="/serverless/security/rules-create" section="create-indicator-rule" text="Indicator match" />
 * <DocLink slug="/serverless/security/rules-create" section="create-eql-rule" text="Event correlation" /> (non-sequence queries only)
 * <DocLink slug="/serverless/security/rules-create" section="create-new-terms-rule" text="New terms" />
+* <DocLink slug="/serverless/security/rules-create" section="create-esql-rule" text="ES|QL" />
+* <DocLink slug="/serverless/security/rules-create" section="create-ml-rule" text="Machine learning" />
 
 Normally, when a rule meets its criteria repeatedly, it creates multiple alerts, one for each time the rule's criteria are met. When alert suppression is configured, duplicate qualifying events are grouped, and only one alert is created for each group. Depending on the rule type, you can configure alert suppression to create alerts each time the rule runs, or once within a specified time window. You can also specify multiple fields to group events by unique combinations of values.
 
@@ -31,28 +33,25 @@ Alert suppression is not available for Elastic prebuilt rules. However, if you w
 
 ## Configure alert suppression
 
-You can configure alert suppression when you create or edit a supported rule type. Refer to documentation for creating <DocLink slug="/serverless/security/rules-create" section="create-custom-rule" text="custom query"/>, <DocLink slug="/serverless/security/rules-create" section="create-threshold-rule" text="threshold"/>, <DocLink slug="/serverless/security/rules-create" section="create-indicator-rule" text="indicator match"/>, <DocLink slug="/serverless/security/rules-create" section="create-eql-rule" text="event correlation"/>, or <DocLink slug="/serverless/security/rules-create" section="create-new-terms-rule" text="new terms" /> rules for detailed instructions.
+You can configure alert suppression when you create or edit a supported rule type. Refer to documentation for creating <DocLink slug="/serverless/security/rules-create" section="create-custom-rule" text="custom query"/>, <DocLink slug="/serverless/security/rules-create" section="create-threshold-rule" text="threshold"/>, <DocLink slug="/serverless/security/rules-create" section="create-indicator-rule" text="indicator match"/>, <DocLink slug="/serverless/security/rules-create" section="create-eql-rule" text="event correlation"/>, <DocLink slug="/serverless/security/rules-create" section="create-new-terms-rule" text="new terms" />, <DocLink slug="/serverless/security/rules-create" section="create-esql-rule" text="ES|QL" />, or <DocLink slug="/serverless/security/rules-create" section="create-ml-rule" text="machine learning" /> rules for detailed instructions.
 
 1. When configuring the rule type (the **Define rule** step for a new rule, or the **Definition** tab for an existing rule), specify how you want to group events for alert suppression:
 
-    * Custom query rule: In **Suppress alerts by**, enter 1-3 field names to group events by the fields' values.  
-    * Threshold rule: In **Group by**, enter up to 3 field names to group events by the fields' values, or leave the setting empty to group all qualifying events together. 
-    * Indicator match rule: In **Suppress alerts by**, enter 1-3 field names to group events by the fields' values. 
-    * Event correlation rule (non-sequence queries only): In **Suppress alerts by**, enter 1-3 field names to group events by the fields' values.  
-    * New terms rule: In **Suppress alerts by**, enter 1-3 field names to group events by the fields' values. 
+    * **Custom query rule, indicator match, threshold, event correlation (non-sequence queries only), new terms, ((esql)), or ((ml)) rules:** In **Suppress alerts by**, enter 1-3 field names to group events by the fields' values.  
+    * **Threshold rule:** In **Group by**, enter up to 3 field names to group events by the fields' values, or leave the setting empty to group all qualifying events together. 
 
     <DocCallOut title="Note">  
     If you specify a field with multiple values, alerts with that field are handled as follows:
 
-    * **Custom query or threshold rules** - Alerts are grouped by each unique value. For example, if you suppress alerts by `destination.ip` of `[127.0.0.1, 127.0.0.2, 127.0.0.3]`, alerts will be suppressed separately for each value of `127.0.0.1`, `127.0.0.2`, and `127.0.0.3`. 
-    * **Indicator match, event correlation (non-sequence queries only), or new terms rule** - Alerts with the specified field name and identical array values are grouped together. For example, if you suppress alerts by `destination.ip` of `[127.0.0.1, 127.0.0.2, 127.0.0.3]`, alerts with the entire array are grouped and only one alert is created for the group. 
+    * **Custom query or threshold rules:** Alerts are grouped by each unique value. For example, if you suppress alerts by `destination.ip` of `[127.0.0.1, 127.0.0.2, 127.0.0.3]`, alerts will be suppressed separately for each value of `127.0.0.1`, `127.0.0.2`, and `127.0.0.3`. 
+    * **Indicator match, event correlation (non-sequence queries only), new terms, ((esql)), or ((ml)) rules:** Alerts with the specified field name and identical array values are grouped together. For example, if you suppress alerts by `destination.ip` of `[127.0.0.1, 127.0.0.2, 127.0.0.3]`, alerts with the entire array are grouped and only one alert is created for the group. 
 
     </DocCallOut>
 
 1. If available, select how often to create alerts for duplicate events:
 
      <DocCallOut title="NOTE">
-     Both options are available for custom query, indicator match, event correlation, and new terms rules. Threshold rules only have the **Per time period** option. 
+     Both options are available for custom query, indicator match, event correlation, new terms, ((esql)), and ((ml)) rules. Threshold rules only have the **Per time period** option. 
      </DocCallOut>
 
     * **Per rule execution**: Create an alert each time the rule runs and an event meets its criteria. 
@@ -112,5 +111,5 @@ With alert suppression, detection alerts aren't created for the grouped source e
 
 Some rule types have a maximum number of alerts that can be suppressed (custom query rules don't have a suppression limit):
 
-* **Threshold and event correlation (non-sequence queries only)** - The maximum number is the value you choose for the rule's **Max alerts per run** <DocLink slug="/serverless/security/rules-create" section="rule-ui-advanced-params">advanced setting</DocLink>, which is `100` by default.
-* **Indicator match and new terms** - The maximum number is five times the value you choose for the the rule's **Max alerts per run** <DocLink slug="/serverless/security/rules-create" section="rule-ui-advanced-params">advanced setting</DocLink>. The default value is `100`, which means the default maximum limit for indicator match rules and new terms rules is `500`.
+* **Threshold, event correlation (non-sequence queries only, ((esql)), and ((ml)):** The maximum number is the value you choose for the rule's **Max alerts per run** <DocLink slug="/serverless/security/rules-create" section="rule-ui-advanced-params">advanced setting</DocLink>, which is `100` by default.
+* **Indicator match and new terms:** The maximum number is five times the value you choose for the the rule's **Max alerts per run** <DocLink slug="/serverless/security/rules-create" section="rule-ui-advanced-params">advanced setting</DocLink>. The default value is `100`, which means the default maximum limit for indicator match rules and new terms rules is `500`.