Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Response Ops][Alerting] Update framework alerts client to write flattened alerts docs #167691

Merged
merged 6 commits into from
Oct 2, 2023

Conversation

ymao1
Copy link
Contributor

@ymao1 ymao1 commented Sep 29, 2023

Resolves #166946

PRs to this feature branch

Summary

The rule registry has traditionally written out AAD docs with flattened keys, like

{
  "kibana.alert.rule.name": "test"
}

The framework alerts client has been writing out AAD docs as objects, like

{
  "kibana": {
    "alert": {
      "rule": {
        "name": "test"
      }
    }
  }
}

We've identified a few places where we're updating the docs where having this divergence makes things more difficult, so this is to switch the framework to writing flattened alert docs before onboarding more rule types.

This PR is targeted for 8.11, which is also when we onboarded the index threshold rule type and the ML anomaly detection rule type to FAAD. For the ES query rule, which started writing unflattened AaD docs in 8.10, this PR adds special handling to ensure that those unflattened docs are correctly updated with flattened fields.

To Verify

ES Query and Index Threshold AaD

Create these rules that trigger alerts and verify that their AaD docs are written out as flattened. For the ES Query rule type, select a Metrics/Logs consumer and verify that they appear on the O11y alerts table.

ML alerts

ML alerts added in #166349 looked like:

Unflattened
{
	"kibana": {
		"alert": {
			"url": "/app/ml/explorer/?_g=(ml%3A(jobIds%3A!(rt-anomaly-mean-value))%2Ctime%3A(from%3A'2023-09-28T14%3A57%3A00.000Z'%2Cmode%3Aabsolute%2Cto%3A'2023-09-28T15%3A17%3A00.000Z'))&_a=(explorer%3A(mlExplorerFilter%3A(filterActive%3A!t%2CfilteredFields%3A!(key%2Cthird-key)%2CinfluencersFilterQuery%3A(bool%3A(minimum_should_match%3A1%2Cshould%3A!((match_phrase%3A(key%3Athird-key)))))%2CqueryString%3A'key%3A%22third-key%22')%2CmlExplorerSwimlane%3A()))",
			"reason": "Alerts are raised based on real-time scores. Remember that scores may be adjusted over time as data continues to be analyzed.",
			"job_id": "rt-anomaly-mean-value",
			"anomaly_score": 73.63508175828011,
			"is_interim": false,
			"anomaly_timestamp": 1695913620000,
			"top_records": [{
				"job_id": "rt-anomaly-mean-value",
				"record_score": 73.63516446528412,
				"initial_record_score": 73.63516446528412,
				"detector_index": 0,
				"is_interim": false,
				"timestamp": 1695913620000,
				"partition_field_name": "key",
				"partition_field_value": "third-key",
				"function": "mean",
				"actual": [
					3
				],
				"typical": [
					4.187715468532429
				]
			}],
			"top_influencers": [{
				"job_id": "rt-anomaly-mean-value",
				"influencer_field_name": "key",
				"influencer_field_value": "third-key",
				"influencer_score": 73.63508175828011,
				"initial_influencer_score": 73.63508175828011,
				"is_interim": false,
				"timestamp": 1695913620000
			}],
			"action_group": "anomaly_score_match",
			"flapping": false,
			"flapping_history": [
				true,
				false,
				false,
				false
			],
			"instance": {
				"id": "rt-anomaly-mean-value"
			},
			"maintenance_window_ids": [],
			"rule": {
				"category": "Anomaly detection alert",
				"consumer": "alerts",
				"execution": {
					"uuid": "e9e681d4-c8e4-43eb-82e5-a58bdf7ffe12"
				},
				"name": "rt-ad-alert-influencer",
				"parameters": {
					"severity": 5,
					"resultType": "influencer",
					"includeInterim": false,
					"jobSelection": {
						"jobIds": [
							"rt-anomaly-mean-value"
						],
						"groupIds": []
					},
					"lookbackInterval": null,
					"topNBuckets": null
				},
				"producer": "ml",
				"revision": 0,
				"rule_type_id": "xpack.ml.anomaly_detection_alert",
				"tags": [],
				"uuid": "9e1d6bc0-5e10-11ee-8416-3bf48cca0922"
			},
			"status": "active",
			"uuid": "c9c1f075-9985-4c55-8ff8-22349cb30269",
			"workflow_status": "open",
			"duration": {
				"us": "99021000000"
			},
			"start": "2023-09-28T15:07:12.868Z",
			"time_range": {
				"gte": "2023-09-28T15:07:12.868Z"
			}
		},
		"space_ids": [
			"default"
		],
		"version": "8.11.0"
	},
	"@timestamp": "2023-09-28T15:08:51.889Z",
	"event": {
		"action": "active",
		"kind": "signal"
	},
	"tags": []
}

Now they look like:

Flattened
{
	"kibana.alert.url": "/app/ml/explorer/?_g=(ml%3A(jobIds%3A!(rt-anomaly-mean-value))%2Ctime%3A(from%3A'2023-09-28T15%3A03%3A00.000Z'%2Cmode%3Aabsolute%2Cto%3A'2023-09-28T15%3A23%3A00.000Z'))&_a=(explorer%3A(mlExplorerFilter%3A(filterActive%3A!t%2CfilteredFields%3A!(key%2Cthird-key)%2CinfluencersFilterQuery%3A(bool%3A(minimum_should_match%3A1%2Cshould%3A!((match_phrase%3A(key%3Athird-key)))))%2CqueryString%3A'key%3A%22third-key%22')%2CmlExplorerSwimlane%3A()))",
	"kibana.alert.reason": "Alerts are raised based on real-time scores. Remember that scores may be adjusted over time as data continues to be analyzed.",
	"kibana.alert.job_id": "rt-anomaly-mean-value",
	"kibana.alert.anomaly_score": 72.75515452061356,
	"kibana.alert.is_interim": false,
	"kibana.alert.anomaly_timestamp": 1695913980000,
	"kibana.alert.top_records": [{
		"job_id": "rt-anomaly-mean-value",
		"record_score": 72.75515452061356,
		"initial_record_score": 72.75515452061356,
		"detector_index": 0,
		"is_interim": false,
		"timestamp": 1695913980000,
		"partition_field_name": "key",
		"partition_field_value": "third-key",
		"function": "mean",
		"actual": [
			0.5
		],
		"typical": [
			4.138745343296527
		]
	}],
	"kibana.alert.top_influencers": [{
		"job_id": "rt-anomaly-mean-value",
		"influencer_field_name": "key",
		"influencer_field_value": "third-key",
		"influencer_score": 72.75515452061356,
		"initial_influencer_score": 72.75515452061356,
		"is_interim": false,
		"timestamp": 1695913980000
	}],
	"kibana.alert.rule.category": "Anomaly detection alert",
	"kibana.alert.rule.consumer": "alerts",
	"kibana.alert.rule.execution.uuid": "17fef3d3-d595-4362-837e-b2a73650169e",
	"kibana.alert.rule.name": "rt-ad-alert-influencer",
	"kibana.alert.rule.parameters": {
		"severity": 5,
		"resultType": "influencer",
		"includeInterim": false,
		"jobSelection": {
			"jobIds": [
				"rt-anomaly-mean-value"
			],
			"groupIds": []
		},
		"lookbackInterval": null,
		"topNBuckets": null
	},
	"kibana.alert.rule.producer": "ml",
	"kibana.alert.rule.revision": 0,
	"kibana.alert.rule.rule_type_id": "xpack.ml.anomaly_detection_alert",
	"kibana.alert.rule.tags": [],
	"kibana.alert.rule.uuid": "757c7610-5e11-11ee-8bc6-a95c3ced4757",
	"kibana.space_ids": [
		"default"
	],
	"@timestamp": "2023-09-28T15:14:52.057Z",
	"event.action": "active",
	"event.kind": "signal",
	"kibana.alert.action_group": "anomaly_score_match",
	"kibana.alert.flapping": false,
	"kibana.alert.flapping_history": [
		true,
		false,
		false,
		false
	],
	"kibana.alert.instance.id": "rt-anomaly-mean-value",
	"kibana.alert.maintenance_window_ids": [],
	"kibana.alert.status": "active",
	"kibana.alert.uuid": "ac1f0d7c-461b-4fc6-b4c3-04416ac876d3",
	"kibana.alert.workflow_status": "open",
	"kibana.alert.duration.us": "99028000000",
	"kibana.alert.start": "2023-09-28T15:13:13.028Z",
	"kibana.alert.time_range": {
		"gte": "2023-09-28T15:13:13.028Z"
	},
	"kibana.version": "8.11.0",
	"tags": []
}

…tened alerts docs (#167439)

Resolves #166946

## Summary

The rule registry has traditionally written out AAD docs with flattened
keys, like

```
{
  "kibana.alert.rule.name": "test"
}
```

The framework alerts client has been writing out AAD docs as objects,
like

```
{
  "kibana": {
    "alert": {
      "rule": {
        "name": "test"
      }
    }
  }
}
```

We've identified a few places where we're updating the docs where having
this divergence makes things more difficult, so this is to switch the
framework to writing flattened alert docs before onboarding more rule
types.

This PR is targeted for 8.11, which is also when we onboarded the index
threshold rule type to FAAD. The only other rule type using FAAD to
write docs is ES query, which landed in 8.10 so there will be a followup
issue to handle the case of updating unflattened ES query AAD docs from
8.10

## To Verify

### ES Query and Index Threshold AaD

Create these rules that trigger alerts and verify that their AaD docs
are written out as flattened. For the ES Query rule type, select a
Metrics/Logs consumer and verify that they appear on the O11y alerts
table.

### ML alerts

ML alerts added in #166349 looked
like:

<details>
  <summary>Unflattened</summary>

```
{
	"kibana": {
		"alert": {
			"url": "/app/ml/explorer/?_g=(ml%3A(jobIds%3A!(rt-anomaly-mean-value))%2Ctime%3A(from%3A'2023-09-28T14%3A57%3A00.000Z'%2Cmode%3Aabsolute%2Cto%3A'2023-09-28T15%3A17%3A00.000Z'))&_a=(explorer%3A(mlExplorerFilter%3A(filterActive%3A!t%2CfilteredFields%3A!(key%2Cthird-key)%2CinfluencersFilterQuery%3A(bool%3A(minimum_should_match%3A1%2Cshould%3A!((match_phrase%3A(key%3Athird-key)))))%2CqueryString%3A'key%3A%22third-key%22')%2CmlExplorerSwimlane%3A()))",
			"reason": "Alerts are raised based on real-time scores. Remember that scores may be adjusted over time as data continues to be analyzed.",
			"job_id": "rt-anomaly-mean-value",
			"anomaly_score": 73.63508175828011,
			"is_interim": false,
			"anomaly_timestamp": 1695913620000,
			"top_records": [{
				"job_id": "rt-anomaly-mean-value",
				"record_score": 73.63516446528412,
				"initial_record_score": 73.63516446528412,
				"detector_index": 0,
				"is_interim": false,
				"timestamp": 1695913620000,
				"partition_field_name": "key",
				"partition_field_value": "third-key",
				"function": "mean",
				"actual": [
					3
				],
				"typical": [
					4.187715468532429
				]
			}],
			"top_influencers": [{
				"job_id": "rt-anomaly-mean-value",
				"influencer_field_name": "key",
				"influencer_field_value": "third-key",
				"influencer_score": 73.63508175828011,
				"initial_influencer_score": 73.63508175828011,
				"is_interim": false,
				"timestamp": 1695913620000
			}],
			"action_group": "anomaly_score_match",
			"flapping": false,
			"flapping_history": [
				true,
				false,
				false,
				false
			],
			"instance": {
				"id": "rt-anomaly-mean-value"
			},
			"maintenance_window_ids": [],
			"rule": {
				"category": "Anomaly detection alert",
				"consumer": "alerts",
				"execution": {
					"uuid": "e9e681d4-c8e4-43eb-82e5-a58bdf7ffe12"
				},
				"name": "rt-ad-alert-influencer",
				"parameters": {
					"severity": 5,
					"resultType": "influencer",
					"includeInterim": false,
					"jobSelection": {
						"jobIds": [
							"rt-anomaly-mean-value"
						],
						"groupIds": []
					},
					"lookbackInterval": null,
					"topNBuckets": null
				},
				"producer": "ml",
				"revision": 0,
				"rule_type_id": "xpack.ml.anomaly_detection_alert",
				"tags": [],
				"uuid": "9e1d6bc0-5e10-11ee-8416-3bf48cca0922"
			},
			"status": "active",
			"uuid": "c9c1f075-9985-4c55-8ff8-22349cb30269",
			"workflow_status": "open",
			"duration": {
				"us": "99021000000"
			},
			"start": "2023-09-28T15:07:12.868Z",
			"time_range": {
				"gte": "2023-09-28T15:07:12.868Z"
			}
		},
		"space_ids": [
			"default"
		],
		"version": "8.11.0"
	},
	"@timestamp": "2023-09-28T15:08:51.889Z",
	"event": {
		"action": "active",
		"kind": "signal"
	},
	"tags": []
}
```
</details>

Now they look like:

<details>
  <summary>Flattened</summary>

```
{
	"kibana.alert.url": "/app/ml/explorer/?_g=(ml%3A(jobIds%3A!(rt-anomaly-mean-value))%2Ctime%3A(from%3A'2023-09-28T15%3A03%3A00.000Z'%2Cmode%3Aabsolute%2Cto%3A'2023-09-28T15%3A23%3A00.000Z'))&_a=(explorer%3A(mlExplorerFilter%3A(filterActive%3A!t%2CfilteredFields%3A!(key%2Cthird-key)%2CinfluencersFilterQuery%3A(bool%3A(minimum_should_match%3A1%2Cshould%3A!((match_phrase%3A(key%3Athird-key)))))%2CqueryString%3A'key%3A%22third-key%22')%2CmlExplorerSwimlane%3A()))",
	"kibana.alert.reason": "Alerts are raised based on real-time scores. Remember that scores may be adjusted over time as data continues to be analyzed.",
	"kibana.alert.job_id": "rt-anomaly-mean-value",
	"kibana.alert.anomaly_score": 72.75515452061356,
	"kibana.alert.is_interim": false,
	"kibana.alert.anomaly_timestamp": 1695913980000,
	"kibana.alert.top_records": [{
		"job_id": "rt-anomaly-mean-value",
		"record_score": 72.75515452061356,
		"initial_record_score": 72.75515452061356,
		"detector_index": 0,
		"is_interim": false,
		"timestamp": 1695913980000,
		"partition_field_name": "key",
		"partition_field_value": "third-key",
		"function": "mean",
		"actual": [
			0.5
		],
		"typical": [
			4.138745343296527
		]
	}],
	"kibana.alert.top_influencers": [{
		"job_id": "rt-anomaly-mean-value",
		"influencer_field_name": "key",
		"influencer_field_value": "third-key",
		"influencer_score": 72.75515452061356,
		"initial_influencer_score": 72.75515452061356,
		"is_interim": false,
		"timestamp": 1695913980000
	}],
	"kibana.alert.rule.category": "Anomaly detection alert",
	"kibana.alert.rule.consumer": "alerts",
	"kibana.alert.rule.execution.uuid": "17fef3d3-d595-4362-837e-b2a73650169e",
	"kibana.alert.rule.name": "rt-ad-alert-influencer",
	"kibana.alert.rule.parameters": {
		"severity": 5,
		"resultType": "influencer",
		"includeInterim": false,
		"jobSelection": {
			"jobIds": [
				"rt-anomaly-mean-value"
			],
			"groupIds": []
		},
		"lookbackInterval": null,
		"topNBuckets": null
	},
	"kibana.alert.rule.producer": "ml",
	"kibana.alert.rule.revision": 0,
	"kibana.alert.rule.rule_type_id": "xpack.ml.anomaly_detection_alert",
	"kibana.alert.rule.tags": [],
	"kibana.alert.rule.uuid": "757c7610-5e11-11ee-8bc6-a95c3ced4757",
	"kibana.space_ids": [
		"default"
	],
	"@timestamp": "2023-09-28T15:14:52.057Z",
	"event.action": "active",
	"event.kind": "signal",
	"kibana.alert.action_group": "anomaly_score_match",
	"kibana.alert.flapping": false,
	"kibana.alert.flapping_history": [
		true,
		false,
		false,
		false
	],
	"kibana.alert.instance.id": "rt-anomaly-mean-value",
	"kibana.alert.maintenance_window_ids": [],
	"kibana.alert.status": "active",
	"kibana.alert.uuid": "ac1f0d7c-461b-4fc6-b4c3-04416ac876d3",
	"kibana.alert.workflow_status": "open",
	"kibana.alert.duration.us": "99028000000",
	"kibana.alert.start": "2023-09-28T15:13:13.028Z",
	"kibana.alert.time_range": {
		"gte": "2023-09-28T15:13:13.028Z"
	},
	"kibana.version": "8.11.0",
	"tags": []
}
```
</details>
@ymao1 ymao1 self-assigned this Sep 29, 2023
@ymao1 ymao1 added Feature:Alerting release_note:skip Skip the PR/issue when compiling release notes Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) Feature:Alerting/Alerts-as-Data Issues related to Alerts-as-data and RuleRegistry v8.11.0 labels Sep 29, 2023
Copy link
Contributor

@darnautov darnautov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ML changes tested and LGTM.

ymao1 and others added 2 commits October 2, 2023 11:16
This is targeting a feature branch

## Summary

Since we are switching to writing out flattened AaD docs, we may
encounter a situation where existing ES query AaD docs are written out
and need to be updated after upgrade. This PR tries to handle this case
and update those docs so no duplicate data (flattened & unflattened) is
written to the doc.

## To Verify

On `main`, create an ES query rule that writes an (unflattened) alert
doc. Then switch to this branch and let the rule run so that the alert
is updated (either updated as ongoing or set as recovered). Inspect the
doc and verify that while the doc might be a mix of flattened & expanded
keys, there is no duplicate data.

### Updating a "new" alert

<details>
  <summary>Unflattened "new" alert</summary>

```
{
	"kibana": {
		"alert": {
			"url": "/app/management/insightsAndAlerting/triggersActions/rule/a3c20a80-5ef7-11ee-8fba-6f25f0ebd840",
			"reason": "rule 'es query rule new alert' is active: - Value: 98 - Conditions Met: Number of matching documents is greater than 0 over 5 h - Timestamp: 2023 - 09 - 29 T18: 40: 24.504 Z - Link: /app/management/insightsAndAlerting/triggersActions/rule/a3c20a80-5ef7-11ee-8fba-6f25f0ebd840",
			"title": "rule 'es query rule new alert' matched query",
			"evaluation": {
				"conditions": "Number of matching documents is greater than 0",
				"value": 98
			},
			"action_group": "query matched",
			"flapping": false,
			"flapping_history": [
				true
			],
			"instance": {
				"id": "query matched"
			},
			"maintenance_window_ids": [],
			"rule": {
				"category": "Elasticsearch query",
				"consumer": "stackAlerts",
				"execution": {
					"uuid": "c1fb180b-e6d6-4187-b01b-48dad60fb9ef"
				},
				"name": "es query rule new alert",
				"parameters": {
					"searchType": "esQuery",
					"timeWindowSize": 5,
					"timeWindowUnit": "h",
					"threshold": [
						0
					],
					"thresholdComparator": ">",
					"size": 100,
					"esQuery": "{\"query\": {\"match_all\": {}}}",
					"aggType": "count",
					"groupBy": "all",
					"termSize": 5,
					"excludeHitsFromPreviousRun": false,
					"index": [
						".kibana-event-log*"
					],
					"timeField": "@timestamp"
				},
				"producer": "stackAlerts",
				"revision": 0,
				"rule_type_id": ".es-query",
				"tags": [],
				"uuid": "a3c20a80-5ef7-11ee-8fba-6f25f0ebd840"
			},
			"status": "active",
			"uuid": "2049c3d7-16f6-409d-a40d-7a831b03652d",
			"workflow_status": "open",
			"duration": {
				"us": "0"
			},
			"start": "2023-09-29T18:40:24.516Z",
			"time_range": {
				"gte": "2023-09-29T18:40:24.516Z"
			}
		},
		"space_ids": [
			"default"
		],
		"version": "8.11.0"
	},
	"@timestamp": "2023-09-29T18:40:24.517Z",
	"event": {
		"action": "open",
		"kind": "signal"
	},
	"tags": []
}
```
</details>

<details>
  <summary>After update to "active"</summary>

```
{
	"kibana": {
		"alert": {
			"instance": {
				"id": "query matched"
			},
			"uuid": "2049c3d7-16f6-409d-a40d-7a831b03652d",
			"start": "2023-09-29T18:40:24.516Z"
		}
	},
	"@timestamp": "2023-09-29T18:49:13.502Z",
	"event": {
		"kind": "signal"
	},
	"kibana.alert.workflow_status": "open",
	"kibana.alert.status": "active",
	"kibana.alert.url": "/app/management/insightsAndAlerting/triggersActions/rule/a3c20a80-5ef7-11ee-8fba-6f25f0ebd840",
	"kibana.alert.reason": "rule 'es query rule new alert' is active: - Value: 765 - Conditions Met: Number of matching documents is greater than 0 over 5 h - Timestamp: 2023 - 09 - 29 T18: 40: 24.504 Z - Link: /app/management/insightsAndAlerting/triggersActions/rule/a3c20a80-5ef7-11ee-8fba-6f25f0ebd840",
	"kibana.alert.title": "rule 'es query rule new alert' matched query",
	"kibana.alert.evaluation.conditions": "Number of matching documents is greater than 0",
	"kibana.alert.evaluation.value": "765",
	"kibana.alert.rule.category": "Elasticsearch query",
	"kibana.alert.rule.consumer": "stackAlerts",
	"kibana.alert.rule.execution.uuid": "b6b4e596-6025-47c5-9344-0d4b081549bb",
	"kibana.alert.rule.name": "es query rule new alert",
	"kibana.alert.rule.parameters": {
		"searchType": "esQuery",
		"timeWindowSize": 5,
		"timeWindowUnit": "h",
		"threshold": [
			0
		],
		"thresholdComparator": ">",
		"size": 100,
		"esQuery": "{\"query\": {\"match_all\": {}}}",
		"aggType": "count",
		"groupBy": "all",
		"termSize": 5,
		"excludeHitsFromPreviousRun": false,
		"index": [
			".kibana-event-log*"
		],
		"timeField": "@timestamp"
	},
	"kibana.alert.rule.producer": "stackAlerts",
	"kibana.alert.rule.revision": 0,
	"kibana.alert.rule.rule_type_id": ".es-query",
	"kibana.alert.rule.tags": [],
	"kibana.alert.rule.uuid": "a3c20a80-5ef7-11ee-8fba-6f25f0ebd840",
	"kibana.space_ids": [
		"default"
	],
	"event.action": "active",
	"kibana.alert.action_group": "query matched",
	"kibana.alert.flapping": false,
	"kibana.alert.flapping_history": [
		true,
		false
	],
	"kibana.alert.maintenance_window_ids": [],
	"kibana.alert.time_range": {
		"gte": "2023-09-29T18:40:24.516Z"
	},
	"kibana.alert.duration.us": "528985000000",
	"kibana.version": "8.11.0",
	"tags": []
}
```
</details>


### Updating an "active" alert

<details>
  <summary>Unflattened "active" alert</summary>

```
{
	"kibana": {
		"alert": {
			"url": "/app/management/insightsAndAlerting/triggersActions/rule/cc998410-5ef7-11ee-8fba-6f25f0ebd840",
			"reason": "rule 'es query rule new alert' is active: - Value: 265 - Conditions Met: Number of matching documents is greater than 0 over 5 h - Timestamp: 2023-09-29T18:42:06.590Z - Link: /app/management/insightsAndAlerting/triggersActions/rule/cc998410-5ef7-11ee-8fba-6 f25f0ebd840",
			"title": "rule 'es query rule active alert' matched query",
			"evaluation": {
				"conditions": "Number of matching documents is greater than 0",
				"value": 265
			},
			"action_group": "query matched",
			"flapping": false,
			"flapping_history": [
				true,
				false
			],
			"instance": {
				"id": "query matched"
			},
			"maintenance_window_ids": [],
			"rule": {
				"category": "Elasticsearch query",
				"consumer": "stackAlerts",
				"execution": {
					"uuid": "71019c6f-b64c-4f16-a07b-9d8f32014d13"
				},
				"name": "es query rule active alert",
				"parameters": {
					"searchType": "esQuery",
					"timeWindowSize": 5,
					"timeWindowUnit": "h",
					"threshold": [
						0
					],
					"thresholdComparator": ">",
					"size": 100,
					"esQuery": "{\"query\": {\"match_all\": {}}}",
					"aggType": "count",
					"groupBy": "all",
					"termSize": 5,
					"excludeHitsFromPreviousRun": false,
					"index": [
						".kibana-event-log*"
					],
					"timeField": "@timestamp"
				},
				"producer": "stackAlerts",
				"revision": 0,
				"rule_type_id": ".es-query",
				"tags": [],
				"uuid": "cc998410-5ef7-11ee-8fba-6f25f0ebd840"
			},
			"status": "active",
			"uuid": "68fcea61-8f0c-4f1f-8c70-93b6a0ba3cd2",
			"workflow_status": "open",
			"duration": {
				"us": "36113000000"
			},
			"start": "2023-09-29T18:41:30.482Z",
			"time_range": {
				"gte": "2023-09-29T18:41:30.482Z"
			}
		},
		"space_ids": [
			"default"
		],
		"version": "8.11.0"
	},
	"@timestamp": "2023-09-29T18:42:06.596Z",
	"event": {
		"action": "active",
		"kind": "signal"
	},
	"tags": []
}
```
</details>

<details>
  <summary>After update to "recovered"</summary>

```
{
	"kibana": {
		"alert": {
			"instance": {
				"id": "query matched"
			},
			"uuid": "68fcea61-8f0c-4f1f-8c70-93b6a0ba3cd2"
		}
	},
	"@timestamp": "2023-09-29T19:03:58.728Z",
	"event": {
		"kind": "signal"
	},
	"kibana.alert.workflow_status": "open",
	"kibana.alert.status": "recovered",
	"kibana.alert.url": "/app/management/insightsAndAlerting/triggersActions/rule/cc998410-5ef7-11ee-8fba-6f25f0ebd840",
	"kibana.alert.reason": "rule 'es query rule new alert' is recovered: - Value: 0 - Conditions Met: Number of matching documents is NOT less than 0 over 5 h - Timestamp: 2023 - 09 - 29 T19: 03: 58.722 Z - Link: /app/management/insightsAndAlerting/triggersActions/rule/cc998410-5ef7-11ee-8fba-6 f25f0ebd840",
	"kibana.alert.title": "rule 'es query rule active alert' recovered",
	"kibana.alert.evaluation.conditions": "Number of matching documents is NOT less than 0",
	"kibana.alert.evaluation.value": "0",
	"kibana.alert.rule.category": "Elasticsearch query",
	"kibana.alert.rule.consumer": "stackAlerts",
	"kibana.alert.rule.execution.uuid": "8148e81a-197d-4783-ae0c-95eea44b5148",
	"kibana.alert.rule.name": "es query rule active alert",
	"kibana.alert.rule.parameters": {
		"searchType": "esQuery",
		"timeWindowSize": 5,
		"timeWindowUnit": "h",
		"threshold": [
			0
		],
		"thresholdComparator": "<",
		"size": 100,
		"esQuery": "{\"query\": {\"match_all\": {}}}",
		"aggType": "count",
		"groupBy": "all",
		"termSize": 5,
		"excludeHitsFromPreviousRun": false,
		"index": [
			".kibana-event-log*"
		],
		"timeField": "@timestamp"
	},
	"kibana.alert.rule.producer": "stackAlerts",
	"kibana.alert.rule.revision": 1,
	"kibana.alert.rule.rule_type_id": ".es-query",
	"kibana.alert.rule.tags": [],
	"kibana.alert.rule.uuid": "cc998410-5ef7-11ee-8fba-6f25f0ebd840",
	"kibana.space_ids": [
		"default"
	],
	"event.action": "close",
	"kibana.alert.action_group": "recovered",
	"kibana.alert.flapping": false,
	"kibana.alert.flapping_history": [
		true,
		false,
		true
	],
	"kibana.alert.maintenance_window_ids": [],
	"kibana.alert.duration.us": "1348246000000",
	"kibana.alert.start": "2023-09-29T18:41:30.482Z",
	"kibana.alert.end": "2023-09-29T19:03:58.728Z",
	"kibana.alert.time_range": {
		"gte": "2023-09-29T18:41:30.482Z",
		"lte": "2023-09-29T19:03:58.728Z"
	},
	"kibana.version": "8.11.0",
	"tags": []

}
```
</details>

### Updating a "recovered" alert

<details>
  <summary>Unflattened "recovered" alert</summary>

```
{
	"kibana": {
		"alert": {
			"url": "/app/management/insightsAndAlerting/triggersActions/rule/0b5ba5c0-5ef8-11ee-8fba-6f25f0ebd840",
			"reason": "rule 'es query rule new alert' is recovered: - Value: 0 - Conditions Met: Number of matching documents is NOT less than 0 over 5 h - Timestamp: 2023 - 09 - 29 T18: 43: 54.553 Z - Link: /app/management/insightsAndAlerting/triggersActions/rule/0b5ba5c0-5ef8-11ee-8fba-6f25f0ebd840",
			"title": "rule 'es query recovered alert' recovered",
			"evaluation": {
				"conditions": "Number of matching documents is NOT less than 0",
				"value": 0
			},
			"action_group": "recovered",
			"flapping": false,
			"flapping_history": [
				true,
				false,
				true
			],
			"instance": {
				"id": "query matched"
			},
			"maintenance_window_ids": [],
			"rule": {
				"category": "Elasticsearch query",
				"consumer": "stackAlerts",
				"execution": {
					"uuid": "b0de00a7-ee0d-4f6e-9c48-3057ec51eb36"
				},
				"name": "es query recovered alert",
				"parameters": {
					"searchType": "esQuery",
					"timeWindowSize": 5,
					"timeWindowUnit": "h",
					"threshold": [
						0
					],
					"thresholdComparator": "<",
					"size": 100,
					"esQuery": "{\"query\": {\"match_all\": {}}}",
					"aggType": "count",
					"groupBy": "all",
					"termSize": 5,
					"excludeHitsFromPreviousRun": false,
					"index": [
						".kibana-event-log*"
					],
					"timeField": "@timestamp"
				},
				"producer": "stackAlerts",
				"revision": 1,
				"rule_type_id": ".es-query",
				"tags": [],
				"uuid": "0b5ba5c0-5ef8-11ee-8fba-6f25f0ebd840"
			},
			"status": "recovered",
			"uuid": "0ec02f4a-b898-4a5b-8f4c-599c5f08679a",
			"workflow_status": "open",
			"duration": {
				"us": "38971000000"
			},
			"start": "2023-09-29T18:43:15.587Z",
			"time_range": {
				"gte": "2023-09-29T18:43:15.587Z",
				"lte": "2023-09-29T18:43:54.558Z"
			},
			"end": "2023-09-29T18:43:54.558Z"
		},
		"space_ids": [
			"default"
		],
		"version": "8.11.0"
	},
	"@timestamp": "2023-09-29T18:43:54.559Z",
	"event": {
		"action": "close",
		"kind": "signal"
	},
	"tags": []
}
```
</details>

<details>
  <summary>After another update to "recovered"</summary>

```
{
	"kibana": {
		"alert": {
			"url": "/app/management/insightsAndAlerting/triggersActions/rule/0b5ba5c0-5ef8-11ee-8fba-6f25f0ebd840",
			"reason": "rule 'es query rule new alert' is recovered: - Value: 0 - Conditions Met: Number of matching documents is NOT less than 0 over 5 h - Timestamp: 2023 - 09 - 29 T18: 43: 54.553 Z - Link: /app/management/insightsAndAlerting/triggersActions/rule/0b5ba5c0-5ef8-11ee-8fba-6f25f0ebd840",
			"title": "rule 'es query recovered alert' recovered",
			"evaluation": {
				"conditions": "Number of matching documents is NOT less than 0"
			},
			"instance": {
				"id": "query matched"
			},
			"uuid": "0ec02f4a-b898-4a5b-8f4c-599c5f08679a",
			"duration": {
				"us": "38971000000"
			},
			"start": "2023-09-29T18:43:15.587Z",
			"time_range": {
				"gte": "2023-09-29T18:43:15.587Z",
				"lte": "2023-09-29T18:43:54.558Z"
			},
			"end": "2023-09-29T18:43:54.558Z"
		}
	},
	"@timestamp": "2023-09-29T19:08:37.786Z",
	"event": {
		"kind": "signal"
	},
	"kibana.alert.workflow_status": "open",
	"kibana.alert.status": "recovered",
	"kibana.alert.rule.category": "Elasticsearch query",
	"kibana.alert.rule.consumer": "stackAlerts",
	"kibana.alert.rule.execution.uuid": "4fd57a70-4079-4e4b-88f7-97d9a47f8b7c",
	"kibana.alert.rule.name": "es query recovered alert",
	"kibana.alert.rule.parameters": {
		"searchType": "esQuery",
		"timeWindowSize": 5,
		"timeWindowUnit": "h",
		"threshold": [
			0
		],
		"thresholdComparator": "<",
		"size": 100,
		"esQuery": "{\"query\": {\"match_all\": {}}}",
		"aggType": "count",
		"groupBy": "all",
		"termSize": 5,
		"excludeHitsFromPreviousRun": false,
		"index": [
			".kibana-event-log*"
		],
		"timeField": "@timestamp"
	},
	"kibana.alert.rule.producer": "stackAlerts",
	"kibana.alert.rule.revision": 1,
	"kibana.alert.rule.rule_type_id": ".es-query",
	"kibana.alert.rule.tags": [],
	"kibana.alert.rule.uuid": "0b5ba5c0-5ef8-11ee-8fba-6f25f0ebd840",
	"kibana.space_ids": [
		"default"
	],
	"event.action": "close",
	"kibana.alert.action_group": "recovered",
	"kibana.alert.flapping": false,
	"kibana.alert.flapping_history": [
		true,
		false,
		true,
		false
	],
	"kibana.alert.maintenance_window_ids": [],
	"kibana.version": "8.11.0",
	"tags": []
}
```
</details>

---------

Co-authored-by: kibanamachine <[email protected]>
@ymao1 ymao1 marked this pull request as ready for review October 2, 2023 15:26
@ymao1 ymao1 requested review from a team as code owners October 2, 2023 15:26
@elasticmachine
Copy link
Contributor

Pinging @elastic/response-ops (Team:ResponseOps)

@kibana-ci
Copy link
Collaborator

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

  • [job] [logs] FTR Configs #12 / Alerting APIs Alerting rules "after each" hook for "should retry when appropriate"
  • [job] [logs] FTR Configs #43 / serverless security UI Cases List "after all" hook in "Cases List"
  • [job] [logs] FTR Configs #43 / serverless security UI Cases List "before all" hook in "Cases List"

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
apm 3.7MB 3.7MB +23.0B
infra 1.9MB 1.9MB +23.0B
observability 1.0MB 1.0MB +23.0B
securitySolution 13.0MB 13.0MB +23.0B
triggersActionsUi 1.4MB 1.4MB +23.0B
total +115.0B

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @ymao1

Copy link
Contributor

@ersin-erdal ersin-erdal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
As the child PRs already have already been approved, i just ran the PR locally and tested the data.
Observed the alert data is flattened and visible under observability->alerts.

@ymao1 ymao1 merged commit 3bff7fb into main Oct 2, 2023
@kibanamachine kibanamachine added the backport:skip This commit does not require backporting label Oct 2, 2023
@ymao1 ymao1 deleted the alerting/flattened-faad-docs branch October 2, 2023 18:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport:skip This commit does not require backporting Feature:Alerting/Alerts-as-Data Issues related to Alerts-as-data and RuleRegistry Feature:Alerting release_note:skip Skip the PR/issue when compiling release notes Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) v8.11.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Response Ops][Alerting] Should framework alerts as data write out data flattened keys
6 participants