[Response Ops][Alerting] Handle unflattened ES query docs #167583
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…-ref HEAD~1..HEAD --fix'
…:ymao1/kibana into alerting/handle-unflattened-esquery-docs
…-unflattened-esquery-docs
ymao1
changed the title
💛 Build succeeded, but was flaky
Failed CI StepsTest Failures
Metrics [docs]
History
To update your PR or re-run it, just comment with: cc @ymao1 |
ersin-erdal
approved these changes
Oct 2, 2023
ymao1
added a commit
that referenced
this pull request
Oct 2, 2023
…tened alerts docs (#167691) Resolves #166946 ## PRs to this feature branch * #167439 * #167583 ## Summary The rule registry has traditionally written out AAD docs with flattened keys, like ``` { "kibana.alert.rule.name": "test" } ``` The framework alerts client has been writing out AAD docs as objects, like ``` { "kibana": { "alert": { "rule": { "name": "test" } } } } ``` We've identified a few places where we're updating the docs where having this divergence makes things more difficult, so this is to switch the framework to writing flattened alert docs before onboarding more rule types. This PR is targeted for 8.11, which is also when we onboarded the index threshold rule type and the ML anomaly detection rule type to FAAD. For the ES query rule, which started writing unflattened AaD docs in 8.10, this PR adds special handling to ensure that those unflattened docs are correctly updated with flattened fields. ## To Verify ### ES Query and Index Threshold AaD Create these rules that trigger alerts and verify that their AaD docs are written out as flattened. For the ES Query rule type, select a Metrics/Logs consumer and verify that they appear on the O11y alerts table. ### ML alerts ML alerts added in #166349 looked like: <details> <summary>Unflattened</summary> ``` { "kibana": { "alert": { "url": "/app/ml/explorer/?_g=(ml%3A(jobIds%3A!(rt-anomaly-mean-value))%2Ctime%3A(from%3A'2023-09-28T14%3A57%3A00.000Z'%2Cmode%3Aabsolute%2Cto%3A'2023-09-28T15%3A17%3A00.000Z'))&_a=(explorer%3A(mlExplorerFilter%3A(filterActive%3A!t%2CfilteredFields%3A!(key%2Cthird-key)%2CinfluencersFilterQuery%3A(bool%3A(minimum_should_match%3A1%2Cshould%3A!((match_phrase%3A(key%3Athird-key)))))%2CqueryString%3A'key%3A%22third-key%22')%2CmlExplorerSwimlane%3A()))", "reason": "Alerts are raised based on real-time scores. Remember that scores may be adjusted over time as data continues to be analyzed.", "job_id": "rt-anomaly-mean-value", "anomaly_score": 73.63508175828011, "is_interim": false, "anomaly_timestamp": 1695913620000, "top_records": [{ "job_id": "rt-anomaly-mean-value", "record_score": 73.63516446528412, "initial_record_score": 73.63516446528412, "detector_index": 0, "is_interim": false, "timestamp": 1695913620000, "partition_field_name": "key", "partition_field_value": "third-key", "function": "mean", "actual": [ 3 ], "typical": [ 4.187715468532429 ] }], "top_influencers": [{ "job_id": "rt-anomaly-mean-value", "influencer_field_name": "key", "influencer_field_value": "third-key", "influencer_score": 73.63508175828011, "initial_influencer_score": 73.63508175828011, "is_interim": false, "timestamp": 1695913620000 }], "action_group": "anomaly_score_match", "flapping": false, "flapping_history": [ true, false, false, false ], "instance": { "id": "rt-anomaly-mean-value" }, "maintenance_window_ids": [], "rule": { "category": "Anomaly detection alert", "consumer": "alerts", "execution": { "uuid": "e9e681d4-c8e4-43eb-82e5-a58bdf7ffe12" }, "name": "rt-ad-alert-influencer", "parameters": { "severity": 5, "resultType": "influencer", "includeInterim": false, "jobSelection": { "jobIds": [ "rt-anomaly-mean-value" ], "groupIds": [] }, "lookbackInterval": null, "topNBuckets": null }, "producer": "ml", "revision": 0, "rule_type_id": "xpack.ml.anomaly_detection_alert", "tags": [], "uuid": "9e1d6bc0-5e10-11ee-8416-3bf48cca0922" }, "status": "active", "uuid": "c9c1f075-9985-4c55-8ff8-22349cb30269", "workflow_status": "open", "duration": { "us": "99021000000" }, "start": "2023-09-28T15:07:12.868Z", "time_range": { "gte": "2023-09-28T15:07:12.868Z" } }, "space_ids": [ "default" ], "version": "8.11.0" }, "@timestamp": "2023-09-28T15:08:51.889Z", "event": { "action": "active", "kind": "signal" }, "tags": [] } ``` </details> Now they look like: <details> <summary>Flattened</summary> ``` { "kibana.alert.url": "/app/ml/explorer/?_g=(ml%3A(jobIds%3A!(rt-anomaly-mean-value))%2Ctime%3A(from%3A'2023-09-28T15%3A03%3A00.000Z'%2Cmode%3Aabsolute%2Cto%3A'2023-09-28T15%3A23%3A00.000Z'))&_a=(explorer%3A(mlExplorerFilter%3A(filterActive%3A!t%2CfilteredFields%3A!(key%2Cthird-key)%2CinfluencersFilterQuery%3A(bool%3A(minimum_should_match%3A1%2Cshould%3A!((match_phrase%3A(key%3Athird-key)))))%2CqueryString%3A'key%3A%22third-key%22')%2CmlExplorerSwimlane%3A()))", "kibana.alert.reason": "Alerts are raised based on real-time scores. Remember that scores may be adjusted over time as data continues to be analyzed.", "kibana.alert.job_id": "rt-anomaly-mean-value", "kibana.alert.anomaly_score": 72.75515452061356, "kibana.alert.is_interim": false, "kibana.alert.anomaly_timestamp": 1695913980000, "kibana.alert.top_records": [{ "job_id": "rt-anomaly-mean-value", "record_score": 72.75515452061356, "initial_record_score": 72.75515452061356, "detector_index": 0, "is_interim": false, "timestamp": 1695913980000, "partition_field_name": "key", "partition_field_value": "third-key", "function": "mean", "actual": [ 0.5 ], "typical": [ 4.138745343296527 ] }], "kibana.alert.top_influencers": [{ "job_id": "rt-anomaly-mean-value", "influencer_field_name": "key", "influencer_field_value": "third-key", "influencer_score": 72.75515452061356, "initial_influencer_score": 72.75515452061356, "is_interim": false, "timestamp": 1695913980000 }], "kibana.alert.rule.category": "Anomaly detection alert", "kibana.alert.rule.consumer": "alerts", "kibana.alert.rule.execution.uuid": "17fef3d3-d595-4362-837e-b2a73650169e", "kibana.alert.rule.name": "rt-ad-alert-influencer", "kibana.alert.rule.parameters": { "severity": 5, "resultType": "influencer", "includeInterim": false, "jobSelection": { "jobIds": [ "rt-anomaly-mean-value" ], "groupIds": [] }, "lookbackInterval": null, "topNBuckets": null }, "kibana.alert.rule.producer": "ml", "kibana.alert.rule.revision": 0, "kibana.alert.rule.rule_type_id": "xpack.ml.anomaly_detection_alert", "kibana.alert.rule.tags": [], "kibana.alert.rule.uuid": "757c7610-5e11-11ee-8bc6-a95c3ced4757", "kibana.space_ids": [ "default" ], "@timestamp": "2023-09-28T15:14:52.057Z", "event.action": "active", "event.kind": "signal", "kibana.alert.action_group": "anomaly_score_match", "kibana.alert.flapping": false, "kibana.alert.flapping_history": [ true, false, false, false ], "kibana.alert.instance.id": "rt-anomaly-mean-value", "kibana.alert.maintenance_window_ids": [], "kibana.alert.status": "active", "kibana.alert.uuid": "ac1f0d7c-461b-4fc6-b4c3-04416ac876d3", "kibana.alert.workflow_status": "open", "kibana.alert.duration.us": "99028000000", "kibana.alert.start": "2023-09-28T15:13:13.028Z", "kibana.alert.time_range": { "gte": "2023-09-28T15:13:13.028Z" }, "kibana.version": "8.11.0", "tags": [] } ``` </details> --------- Co-authored-by: kibanamachine <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is targeting a feature branch
Summary
Since we are switching to writing out flattened AaD docs, we may encounter a situation where existing ES query AaD docs are written out and need to be updated after upgrade. This PR tries to handle this case and update those docs so no duplicate data (flattened & unflattened) is written to the doc.
To Verify
On
main, create an ES query rule that writes an (unflattened) alert doc. Then switch to this branch and let the rule run so that the alert is updated (either updated as ongoing or set as recovered). Inspect the doc and verify that while the doc might be a mix of flattened & expanded keys, there is no duplicate data.Updating a "new" alert
Unflattened "new" alert
After update to "active"
Updating an "active" alert
Unflattened "active" alert
After update to "recovered"
Updating a "recovered" alert
Unflattened "recovered" alert
After another update to "recovered"