Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Synthetics] Alerts do not fire for monitors created in earlier versions #156223

Closed
dominiqueclarke opened this issue Apr 28, 2023 · 2 comments · Fixed by #156432
Closed

[Synthetics] Alerts do not fire for monitors created in earlier versions #156223

dominiqueclarke opened this issue Apr 28, 2023 · 2 comments · Fixed by #156432
Assignees
@dominiqueclarke
Labels
bug Fixes for quality problems that affect the customer experience Team:Uptime - DEPRECATED Synthetics & RUM sub-team of Application Observability v8.8.0

Comments

@dominiqueclarke
Copy link
Contributor

dominiqueclarke commented Apr 28, 2023 •
edited
Loading

Kibana version: 8.8.0

Description of the problem including expected versus actual behavior:

Alerts are not firing for monitors created in earlier versions of the stack, such as 8.5.3

Screen Shot 2023-04-28 at 4 06 26 PM

Even though the latest alert ran after the timestamp of the failed check, it seems the rule didn't trigger
Screen Shot 2023-04-28 at 4 06 01 PM
@dominiqueclarke dominiqueclarke added bug Fixes for quality problems that affect the customer experience Team:Uptime - DEPRECATED Synthetics & RUM sub-team of Application Observability v8.8.0 labels Apr 28, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/uptime (Team:uptime)

@dominiqueclarke
Copy link
Contributor Author

dominiqueclarke commented Apr 29, 2023 •
edited
Loading

I've investigated this further and this isn't a timing issue. It's actually a migration issue.

Monitors created in earlier versions may not have the label for the location saved in the saved object. The alerting logic relies on this label.

We can't actually write a saved object migration for adding the location label, because it would require us to fetch the information from the location manifest and async operations aren't supported in saved object migrations.

We can move towards relying on the id instead of the label for alerting, since the id should be available since adding the run_from field to heartbeat.

@dominiqueclarke dominiqueclarke changed the title [Synthetics] Alert timing is off [Synthetics] Alerts do not fire for monitors created in earlier versions Apr 29, 2023
@dominiqueclarke dominiqueclarke self-assigned this May 1, 2023
@dominiqueclarke dominiqueclarke mentioned this issue May 2, 2023
Merged
dominiqueclarke added a commit that referenced this issue May 3, 2023
@dominiqueclarke @shahzad31
[Synthetics] adjust monitor status rule location logic (#156432)
77cebee 
## Summary

Resolves #156223

In previous stack versions, the label was not saved to the saved object.

This ensures that alerting works both for those early versions as well
current versions, by adjusting the logic to consider both the
location.label and location.id in alerting logic.

### Testing

1. Create a new oblt-cli cluster and connect to the synthetics service,
locally or via the dev environment
2. Checkout v8.5.3
3. Create a monitor. Make sure it's down and wait for the first down
check
4. Check out this branch
5. Navigate to Observabillity alerts. Ensure that an alert is generated
for the monitor. You may have to wait a few minutes.

---------

Co-authored-by: Shahzad <[email protected]>
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue May 3, 2023
@dominiqueclarke
[Synthetics] adjust monitor status rule location logic (elastic#156432)
3671afa 
## Summary

Resolves elastic#156223

In previous stack versions, the label was not saved to the saved object.

This ensures that alerting works both for those early versions as well
current versions, by adjusting the logic to consider both the
location.label and location.id in alerting logic.

### Testing

1. Create a new oblt-cli cluster and connect to the synthetics service,
locally or via the dev environment
2. Checkout v8.5.3
3. Create a monitor. Make sure it's down and wait for the first down
check
4. Check out this branch
5. Navigate to Observabillity alerts. Ensure that an alert is generated
for the monitor. You may have to wait a few minutes.

---------

Co-authored-by: Shahzad <[email protected]>
(cherry picked from commit 77cebee)
kibanamachine referenced this issue May 3, 2023
@kibanamachine @dominiqueclarke
[8.8] [Synthetics] adjust monitor status rule location logic (#156432) (
e204707 
#156546)

# Backport

This will backport the following commits from `main` to `8.8`:
- [[Synthetics] adjust monitor status rule location logic
(#156432)](#156432)

<!--- Backport version: 8.9.7 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Dominique
Clarke","email":"[email protected]"},"sourceCommit":{"committedDate":"2023-05-03T13:41:19Z","message":"[Synthetics]
adjust monitor status rule location logic (#156432)\n\n##
Summary\r\n\r\nResolves
https://github.com/elastic/kibana/issues/156223\r\n\r\nIn previous stack
versions, the label was not saved to the saved object.\r\n\r\nThis
ensures that alerting works both for those early versions as
well\r\ncurrent versions, by adjusting the logic to consider both
the\r\nlocation.label and location.id in alerting logic.\r\n\r\n###
Testing\r\n\r\n1. Create a new oblt-cli cluster and connect to the
synthetics service,\r\nlocally or via the dev environment\r\n2. Checkout
v8.5.3\r\n3. Create a monitor. Make sure it's down and wait for the
first down\r\ncheck\r\n4. Check out this branch\r\n5. Navigate to
Observabillity alerts. Ensure that an alert is generated\r\nfor the
monitor. You may have to wait a few
minutes.\r\n\r\n---------\r\n\r\nCo-authored-by: Shahzad
<[email protected]>","sha":"77cebee6f754d8bbd439008f8c8466bceebd54ac","branchLabelMapping":{"^v8.9.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:uptime","release_note:skip","v8.8.0","v8.9.0"],"number":156432,"url":"https://github.com/elastic/kibana/pull/156432","mergeCommit":{"message":"[Synthetics]
adjust monitor status rule location logic (#156432)\n\n##
Summary\r\n\r\nResolves
https://github.com/elastic/kibana/issues/156223\r\n\r\nIn previous stack
versions, the label was not saved to the saved object.\r\n\r\nThis
ensures that alerting works both for those early versions as
well\r\ncurrent versions, by adjusting the logic to consider both
the\r\nlocation.label and location.id in alerting logic.\r\n\r\n###
Testing\r\n\r\n1. Create a new oblt-cli cluster and connect to the
synthetics service,\r\nlocally or via the dev environment\r\n2. Checkout
v8.5.3\r\n3. Create a monitor. Make sure it's down and wait for the
first down\r\ncheck\r\n4. Check out this branch\r\n5. Navigate to
Observabillity alerts. Ensure that an alert is generated\r\nfor the
monitor. You may have to wait a few
minutes.\r\n\r\n---------\r\n\r\nCo-authored-by: Shahzad
<[email protected]>","sha":"77cebee6f754d8bbd439008f8c8466bceebd54ac"}},"sourceBranch":"main","suggestedTargetBranches":["8.8"],"targetPullRequestStates":[{"branch":"8.8","label":"v8.8.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.9.0","labelRegex":"^v8.9.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/156432","number":156432,"mergeCommit":{"message":"[Synthetics]
adjust monitor status rule location logic (#156432)\n\n##
Summary\r\n\r\nResolves
https://github.com/elastic/kibana/issues/156223\r\n\r\nIn previous stack
versions, the label was not saved to the saved object.\r\n\r\nThis
ensures that alerting works both for those early versions as
well\r\ncurrent versions, by adjusting the logic to consider both
the\r\nlocation.label and location.id in alerting logic.\r\n\r\n###
Testing\r\n\r\n1. Create a new oblt-cli cluster and connect to the
synthetics service,\r\nlocally or via the dev environment\r\n2. Checkout
v8.5.3\r\n3. Create a monitor. Make sure it's down and wait for the
first down\r\ncheck\r\n4. Check out this branch\r\n5. Navigate to
Observabillity alerts. Ensure that an alert is generated\r\nfor the
monitor. You may have to wait a few
minutes.\r\n\r\n---------\r\n\r\nCo-authored-by: Shahzad
<[email protected]>","sha":"77cebee6f754d8bbd439008f8c8466bceebd54ac"}}]}]
BACKPORT-->

Co-authored-by: Dominique Clarke <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dominiqueclarke dominiqueclarke

Labels
bug Fixes for quality problems that affect the customer experience Team:Uptime - DEPRECATED Synthetics & RUM sub-team of Application Observability v8.8.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Synthetics] adjust monitor status rule location logic dominiqueclarke/kibana
2 participants
@dominiqueclarke @elasticmachine