Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cache query delay settings in the alerting task runner #184321

Closed
mikecote opened this issue May 27, 2024 · 2 comments · Fixed by #192404
Closed

Cache query delay settings in the alerting task runner #184321

mikecote opened this issue May 27, 2024 · 2 comments · Fixed by #192404
Assignees
@ymao1
Labels
Feature:Alerting Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams)

Comments

@mikecote
Copy link
Contributor

Similar to #149884, we shouldn't load the query delay settings on every rule run. Instead, we should cache the result for a period of time before going back to update the settings. This way running many rules in a short timeframe will not cause a large numbers of i/o to Elasticsearch to get the same value.
@mikecote mikecote added Feature:Alerting Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) labels May 27, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/response-ops (Team:ResponseOps)

@pmuellr
Copy link
Member

pmuellr commented May 30, 2024

I opened an issue a while back against the SO framework to provide a "cheap" exists operation - #148351 . Not really applicable directly to this issue, but perhaps there is something we could do at the SO level to help out here. An operation to only do the get if the doc has changed, otherwise return an indication it hasn't changed. Kinda thing.

If this were HTTP, we could use something like ETag and the If-None-Match header, to always make the request, but it's "fast" if the doc hasn't changed, and returns a status (with no body and a 304 Not Modified status). I don't think such things are possible in ES, but not sure.

@mikecote mikecote mentioned this issue Jul 3, 2024
Closed
@mikecote mikecote mentioned this issue Jul 12, 2024
Open
48 tasks
This was referenced Sep 10, 2024
Closed
Merged
ymao1 added a commit that referenced this issue Sep 16, 2024
@ymao1 @elasticmachine
[Response Ops][Alerting] Creating global service for fetching and cac…
7a0fe2e 
…hing rules settings (#192404)

Resolves #184321 and
#149884

## Summary

Created a `RulesSettingsService` that caches flapping and query delay
settings per space for 1 minute so rules are not accessing the SO index
with every execution.

Also moved all classes related to the rules settings into the
`rules_settings` folder so some of these changes are just renaming
changes.

## To Verify
1. Add some logging to
`x-pack/plugins/alerting/server/rules_settings/rules_settings_service.ts`
to indicate when settings are being fetched and when they're returning
from the cache.
2. Create and run some rules in different spaces to see that flapping
and query delay settings are being returned from the cache when the
cache has not expired.

---------

Co-authored-by: Elastic Machine <[email protected]>
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Sep 16, 2024
@ymao1
[Response Ops][Alerting] Creating global service for fetching and cac…
87805f3 
…hing rules settings (elastic#192404)

Resolves elastic#184321 and
elastic#149884

## Summary

Created a `RulesSettingsService` that caches flapping and query delay
settings per space for 1 minute so rules are not accessing the SO index
with every execution.

Also moved all classes related to the rules settings into the
`rules_settings` folder so some of these changes are just renaming
changes.

## To Verify
1. Add some logging to
`x-pack/plugins/alerting/server/rules_settings/rules_settings_service.ts`
to indicate when settings are being fetched and when they're returning
from the cache.
2. Create and run some rules in different spaces to see that flapping
and query delay settings are being returned from the cache when the
cache has not expired.

---------

Co-authored-by: Elastic Machine <[email protected]>
(cherry picked from commit 7a0fe2e)
kibanamachine added a commit that referenced this issue Sep 16, 2024
@kibanamachine @ymao1
[8.x] [Response Ops][Alerting] Creating global service for fetching a…
7939f22 
…nd caching rules settings (#192404) (#193011)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[Response Ops][Alerting] Creating global service for fetching and
caching rules settings
(#192404)](#192404)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Ying
Mao","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-09-16T13:22:44Z","message":"[Response
Ops][Alerting] Creating global service for fetching and caching rules
settings (#192404)\n\nResolves
#184321
and\r\nhttps://github.com//issues/149884\r\n\r\n##
Summary\r\n\r\nCreated a `RulesSettingsService` that caches flapping and
query delay\r\nsettings per space for 1 minute so rules are not
accessing the SO index\r\nwith every execution.\r\n\r\nAlso moved all
classes related to the rules settings into the\r\n`rules_settings`
folder so some of these changes are just renaming\r\nchanges.\r\n\r\n##
To Verify\r\n1. Add some logging
to\r\n`x-pack/plugins/alerting/server/rules_settings/rules_settings_service.ts`\r\nto
indicate when settings are being fetched and when they're
returning\r\nfrom the cache.\r\n2. Create and run some rules in
different spaces to see that flapping\r\nand query delay settings are
being returned from the cache when the\r\ncache has not
expired.\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine
<[email protected]>","sha":"7a0fe2e83683cf38c537694daa88a3a75bfe3e57","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Feature:Alerting","release_note:skip","Team:ResponseOps","v9.0.0","backport:prev-minor","v8.16.0"],"title":"[Response
Ops][Alerting] Creating global service for fetching and caching rules
settings","number":192404,"url":"https://github.com/elastic/kibana/pull/192404","mergeCommit":{"message":"[Response
Ops][Alerting] Creating global service for fetching and caching rules
settings (#192404)\n\nResolves
#184321
and\r\nhttps://github.com//issues/149884\r\n\r\n##
Summary\r\n\r\nCreated a `RulesSettingsService` that caches flapping and
query delay\r\nsettings per space for 1 minute so rules are not
accessing the SO index\r\nwith every execution.\r\n\r\nAlso moved all
classes related to the rules settings into the\r\n`rules_settings`
folder so some of these changes are just renaming\r\nchanges.\r\n\r\n##
To Verify\r\n1. Add some logging
to\r\n`x-pack/plugins/alerting/server/rules_settings/rules_settings_service.ts`\r\nto
indicate when settings are being fetched and when they're
returning\r\nfrom the cache.\r\n2. Create and run some rules in
different spaces to see that flapping\r\nand query delay settings are
being returned from the cache when the\r\ncache has not
expired.\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine
<[email protected]>","sha":"7a0fe2e83683cf38c537694daa88a3a75bfe3e57"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/192404","number":192404,"mergeCommit":{"message":"[Response
Ops][Alerting] Creating global service for fetching and caching rules
settings (#192404)\n\nResolves
#184321
and\r\nhttps://github.com//issues/149884\r\n\r\n##
Summary\r\n\r\nCreated a `RulesSettingsService` that caches flapping and
query delay\r\nsettings per space for 1 minute so rules are not
accessing the SO index\r\nwith every execution.\r\n\r\nAlso moved all
classes related to the rules settings into the\r\n`rules_settings`
folder so some of these changes are just renaming\r\nchanges.\r\n\r\n##
To Verify\r\n1. Add some logging
to\r\n`x-pack/plugins/alerting/server/rules_settings/rules_settings_service.ts`\r\nto
indicate when settings are being fetched and when they're
returning\r\nfrom the cache.\r\n2. Create and run some rules in
different spaces to see that flapping\r\nand query delay settings are
being returned from the cache when the\r\ncache has not
expired.\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine
<[email protected]>","sha":"7a0fe2e83683cf38c537694daa88a3a75bfe3e57"}},{"branch":"8.x","label":"v8.16.0","branchLabelMappingKey":"^v8.16.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

Co-authored-by: Ying Mao <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ymao1 ymao1

Labels
Feature:Alerting Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams)
Projects
None yet
Milestone
No milestone
4 participants
@pmuellr @mikecote @ymao1 @elasticmachine