Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bulk enable/disable Kibana Rules #116017

Open
stefnestor opened this issue Oct 21, 2021 · 7 comments
Open

Bulk enable/disable Kibana Rules #116017

stefnestor opened this issue Oct 21, 2021 · 7 comments
Labels
enhancement New value added to drive a business result estimate:needs-research Estimated as too large and requires research to break down into workable issues Feature:Alerting/RulesFramework Issues related to the Alerting Rules Framework Feature:Alerting/RulesManagement Issues related to the Rules Management UX Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams)

Comments

@stefnestor
Copy link
Contributor

stefnestor commented Oct 21, 2021

Hello, team! In troubleshooting Kibana Alerts we've ran into the corner scenario:

  1. Enable all prebuilt detection rules
  2. Oops, a bit later I realize I haven't sized Kibana to be able to handle this load
  3. [WANT] Disable all rules (not just new) so can review & enable the ones

Currently, you have to restart Kibana after setting temporary settings

  • xpack.task_manager.max_workers: 1
  • xpack.task_manager.poll_interval: 1m

Alternatively you can completely disable alerts xpack.alerts.enabled: false (not sure if you can still see them in the UI after this, though).

Then, in UI you have to go page-by-page of rules/alerts and

  1. highlight all & mark disable

However, this can be highly annoying if you have a lot of rules. Desire is to be able to run a Kibana APi like POST /api/alerting/rule/_all/_disable to do all at once. Alternatively, do an update_by_query like command to disable all alerts of certain type, or w/name string, or created recently, etc. (@gmmorris - striked out due to this: #116017 (comment))

@stefnestor stefnestor added enhancement New value added to drive a business result Feature:Alerting Feature:Detection Alerts Security Solution Detection Alerts Feature labels Oct 21, 2021
@botelastic botelastic bot added the needs-team Issues missing a team label label Oct 21, 2021
@stefnestor
Copy link
Contributor Author

cc: @mikecote @pmuellr @chrisronline

@stefnestor stefnestor added the Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) label Oct 21, 2021
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

@botelastic botelastic bot removed the needs-team Issues missing a team label label Oct 21, 2021
@pmuellr pmuellr added the Feature:Alerting/RulesManagement Issues related to the Rules Management UX label Oct 21, 2021
@pmuellr
Copy link
Member

pmuellr commented Oct 21, 2021

For this scenario, it would be nice if you could just stop alerting, rather than bulk disable the rules. Right now, if you disable the rules, they will need to be specifically enabled - and the user that enables them will end up having an API key generated with their privileges, associated with them. So, while a "bulk disable" would be nice, trying to re-enable these would be pretty painful, I think.

Even to bulk disable all of the rules today, you'd have to do it per space.

It really feels like it's time to have some kind of "rule administrator" role, which we could use to gate some "global" activities like this.

In lieu of any of that, perhaps just a config key, that indicated that none of the rules should actually run, would be good. We'd just check that config in the rule task runner, and not actually run the executors. And provide some UX in the rule pages to indicate "rules are not running because of the config key" kind of thing.

@stefnestor
Copy link
Contributor Author

cc: @bsturg

@gmmorris gmmorris added estimate:needs-research Estimated as too large and requires research to break down into workable issues Feature:Alerting/RulesFramework Issues related to the Alerting Rules Framework and removed Feature:Alerting Feature:Detection Alerts Security Solution Detection Alerts Feature labels Oct 27, 2021
@gmmorris
Copy link
Contributor

gmmorris commented Oct 27, 2021

Alternatively, do an update_by_query like command to disable all alerts of certain type, or w/name string, or created recently, etc.

TBC this is not an option as it will corrupt all these rules due to our use of encrypted saved objects (we rely on AAD to prevent users from tampering with SOs).
I'd hate for someone to see this in the issue and try it. 😬

@gmmorris gmmorris changed the title Bulk enable/disable Kibana Alerts Bulk enable/disable Kibana Rules Oct 27, 2021
@stefnestor
Copy link
Contributor Author

Cross-pasting from #90934, #111036 -- in 8.0 you can no longer set xpack.alerts.enabled: false so instead will set xpack.task_manager.internal.exclude_task_types: ['alerting:*'] (starting v7.16.0, not in docs).

stefnestor added a commit that referenced this issue Jan 10, 2022
🙏🏼 per #116017, adds insight on how to temporarily disable Kibana Rules for clusters which need breathing room.
@stefnestor stefnestor reopened this Jan 10, 2022
@stefnestor
Copy link
Contributor Author

miss click sorry 😂

@botelastic botelastic bot removed the needs-team Issues missing a team label label Jan 31, 2022
stefnestor added a commit that referenced this issue Mar 3, 2022
👋🏼 @gchaps asked me to file a new PR since my last #122573 got too far behind.

Summary

🙏🏼 per #116017, adds insight on how to temporarily disable Kibana Rules for clusters which need breathing room. cc: @pmuellr @lcawl @gmmorris
stefnestor added a commit that referenced this issue Jul 13, 2023
👋🏼 @gchaps asked me to file a new PR since my last
#122573 got too far behind.

## Summary

🙏🏼 per #116017, adds insight on how to temporarily disable Kibana Rules
for clusters which need breathing room.

---------

Co-authored-by: Kibana Machine <[email protected]>
Co-authored-by: Lisa Cawley <[email protected]>
kibanamachine pushed a commit that referenced this issue Jul 13, 2023
👋🏼 @gchaps asked me to file a new PR since my last
#122573 got too far behind.

## Summary

🙏🏼 per #116017, adds insight on how to temporarily disable Kibana Rules
for clusters which need breathing room.

---------

Co-authored-by: Kibana Machine <[email protected]>
Co-authored-by: Lisa Cawley <[email protected]>
(cherry picked from commit b1d6196)
kibanamachine added a commit that referenced this issue Jul 13, 2023
# Backport

This will backport the following commits from `main` to `8.9`:
- [[DOCv2] Temporarily disable Kibana Rules
(#126869)](#126869)

<!--- Backport version: 8.9.7 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Stef
Nestor","email":"[email protected]"},"sourceCommit":{"committedDate":"2023-07-13T13:22:55Z","message":"[DOCv2]
Temporarily disable Kibana Rules (#126869)\n\n👋🏼 @gchaps asked me to
file a new PR since my
last\r\nhttps://github.com//pull/122573 got too far
behind.\r\n\r\n## Summary\r\n\r\n🙏🏼 per #116017, adds insight on how to
temporarily disable Kibana Rules\r\nfor clusters which need breathing
room.\r\n\r\n---------\r\n\r\nCo-authored-by: Kibana Machine
<[email protected]>\r\nCo-authored-by:
Lisa Cawley
<[email protected]>","sha":"b1d619617a0321617636c7c1bbcbf74e393a5d9e","branchLabelMapping":{"^v8.10.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Docs","release_note:skip","docs","auto-backport","Feature:Alerting/RulesManagement","v8.9.0","v8.10.0"],"number":126869,"url":"https://github.com/elastic/kibana/pull/126869","mergeCommit":{"message":"[DOCv2]
Temporarily disable Kibana Rules (#126869)\n\n👋🏼 @gchaps asked me to
file a new PR since my
last\r\nhttps://github.com//pull/122573 got too far
behind.\r\n\r\n## Summary\r\n\r\n🙏🏼 per #116017, adds insight on how to
temporarily disable Kibana Rules\r\nfor clusters which need breathing
room.\r\n\r\n---------\r\n\r\nCo-authored-by: Kibana Machine
<[email protected]>\r\nCo-authored-by:
Lisa Cawley
<[email protected]>","sha":"b1d619617a0321617636c7c1bbcbf74e393a5d9e"}},"sourceBranch":"main","suggestedTargetBranches":["8.9"],"targetPullRequestStates":[{"branch":"8.9","label":"v8.9.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.10.0","labelRegex":"^v8.10.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/126869","number":126869,"mergeCommit":{"message":"[DOCv2]
Temporarily disable Kibana Rules (#126869)\n\n👋🏼 @gchaps asked me to
file a new PR since my
last\r\nhttps://github.com//pull/122573 got too far
behind.\r\n\r\n## Summary\r\n\r\n🙏🏼 per #116017, adds insight on how to
temporarily disable Kibana Rules\r\nfor clusters which need breathing
room.\r\n\r\n---------\r\n\r\nCo-authored-by: Kibana Machine
<[email protected]>\r\nCo-authored-by:
Lisa Cawley
<[email protected]>","sha":"b1d619617a0321617636c7c1bbcbf74e393a5d9e"}}]}]
BACKPORT-->

Co-authored-by: Stef Nestor <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New value added to drive a business result estimate:needs-research Estimated as too large and requires research to break down into workable issues Feature:Alerting/RulesFramework Issues related to the Alerting Rules Framework Feature:Alerting/RulesManagement Issues related to the Rules Management UX Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams)
Projects
No open projects
Development

No branches or pull requests

5 participants