Alert when mimir components are restarting too often accross all pipe… #1093
Conversation
…lines to avoid high storage cost
| description: '{{`Mimir containers are restarting too often.`}}' | ||
| expr: | | ||
| increase( | ||
| kube_pod_container_status_restarts_total{cluster_type="management_cluster", namespace="mimir"}[1h] |
There was a problem hiding this comment.
I just executed that rules on grizzly and it will be paged because of prometheus-buddy restarting a lot...
There was a problem hiding this comment.
Then that's something we have to fix :)
There was a problem hiding this comment.
Can we scope this alert to pods requested S3 only ?
There was a problem hiding this comment.
I removed the buddy out of the equation and created a PM to investigate what is happening https://github.com/giantswarm/giantswarm/issues/30403
There was a problem hiding this comment.
Considering this is a BH only alert (at least now), I would prefer that we keep it too broad if we add new components with s3 access for now and reduce the scope later on but I'm open to suggestions from @hervenicol and @QuantumEnigmaa
| @@ -86,3 +82,28 @@ tests: | |||
| description: "Mimir ruler is failing to process PrometheusRules." | |||
| - alertname: MimirRulerEventsFailed | |||
| eval_time: 160m | |||
| - interval: 1m | |||
| input_series: | |||
| - series: 'kube_pod_container_status_restarts_total{cluster_type="management_cluster", namespace="mimir"}' | |||
There was a problem hiding this comment.
| - series: 'kube_pod_container_status_restarts_total{cluster_type="management_cluster", namespace="mimir"}' | |
| - series: 'kube_pod_container_status_restarts_total{cluster_type="management_cluster", namespace="mimir", container!="prometheus"}' |
There was a problem hiding this comment.
I think we need to set a container that is not prometheus 😅
There was a problem hiding this comment.
But then we probably need another input with container=prometheus
There was a problem hiding this comment.
It's fixed in later version :)
QuentinBisson
force-pushed
the
add-cloud-protection-alert-for-mimir
branch
from
7752ee2 to
8632b4d
Compare
…lines to avoid high storage cost
Before adding a new alerting rule into this repository you should consider creating an SLO rules instead.
SLO helps you both increase the quality of your monitoring and reduce the alert noise.
Towards: #1090
This PR duplicates #1090 for mimir components
Checklist
oncall-kaas-cloudGitHub group).