-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Mixin: Update sidecar alert #2002
Conversation
@daixiang0 Would you maybe consider implementing alerts and dashboards for sidecar in |
I would like to do this :) |
Super, cool @daixiang0. Ping me if you need anything on slack :) |
@kakkoyun thanks, i would file another pr for that. |
If you gonna create that PR, you will probably automate the generation of the alert examples so maybe we should close this one and focus on the upcoming PR. @daixiang0 WDYT? |
It would need much time to finish while this useless alert may make users confused i think. |
20320ff
to
ed4aa47
Compare
@kakkoyun please review. |
Signed-off-by: Xiang Dai <[email protected]>
Signed-off-by: Xiang Dai <[email protected]>
ed4aa47
to
d1fab0b
Compare
Signed-off-by: Xiang Dai <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looking good. Besides my inline comments, we aim to generate examples in https://github.com/thanos-io/thanos/blob/master/examples/alerts/alerts.md file, so check
thanos/examples/alerts/alerts.md
Lines 239 to 240 in 56a1fb6
[//]: # "TODO(kakkoyun): Generate sidecar rules using thanos-mixin." | |
<!-- [embedmd]:# (../tmp/thanos-sidecar.rules.yaml yaml) --> |
If you uncomment and make sure the path is correct it'd embed generated file for the documentation.
Better to be consistent and generate examples for
sidecar
like the others.
Signed-off-by: Xiang Dai <[email protected]>
@kakkoyun ping |
impact: We will lose metrics data if not fixed in 24h | ||
action: Check {{ $labels.kubernetes_pod_name }} pod logs in {{ $labels.kubernetes_namespace}} namespace | ||
dashboard: SIDECAR_URL | ||
- alert: ThanosSidecarGrpcErrorRate |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@daixiang0 Let's not lose this alert, we have similar more up-to-date ones in other components' alerts.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Then we're good to merge.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The impact and action part are only seen in this alert, is it necessary to add for all component alert?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, I'd say it's not necessary. At this stage, these are just examples to start from.
Who is generating this? will |
@bwplotka it's Lines 335 to 337 in 56a1fb6
|
Signed-off-by: Xiang Dai [email protected]
Fix #544