Skip to content

Commit

Permalink
Update Alertmanager notification template (#1764)
Browse files Browse the repository at this point in the history
* Update Alertmanager notification template

- Update alert and query URLs for Mimir to point to the Active notification rather than the rule page
- Move link section (runbook, dashboard, explors) before instance to avoid them being lost due to OpsGenie max description being reached
- Move the warnings for missing runbook and dashboard up into the link section
  • Loading branch information
TheoBrigitte authored Dec 17, 2024
1 parent 4a2d8f7 commit c13636e
Show file tree
Hide file tree
Showing 18 changed files with 196 additions and 205 deletions.
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Changed

- Update Alertmanager notification template
- Update alert and query URLs for Mimir to point to the Active notification rather than the rule page
- Move link section (runbook, dashboard, explors) before instance to avoid them being lost due to OpsGenie max description being reached
- Move the warnings for missing runbook and dashboard up into the link section


### Removed

- Get rid of useless `prometheus-agent` after the migration to the new `monitoring-agent` inhibitions.
Expand Down
25 changes: 12 additions & 13 deletions files/templates/alertmanager/notification-template.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

{{ define "__alerturl" }}
[[- if .MimirEnabled -]]
[[ .GrafanaAddress ]]/alerting/Mimir/{{ .CommonLabels.alertname }}/find
[[ .GrafanaAddress ]]/alerting/groups?queryString=alertname%3D%22{{ .CommonLabels.alertname}}%22&alertmanager=Alertmanager
[[- else -]]
{{ .ExternalURL }}/#/alerts?receiver={{ .Receiver }}&silenced=false&inhibited=false&active=true&filter=%7Balertname%3D%22{{ .CommonLabels.alertname }}%22%7D
[[- end -]]
Expand All @@ -13,7 +13,7 @@

{{ define "__queryurl" }}
[[- if .MimirEnabled -]]
[[ .GrafanaAddress ]]/alerting/Mimir/{{ .CommonLabels.alertname }}/find
[[ .GrafanaAddress ]]/alerting/groups?queryString=alertname%3D%22{{ .CommonLabels.alertname}}%22&alertmanager=Alertmanager
[[- else -]]
{{ (index .Alerts 0).GeneratorURL }}
[[- end -]]
Expand Down Expand Up @@ -42,19 +42,16 @@

{{ define "opsgenie.default.message" }}{{ .GroupLabels.installation }} / {{ .GroupLabels.cluster_id }}{{ if (index .Alerts 0).Labels.service }} / {{ (index .Alerts 0).Labels.service }}{{ end }} - {{ index (index .Alerts.Firing 0).Labels `alertname`}}{{ end }}
{{ define "opsgenie.default.source" }}{{ template "__alertmanager" . }}{{ end }}
{{ define "opsgenie.default.description" }}* Team: {{ (index .Alerts 0).Labels.team }}
* Area: {{ (index .Alerts 0).Labels.area }} / {{ (index .Alerts 0).Labels.topic }}

* Instances:{{ range .Alerts.Firing }}
🔥 {{ if .Labels.instance }}{{ .Labels.instance }}: {{ end }}{{ .Annotations.description }}{{ end }}

---

{{ define "opsgenie.default.description" }}
{{ if (index .Alerts 0).Annotations.opsrecipe -}}
📗 Runbook: {{ template "__runbookurl" . }}
{{ end -}}
{{ else -}}
📗 Runbook: ⚠️ There is no **runbook** for this alert, time to get your pen.
{{- end }}
{{ if (index .Alerts 0).Annotations.dashboard -}}
📈 Dashboard: {{ template "__dashboardurl" . }}
{{ else -}}
📈 Dashboard: ⚠️ There is no **dashboard** for this alert, time to sketch.
{{ end -}}
[[- if .MimirEnabled ]]
👀 Explore: {{ template "__queryurl" . }}
Expand All @@ -65,8 +62,10 @@

---

{{ if not (index .Alerts 0).Annotations.opsrecipe }}⚠️ There is no **runbook** for this alert, time to get your pen.{{- end }}
{{ if not (index .Alerts 0).Annotations.dashboard }}⚠️ There is no **dashboard** for this alert, time to sketch.{{- end }}
* Team: {{ (index .Alerts 0).Labels.team }}
* Area: {{ (index .Alerts 0).Labels.area }} / {{ (index .Alerts 0).Labels.topic }}
* Instances:{{ range .Alerts.Firing }}
🔥 {{ if .Labels.instance }}{{ .Labels.instance }}: {{ end }}{{ .Annotations.description }}{{ end }}
{{- end }}

# This builds the silence URL. We exclude the alertname in the range
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,27 +30,26 @@

{{ define "opsgenie.default.message" }}{{ .GroupLabels.installation }} / {{ .GroupLabels.cluster_id }}{{ if (index .Alerts 0).Labels.service }} / {{ (index .Alerts 0).Labels.service }}{{ end }} - {{ index (index .Alerts.Firing 0).Labels `alertname`}}{{ end }}
{{ define "opsgenie.default.source" }}{{ template "__alertmanager" . }}{{ end }}
{{ define "opsgenie.default.description" }}* Team: {{ (index .Alerts 0).Labels.team }}
* Area: {{ (index .Alerts 0).Labels.area }} / {{ (index .Alerts 0).Labels.topic }}

* Instances:{{ range .Alerts.Firing }}
🔥 {{ if .Labels.instance }}{{ .Labels.instance }}: {{ end }}{{ .Annotations.description }}{{ end }}

---

{{ define "opsgenie.default.description" }}
{{ if (index .Alerts 0).Annotations.opsrecipe -}}
📗 Runbook: {{ template "__runbookurl" . }}
{{ end -}}
{{ else -}}
📗 Runbook: ⚠️ There is no **runbook** for this alert, time to get your pen.
{{- end }}
{{ if (index .Alerts 0).Annotations.dashboard -}}
📈 Dashboard: {{ template "__dashboardurl" . }}
{{ else -}}
📈 Dashboard: ⚠️ There is no **dashboard** for this alert, time to sketch.
{{ end -}}
🔔 Alertmanager {{ template "__alerturl" . }}
👀 Query: {{ template "__queryurl" . }}

---

{{ if not (index .Alerts 0).Annotations.opsrecipe }}⚠️ There is no **runbook** for this alert, time to get your pen.{{- end }}
{{ if not (index .Alerts 0).Annotations.dashboard }}⚠️ There is no **dashboard** for this alert, time to sketch.{{- end }}
* Team: {{ (index .Alerts 0).Labels.team }}
* Area: {{ (index .Alerts 0).Labels.area }} / {{ (index .Alerts 0).Labels.topic }}
* Instances:{{ range .Alerts.Firing }}
🔥 {{ if .Labels.instance }}{{ .Labels.instance }}: {{ end }}{{ .Annotations.description }}{{ end }}
{{- end }}

# This builds the silence URL. We exclude the alertname in the range
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,27 +30,26 @@

{{ define "opsgenie.default.message" }}{{ .GroupLabels.installation }} / {{ .GroupLabels.cluster_id }}{{ if (index .Alerts 0).Labels.service }} / {{ (index .Alerts 0).Labels.service }}{{ end }} - {{ index (index .Alerts.Firing 0).Labels `alertname`}}{{ end }}
{{ define "opsgenie.default.source" }}{{ template "__alertmanager" . }}{{ end }}
{{ define "opsgenie.default.description" }}* Team: {{ (index .Alerts 0).Labels.team }}
* Area: {{ (index .Alerts 0).Labels.area }} / {{ (index .Alerts 0).Labels.topic }}

* Instances:{{ range .Alerts.Firing }}
🔥 {{ if .Labels.instance }}{{ .Labels.instance }}: {{ end }}{{ .Annotations.description }}{{ end }}

---

{{ define "opsgenie.default.description" }}
{{ if (index .Alerts 0).Annotations.opsrecipe -}}
📗 Runbook: {{ template "__runbookurl" . }}
{{ end -}}
{{ else -}}
📗 Runbook: ⚠️ There is no **runbook** for this alert, time to get your pen.
{{- end }}
{{ if (index .Alerts 0).Annotations.dashboard -}}
📈 Dashboard: {{ template "__dashboardurl" . }}
{{ else -}}
📈 Dashboard: ⚠️ There is no **dashboard** for this alert, time to sketch.
{{ end -}}
🔔 Alertmanager {{ template "__alerturl" . }}
👀 Query: {{ template "__queryurl" . }}

---

{{ if not (index .Alerts 0).Annotations.opsrecipe }}⚠️ There is no **runbook** for this alert, time to get your pen.{{- end }}
{{ if not (index .Alerts 0).Annotations.dashboard }}⚠️ There is no **dashboard** for this alert, time to sketch.{{- end }}
* Team: {{ (index .Alerts 0).Labels.team }}
* Area: {{ (index .Alerts 0).Labels.area }} / {{ (index .Alerts 0).Labels.topic }}
* Instances:{{ range .Alerts.Firing }}
🔥 {{ if .Labels.instance }}{{ .Labels.instance }}: {{ end }}{{ .Annotations.description }}{{ end }}
{{- end }}

# This builds the silence URL. We exclude the alertname in the range
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,27 +30,26 @@

{{ define "opsgenie.default.message" }}{{ .GroupLabels.installation }} / {{ .GroupLabels.cluster_id }}{{ if (index .Alerts 0).Labels.service }} / {{ (index .Alerts 0).Labels.service }}{{ end }} - {{ index (index .Alerts.Firing 0).Labels `alertname`}}{{ end }}
{{ define "opsgenie.default.source" }}{{ template "__alertmanager" . }}{{ end }}
{{ define "opsgenie.default.description" }}* Team: {{ (index .Alerts 0).Labels.team }}
* Area: {{ (index .Alerts 0).Labels.area }} / {{ (index .Alerts 0).Labels.topic }}

* Instances:{{ range .Alerts.Firing }}
🔥 {{ if .Labels.instance }}{{ .Labels.instance }}: {{ end }}{{ .Annotations.description }}{{ end }}

---

{{ define "opsgenie.default.description" }}
{{ if (index .Alerts 0).Annotations.opsrecipe -}}
📗 Runbook: {{ template "__runbookurl" . }}
{{ end -}}
{{ else -}}
📗 Runbook: ⚠️ There is no **runbook** for this alert, time to get your pen.
{{- end }}
{{ if (index .Alerts 0).Annotations.dashboard -}}
📈 Dashboard: {{ template "__dashboardurl" . }}
{{ else -}}
📈 Dashboard: ⚠️ There is no **dashboard** for this alert, time to sketch.
{{ end -}}
🔔 Alertmanager {{ template "__alerturl" . }}
👀 Query: {{ template "__queryurl" . }}

---

{{ if not (index .Alerts 0).Annotations.opsrecipe }}⚠️ There is no **runbook** for this alert, time to get your pen.{{- end }}
{{ if not (index .Alerts 0).Annotations.dashboard }}⚠️ There is no **dashboard** for this alert, time to sketch.{{- end }}
* Team: {{ (index .Alerts 0).Labels.team }}
* Area: {{ (index .Alerts 0).Labels.area }} / {{ (index .Alerts 0).Labels.topic }}
* Instances:{{ range .Alerts.Firing }}
🔥 {{ if .Labels.instance }}{{ .Labels.instance }}: {{ end }}{{ .Annotations.description }}{{ end }}
{{- end }}

# This builds the silence URL. We exclude the alertname in the range
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,27 +30,26 @@

{{ define "opsgenie.default.message" }}{{ .GroupLabels.installation }} / {{ .GroupLabels.cluster_id }}{{ if (index .Alerts 0).Labels.service }} / {{ (index .Alerts 0).Labels.service }}{{ end }} - {{ index (index .Alerts.Firing 0).Labels `alertname`}}{{ end }}
{{ define "opsgenie.default.source" }}{{ template "__alertmanager" . }}{{ end }}
{{ define "opsgenie.default.description" }}* Team: {{ (index .Alerts 0).Labels.team }}
* Area: {{ (index .Alerts 0).Labels.area }} / {{ (index .Alerts 0).Labels.topic }}

* Instances:{{ range .Alerts.Firing }}
🔥 {{ if .Labels.instance }}{{ .Labels.instance }}: {{ end }}{{ .Annotations.description }}{{ end }}

---

{{ define "opsgenie.default.description" }}
{{ if (index .Alerts 0).Annotations.opsrecipe -}}
📗 Runbook: {{ template "__runbookurl" . }}
{{ end -}}
{{ else -}}
📗 Runbook: ⚠️ There is no **runbook** for this alert, time to get your pen.
{{- end }}
{{ if (index .Alerts 0).Annotations.dashboard -}}
📈 Dashboard: {{ template "__dashboardurl" . }}
{{ else -}}
📈 Dashboard: ⚠️ There is no **dashboard** for this alert, time to sketch.
{{ end -}}
🔔 Alertmanager {{ template "__alerturl" . }}
👀 Query: {{ template "__queryurl" . }}

---

{{ if not (index .Alerts 0).Annotations.opsrecipe }}⚠️ There is no **runbook** for this alert, time to get your pen.{{- end }}
{{ if not (index .Alerts 0).Annotations.dashboard }}⚠️ There is no **dashboard** for this alert, time to sketch.{{- end }}
* Team: {{ (index .Alerts 0).Labels.team }}
* Area: {{ (index .Alerts 0).Labels.area }} / {{ (index .Alerts 0).Labels.topic }}
* Instances:{{ range .Alerts.Firing }}
🔥 {{ if .Labels.instance }}{{ .Labels.instance }}: {{ end }}{{ .Annotations.description }}{{ end }}
{{- end }}

# This builds the silence URL. We exclude the alertname in the range
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,27 +30,26 @@

{{ define "opsgenie.default.message" }}{{ .GroupLabels.installation }} / {{ .GroupLabels.cluster_id }}{{ if (index .Alerts 0).Labels.service }} / {{ (index .Alerts 0).Labels.service }}{{ end }} - {{ index (index .Alerts.Firing 0).Labels `alertname`}}{{ end }}
{{ define "opsgenie.default.source" }}{{ template "__alertmanager" . }}{{ end }}
{{ define "opsgenie.default.description" }}* Team: {{ (index .Alerts 0).Labels.team }}
* Area: {{ (index .Alerts 0).Labels.area }} / {{ (index .Alerts 0).Labels.topic }}

* Instances:{{ range .Alerts.Firing }}
🔥 {{ if .Labels.instance }}{{ .Labels.instance }}: {{ end }}{{ .Annotations.description }}{{ end }}

---

{{ define "opsgenie.default.description" }}
{{ if (index .Alerts 0).Annotations.opsrecipe -}}
📗 Runbook: {{ template "__runbookurl" . }}
{{ end -}}
{{ else -}}
📗 Runbook: ⚠️ There is no **runbook** for this alert, time to get your pen.
{{- end }}
{{ if (index .Alerts 0).Annotations.dashboard -}}
📈 Dashboard: {{ template "__dashboardurl" . }}
{{ else -}}
📈 Dashboard: ⚠️ There is no **dashboard** for this alert, time to sketch.
{{ end -}}
🔔 Alertmanager {{ template "__alerturl" . }}
👀 Query: {{ template "__queryurl" . }}

---

{{ if not (index .Alerts 0).Annotations.opsrecipe }}⚠️ There is no **runbook** for this alert, time to get your pen.{{- end }}
{{ if not (index .Alerts 0).Annotations.dashboard }}⚠️ There is no **dashboard** for this alert, time to sketch.{{- end }}
* Team: {{ (index .Alerts 0).Labels.team }}
* Area: {{ (index .Alerts 0).Labels.area }} / {{ (index .Alerts 0).Labels.topic }}
* Instances:{{ range .Alerts.Firing }}
🔥 {{ if .Labels.instance }}{{ .Labels.instance }}: {{ end }}{{ .Annotations.description }}{{ end }}
{{- end }}

# This builds the silence URL. We exclude the alertname in the range
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,27 +30,26 @@

{{ define "opsgenie.default.message" }}{{ .GroupLabels.installation }} / {{ .GroupLabels.cluster_id }}{{ if (index .Alerts 0).Labels.service }} / {{ (index .Alerts 0).Labels.service }}{{ end }} - {{ index (index .Alerts.Firing 0).Labels `alertname`}}{{ end }}
{{ define "opsgenie.default.source" }}{{ template "__alertmanager" . }}{{ end }}
{{ define "opsgenie.default.description" }}* Team: {{ (index .Alerts 0).Labels.team }}
* Area: {{ (index .Alerts 0).Labels.area }} / {{ (index .Alerts 0).Labels.topic }}

* Instances:{{ range .Alerts.Firing }}
🔥 {{ if .Labels.instance }}{{ .Labels.instance }}: {{ end }}{{ .Annotations.description }}{{ end }}

---

{{ define "opsgenie.default.description" }}
{{ if (index .Alerts 0).Annotations.opsrecipe -}}
📗 Runbook: {{ template "__runbookurl" . }}
{{ end -}}
{{ else -}}
📗 Runbook: ⚠️ There is no **runbook** for this alert, time to get your pen.
{{- end }}
{{ if (index .Alerts 0).Annotations.dashboard -}}
📈 Dashboard: {{ template "__dashboardurl" . }}
{{ else -}}
📈 Dashboard: ⚠️ There is no **dashboard** for this alert, time to sketch.
{{ end -}}
🔔 Alertmanager {{ template "__alerturl" . }}
👀 Query: {{ template "__queryurl" . }}

---

{{ if not (index .Alerts 0).Annotations.opsrecipe }}⚠️ There is no **runbook** for this alert, time to get your pen.{{- end }}
{{ if not (index .Alerts 0).Annotations.dashboard }}⚠️ There is no **dashboard** for this alert, time to sketch.{{- end }}
* Team: {{ (index .Alerts 0).Labels.team }}
* Area: {{ (index .Alerts 0).Labels.area }} / {{ (index .Alerts 0).Labels.topic }}
* Instances:{{ range .Alerts.Firing }}
🔥 {{ if .Labels.instance }}{{ .Labels.instance }}: {{ end }}{{ .Annotations.description }}{{ end }}
{{- end }}

# This builds the silence URL. We exclude the alertname in the range
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,27 +30,26 @@

{{ define "opsgenie.default.message" }}{{ .GroupLabels.installation }} / {{ .GroupLabels.cluster_id }}{{ if (index .Alerts 0).Labels.service }} / {{ (index .Alerts 0).Labels.service }}{{ end }} - {{ index (index .Alerts.Firing 0).Labels `alertname`}}{{ end }}
{{ define "opsgenie.default.source" }}{{ template "__alertmanager" . }}{{ end }}
{{ define "opsgenie.default.description" }}* Team: {{ (index .Alerts 0).Labels.team }}
* Area: {{ (index .Alerts 0).Labels.area }} / {{ (index .Alerts 0).Labels.topic }}

* Instances:{{ range .Alerts.Firing }}
🔥 {{ if .Labels.instance }}{{ .Labels.instance }}: {{ end }}{{ .Annotations.description }}{{ end }}

---

{{ define "opsgenie.default.description" }}
{{ if (index .Alerts 0).Annotations.opsrecipe -}}
📗 Runbook: {{ template "__runbookurl" . }}
{{ end -}}
{{ else -}}
📗 Runbook: ⚠️ There is no **runbook** for this alert, time to get your pen.
{{- end }}
{{ if (index .Alerts 0).Annotations.dashboard -}}
📈 Dashboard: {{ template "__dashboardurl" . }}
{{ else -}}
📈 Dashboard: ⚠️ There is no **dashboard** for this alert, time to sketch.
{{ end -}}
🔔 Alertmanager {{ template "__alerturl" . }}
👀 Query: {{ template "__queryurl" . }}

---

{{ if not (index .Alerts 0).Annotations.opsrecipe }}⚠️ There is no **runbook** for this alert, time to get your pen.{{- end }}
{{ if not (index .Alerts 0).Annotations.dashboard }}⚠️ There is no **dashboard** for this alert, time to sketch.{{- end }}
* Team: {{ (index .Alerts 0).Labels.team }}
* Area: {{ (index .Alerts 0).Labels.area }} / {{ (index .Alerts 0).Labels.topic }}
* Instances:{{ range .Alerts.Firing }}
🔥 {{ if .Labels.instance }}{{ .Labels.instance }}: {{ end }}{{ .Annotations.description }}{{ end }}
{{- end }}

# This builds the silence URL. We exclude the alertname in the range
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,27 +30,26 @@

{{ define "opsgenie.default.message" }}{{ .GroupLabels.installation }} / {{ .GroupLabels.cluster_id }}{{ if (index .Alerts 0).Labels.service }} / {{ (index .Alerts 0).Labels.service }}{{ end }} - {{ index (index .Alerts.Firing 0).Labels `alertname`}}{{ end }}
{{ define "opsgenie.default.source" }}{{ template "__alertmanager" . }}{{ end }}
{{ define "opsgenie.default.description" }}* Team: {{ (index .Alerts 0).Labels.team }}
* Area: {{ (index .Alerts 0).Labels.area }} / {{ (index .Alerts 0).Labels.topic }}

* Instances:{{ range .Alerts.Firing }}
🔥 {{ if .Labels.instance }}{{ .Labels.instance }}: {{ end }}{{ .Annotations.description }}{{ end }}

---

{{ define "opsgenie.default.description" }}
{{ if (index .Alerts 0).Annotations.opsrecipe -}}
📗 Runbook: {{ template "__runbookurl" . }}
{{ end -}}
{{ else -}}
📗 Runbook: ⚠️ There is no **runbook** for this alert, time to get your pen.
{{- end }}
{{ if (index .Alerts 0).Annotations.dashboard -}}
📈 Dashboard: {{ template "__dashboardurl" . }}
{{ else -}}
📈 Dashboard: ⚠️ There is no **dashboard** for this alert, time to sketch.
{{ end -}}
🔔 Alertmanager {{ template "__alerturl" . }}
👀 Query: {{ template "__queryurl" . }}

---

{{ if not (index .Alerts 0).Annotations.opsrecipe }}⚠️ There is no **runbook** for this alert, time to get your pen.{{- end }}
{{ if not (index .Alerts 0).Annotations.dashboard }}⚠️ There is no **dashboard** for this alert, time to sketch.{{- end }}
* Team: {{ (index .Alerts 0).Labels.team }}
* Area: {{ (index .Alerts 0).Labels.area }} / {{ (index .Alerts 0).Labels.topic }}
* Instances:{{ range .Alerts.Firing }}
🔥 {{ if .Labels.instance }}{{ .Labels.instance }}: {{ end }}{{ .Annotations.description }}{{ end }}
{{- end }}

# This builds the silence URL. We exclude the alertname in the range
Expand Down
Loading

0 comments on commit c13636e

Please sign in to comment.