Create alerts based on Nginx dashboard

[Pitasi]

We do have a nginx ingress dashboard here:
https://grafana.emeris.allinbits.com/d/nginx/nginx-ingress-controller?orgId=1

Do not hesitate talking to devops for help in using Grafana and Alert Manager.

DoD

• define some metrics we can build alerts on (e.g. number of 4XX-5XX responses, requests duration)
• build some new charts for those metrics (you can copy the query definition from the nginx dashboards and tweak it a little, we cannot directly use the existing charts because alerts cannot include "template variables" (the four dropdowns you see at the top of the screen: Namespace, Controller Class, Controller, Ingress))
• create alerts from these new charts, use slack as a notification channel