Add alert rules for abnormal CPU usage and load

[przemeklal]

Enhancement Proposal

Currently, grafana-agent doesn't set up any generic alert rules for abnormal CPU usage and load. In terms of defining alert thresholds, Nagios defaults could be a good starting point.

[aieri]

this request should already be fulfilled by canonical/grafana-agent-k8s-operator#191. Are those alerts not working well for you?

[przemeklal]

We had a load-related outage that was not detected, irate(node_pressure_cpu_waiting_seconds_total{}) hit around 0.6 while the total CPU usage was hitting almost 100% and the average system load to CPU count ratio was way above 1.0. The alert threshold for irate(node_pressure_cpu_waiting_seconds_total{}) metric is 0.9 for the record.

That said, we're planning to first experiment with cos-config alert rules and then follow up here or submit a PR directly.

[Abuelodelanada]

That said, we're planning to first experiment with cos-config alert rules and then follow up here or submit a PR directly.

Closing for now.