Add KubePodEvictionRateHigh alert for elevated eviction rates

[mac-chaffee]

Fixes #759

This PR adds an alert to detect high pod eviction rates. Since the underlying metric (kubelet_evictions) doesn't have many labels, this alert just detects the cluster-wide eviction rate rather than any particular namespace or workload.

This is very useful for workloads that may have RAM/ephemeral-storage limits that are set too low. Especially for DaemonSets because DaemonSets will silently remove evicted pods, so the issue is invisible unless you look at the events and pod ages. It could also be an early-warning sign for a DDoS attack.

[arajkumar]

Did you check kube_pod_status_reason ? I think the reason label of this metric shall be used to find whther the pod has been evicted.

[mac-chaffee]

Hmm looks like that metric can't be used to determine the eviction rate since the query returns results for hours/days after the eviction actually happened (those two evictions happened yesterday, but appear when searching for evictions in the last 15 minutes):

Maybe there's a way we could still use kube_pod_status_reason query results in the description? Or is the description limited to using results from the expr?

[povilasv]

We need to figure out a solution, where the responsible team gets the alert rather than a general KubeEvictionHigh alert.

Example: If a namespace owned by a team A sets limits too low, the pod gets evicted Team A should get the alert, not the infrastructure team :)

[mac-chaffee]

Sounds like we're stuck with the limitations of the kubelet_evictions metric, which doesn't include the namespace. I can look into adding that, but it'll be a long time before that lands.

Also that example only applies to clusters where resource limits are mandated, which is sadly not the norm. If limits aren't mandated, one pod using too many resources can cause a cascade of evictions across the whole node, something I think we should send alerts for sooner rather than later. You can always silent alerts, but the alert has to exist first :)

[mac-chaffee]

We can't add a namespace label to the kubelet_evictions due to cardinality issues: kubernetes/kubernetes#104033

So it seems the current implementation is the best we can do.

[povilasv]

What about using kube-state-metrics and looking for Evicted Pods?

[mac-chaffee]

That kube_pod_status_reason metric that @arajkumar mentioned is the only kube-state-metrics metric that tracks evictions, but as mentioned I don't see how to turn that metric into an eviction rate.

[mac-chaffee]

Sounds like there just isn't enough data to make a good, actionable alert. For anyone seeing this PR: I do still recommend using this alert since higher-than-normal eviction rates are always good to catch. You would just have to tune the threshold for your own cluster and have other sources of data (like event logging) to cross-reference.