Critical alerts constantly firing for Kube Proxy, Scheduler and ControllerManager

[staticvoid255]

We're getting a stream of alerts of critical severity:

• KubeSchedulerDown
• KubeProxyDown
• KubeControllerManagerDown

This is a known issue with kube-prometheus-stack, where the ports and/or IP's for these services are either not accessible, or are not set in the scraping configuration

Please see here and here

Target disappeared from Prometheus target discovery.
[[FIRING:1] Notification for cluster "XXXXXX/azure/XXXXX"](https://apps.my-domain.com?receiver=critical-home)
Alert: Target disappeared from Prometheus target discovery. - critical
 Description: KubeProxy has disappeared from Prometheus target discovery.
 Graph: :chart_with_upwards_trend: Runbook: :spiral_note_pad:
 Details:
  • alertname: KubeProxyDown
  • cluster: XXXXXX/azure/XXXXX
  • prometheus: monitoring/po-prometheus
  • severity: critical

[Morriz]

Regression prolly. We had these rules disabled before for isManaged k8s.

[srodenhuis]

You can add this to the raw_values of prometheus operator:

kubeProxy:
enabled: false
kubeScheduler:
enabled: false
kubeControllerManager:
enabled: false