KubeControllerManagerDown & kubeSchedulerDown firing on kubeadm 1.18 cluster #718
Comments
|
We do have a patch to disable those alerts and ServiceMonitors in clusters which don't expose access to controller manager and scheduler metrics. But yes, our documentation is lacking in this field. |
|
It would be amazing if you could propose a PR to add what you described in "the workaround" where you expected to find this documentation when you were looking for it! :) |
|
yes |
|
@brancz sorry, I've been meaning to get back to this but I keep getting side tracked with other things. Do you think the Troubleshooting section on the main page would be a suitable place for this? If yes, I can put something together linking to either the patch for disabling those checks if on a managed cluster, or for updating kubeadm to use 0.0.0.0 instead of the loopback address for the secure port (at a users discretion of course). |
|
Yes! I think the troubleshooting guide is a great place because that's most likely what people look at when they encounter this. |
|
I have changed the bind address, but it did not work. It is strange. |
|
@KeithTt I have encountered the same trouble recently, It took me 5 days to figure it out. Check these points:
FYI. my yaml files: apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
k8s-app: kube-scheduler
name: kube-scheduler
namespace: monitoring
spec:
endpoints:
- bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
interval: 30s
port: https-metrics
scheme: https
tlsConfig:
insecureSkipVerify: true
jobLabel: k8s-app
namespaceSelector:
matchNames:
- kube-system
selector:
matchLabels:
k8s-app: kube-schedulerapiVersion: v1
kind: Service
metadata:
namespace: kube-system
name: kube-scheduler
labels:
k8s-app: kube-scheduler
spec:
ports:
- name: https-metrics
port: 10259
selector:
component: kube-scheduler |
|
Note, that both Binding to I see two approaches:
|
I totally agree, which is partially why I haven't made time to submit the PR for updating the troubleshooting docs - that and I'm swamped.
Yea, if what you're saying is true than that would be necessary - I don't recall having to set this value though. Does prometheus even check for valid certificates on the endpoints it scrapes?
I think this solution is more realistic - I set the bind address to 0.0.0.0 out of laziness, and I suspect most folks are like me (not that this is a good thing!) |
It is. Scraping fails otherwise.
At the moment I have no idea how to make it. Also, I'm not sure if anything relies on scheduler/control-manager being bounded to 127.0.0.1. Opposing to etcd these can be bound only to a single address AFAIK. |
|
kubernetes/kubeadm#2244 - related issue about kube-scheduler and kube-controller-manager certificates. |
|
it seems that in k8s v1.20.2 (probably even before, i didn't check) even though in the documentations they are saying: both /etc/kubernetes/manifests/kube-scheduler.yaml and /etc/kubernetes/manifests/kube-controller-manager.yaml come with: - --port=0 out of the box if someone can approve/deny it i will appreciate it, because i don't like this solution even for a workaround. |
|
@omerozery I think the default value for controllerManager:
extraArgs:
bind-address: 0.0.0.0
scheduler:
extraArgs:
bind-address: 0.0.0.0as posted by the OP. Regarding your workaround, kube-controller-manager v1.20 doesn't have a |
|
i wrote the comment above after i tried it myself.. clearly the 127.0.0.1 is ignored , am i missing something? |
|
@omerozery @weibeld Note that
I made comments in linked kubeadm issue. Generally, IMO |
|
see my comment here:
we've seen the requests about it, but the response has been that we don't want to expose the components outside of localhost due to metrics. the current best workaround:
of course, if there are sufficient votes about this change request let's comment on kubernetes/kubeadm#2388 |
I meant don't use the deprecated |
|
The proxy workaround can be relatively easily implemented by running an HAProxy container with the following configuration as a DaemonSet on each master node: defaults
mode http
timeout connect 5000ms
timeout client 5000ms
timeout server 5000ms
default-server maxconn 10
frontend kube-controller-manager
bind ${NODE_IP}:10257
http-request deny if !{ path /metrics }
default_backend kube-controller-manager
backend kube-controller-manager
server kube-controller-manager 127.0.0.1:10257 ssl verify none
frontend kube-scheduler
bind ${NODE_IP}:10259
http-request deny if !{ path /metrics }
default_backend kube-scheduler
backend kube-scheduler
server kube-scheduler 127.0.0.1:10259 ssl verify noneNote the following:
After deploying the DaemonSet, you can scrape the metrics of kube-controller-manager and kube-scheduler on The main disadvantage of this workaround is that the names of the kube-controller-manager and kube-scheduler Pods are lost. Prometheus discovers only the names of the proxy Pods, which is not really useful. Prometheus discovers the names of the nodes the Pods run on (in the |
|
Thanks! But I think this is‘ot good idea. |
Does anyone know where this patch is? Any 1.20 or above cluster, basically none of these workarounds, including HAProxy, ever expose those metrics. Probably best to offer a quick way to disable this entirely until folks make a decision some day about this. |
|
Try setting |
@jgerry2002 I have this working, but you need to match the service label to firing instead now, it is complaining kube-scheduler and kube-controller-manager pod in kube-system is down 😆 |
So it looks like the solution from the patch is simply not creating any rule for scheduler and controller? |
Actually, no. Selecting Disabling and using |
Thank you for the explanation. I got it now 😊 |
Have you tested in 1.20+? I'm also using Tanzu, which is sort of EKS-ish with some of the security and other (attempted) abstraction that is in there. Which is why I figured I'd ask to have an option to disable these checks cleanly and attempt to figure out a way to add that piece of monitoring back in later on. |
|
I just tried to create the manifest for kubeadm platform and I can't find the service manifest anywhere. Am I doing something wrong? This is the modified example.jsonnet which I called using I modified the namespace intentionally to see if the namespace does change in the generated manifests, only no service for kube-scheduler and kube-controller-manager manifest found anywhere. |
|
Those should be created as files with |
There's no such file created. I also tried changing it to: |
|
Its also not working for me Version: 0.8 minimal example.jsonnet: |
|
Got it. |
|
@freym thank you! |
|
Since this issue is tagged with |
|
Just in case someone else like me having troubles understanding the things everyone here is talking about because they are quite new to all of this, using the helm chart prometheus-community/kube-prometheus-stack to deploy kube-prometheus via helm, and using Docker Desktop for their local development cluster, I had posted a question to solve a problem that turned out to be this on the mailing list of prometheus-community, and I've posted my explanations on why which step is necessary to fix which sub-problem over there once I've figured it out: https://groups.google.com/g/prometheus-users/c/_aI-HySJ-xM/m/kqrL1FYVCQAJ - I hope this helps! |
same to you. service's port name from http-metrics to https-metrics. It works |
Thank you for using your method to solve the problem |
|
|
ok, mega-hack incoming: apiVersion: apps/v1
kind: DaemonSet
metadata:
name: socat-hack-for-kubeadm-bind-on-localhost-only
namespace: kube-system
labels:
app: socat-hack
spec:
selector:
matchLabels:
app: socat-hack
template:
metadata:
labels:
app: socat-hack
spec:
nodeSelector:
node-role.kubernetes.io/control-plane: ""
tolerations:
- key: "node-role.kubernetes.io/control-plane"
operator: "Exists"
effect: "NoSchedule"
hostNetwork: true
containers:
- name: socat-etcd
image: alpine/socat
env:
- name: NODE_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.hostIP
command: ["/bin/sh"]
args:
- -c
- socat TCP4-LISTEN:2381,reuseaddr,bind=${NODE_IP},fork TCP4:127.0.0.1:2381
ports:
- containerPort: 2381
- name: socat-kube-controller-manager
image: alpine/socat
env:
- name: NODE_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.hostIP
command: ["/bin/sh"]
args:
- -c
- socat TCP4-LISTEN:10257,reuseaddr,bind=${NODE_IP},fork TCP4:127.0.0.1:10257
ports:
- containerPort: 10257
- name: socat-kube-scheduler
image: alpine/socat
env:
- name: NODE_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.hostIP
command: ["/bin/sh"]
args:
- -c
- socat TCP4-LISTEN:10259,reuseaddr,bind=${NODE_IP},fork TCP4:127.0.0.1:10259
ports:
- containerPort: 10259
- name: socat-kube-proxy
image: alpine/socat
env:
- name: NODE_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.hostIP
command: ["/bin/sh"]
args:
- -c
- socat TCP4-LISTEN:10249,reuseaddr,bind=${NODE_IP},fork TCP4:127.0.0.1:10249
ports:
- containerPort: 10249Why this works? Because e.g. etcd has bound itself on |
|
I don't think this is as hacky as you make it sound. It's pretty much what I suggested in the k0s issue, except simpler compared to mine. |
|
Here is an awesome writeup I found that gives a more beginner-friendly analysis of this issue https://groups.google.com/g/prometheus-users/c/_aI-HySJ-xM/m/kqrL1FYVCQAJ |
and others
What happened?
Deploying kube-prometheus release-0.6 to a kubeadm boostrapped bare-metal cluster causes KubeControllerManagerDown and kubeSchedulerDown alerts to fire.
Did you expect to see some different?
Alerts should not fire as everything is up.
How to reproduce it (as minimally and precisely as possible):
Deploy release-0.6 with the bellow config to a kubeadm boostrapped cluster running 1.18.x
Prometheus Operator version:
prometheus-operator:v0.42.1
Manifests:
Anything else we need to know?:
This issue is related to Kubernetes 1.18.x, it appears a few changes were made to the kube-controller-manager and kube-scheduler.
Firstly, version 1.18+ now uses the more secure https port of 10257 and disables http by default. Unfortunately, the --secure-port used by both kube-controller-manager and kube-scheduler is bound to 127.0.0.1 and not 0.0.0.0.
As a result, metrics cannot be collected until the bound address for both of these is updated.
The workaround:
Updating the manifests in /etc/kubernetes/manifests/ to use the --bind-address 0.0.0.0 for both the scheduler and the controller manager will relaunch the the pods with the correct bind address, but these settings will not survive a kubeadm upgrade.
In order to persist the settings, the kubeadm-config configmap in the kube-system namespace should also be edited to include the following:
I understand this isn't a bug directly related to kube-prometheus, but I didn't find this documented anywhere and had been scratching my head for a day looking at this. Hoping this will help someone else in the future.
The text was updated successfully, but these errors were encountered: