FR: expose kube-scheduler and kube-controller-manager endpoints in order to determine which instance is the leader

[saphoooo]

This issue is close to this request by providing an enswer for /metrics but not for the leader-election mechanism which rely on endpoints too.

FEATURE REQUEST

Give the ability in kubeadm to expose kube-scheduler and kube-controller-manager through endpoints instead of relying on localhost binding.

Versions

kubeadm version (use kubeadm version): all

Environment: minikube

• Kubernetes version (use kubectl version): it's due to the implementation, the Kubernetes version doesn't matter
• Cloud provider or hardware configuration: minikube
• OS (e.g. from /etc/os-release): doesn't matter as well
• Kernel (e.g. uname -a): doesn't matter as well

What happened?

Monitoring tools (eg Datadog) rely on kube-scheduler and kube-controller-manager endpoint to figure out which isnstance is the leader, but failed because there is no way to expose kube-scheduler and kube-controller-manager via endpoints in kubeadm

What you expected to happen?

Allow a monitoring tool to scrape the endpoint in order to determine which instance is the leader

How to reproduce it (as minimally and precisely as possible)?

Actual behaviour:

$ kubectl describe endpoints kube-scheduler  -n kube-system
Error from server (NotFound): endpoints "kube-scheduler" not found

Expected behaviour:

$ kubectl describe endpoints kube-scheduler  -n kube-system
Name:         kube-scheduler
Namespace:    kube-system
Labels:       <none>
Annotations:  control-plane.alpha.kubernetes.io/leader={"holderIdentity":"controller-0","leaseDurationSeconds":15,"acquireTime":"2018-01-19T13:12:57Z","renewTime":"2018-01-19T13:13:54Z","leaderTransitions":1}
Subsets:
Events:
  Type    Reason          Age   From               Message
  ----    ------          ----  ----               -------
  Normal  LeaderElection  38m   default-scheduler  controller-1 became leader
  Normal  LeaderElection  59s   default-scheduler  controller-0 became leader

[neolit123]

Monitoring tools (eg Datadog) rely on kube-scheduler and kube-controller-manager endpoint to figure out which isnstance is the leader, but failed because there is no way to expose kube-scheduler and kube-controller-manager via endpoints in kubeadm

this is not true.

e.g. here https://www.katacoda.com/courses/kubernetes/playground kubeadm 1.18 is used to create a cluster.
it binds the scheduler to localhost, but the endpoints are still accessible:

kubectl describe endpoints kube-scheduler  -n kube-system

... 

Annotations:  control-plane.alpha.kubernetes.io/leader:
                {"holderIdentity":"controlplane_3ab2471a-48f5-4b35-a04c-3c5366c1ec4f","leaseDurationSeconds":15,"acquireTime":"2021-10-31T16:39:48Z","rene...

binding the components to localhost and endpoint leader election tracking are not related.

to track /metrics on externally exposed IP, you can customize the components:
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/control-plane-flags/

or you can use a proxy service, that allows you to fetch metrics from the locally bound components and expose them externally.

exposing the components to external serving by default in kubeadm is not desired.

[github-actions]

Hello, @saphoooo 🤖 👋

You seem to have troubles using Kubernetes and kubeadm.
Note that our issue trackers should not be used for providing support to users.
There are special channels for that purpose.

Please see:

• https://github.com/kubernetes/kubeadm#support

[saphoooo]

I finally tackled this point, but in my opinion it's poorly documented.

In order to have these endpoints automatically generated, you need some extra configurations for controller-manager and scheduler:

• --leader-elect=true
• --leader-elect-resource-lock=endpoints

To set these flags in Kubeadm, you need to rely on ClusterConfiguration:

---
apiServer:
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager:
  extraArgs:
    leader-elect-resource-lock: endpoints
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: k8s.gcr.io
kind: ClusterConfiguration
kubernetesVersion: 1.22.0
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
scheduler:
  extraArgs:
    leader-elect-resource-lock: endpoints

By default, leader-elect is set to true in Kubeadm and false in Minikube. eader-elect-resource-lock is by default leader, which sounds not the best default choice in order to be able to monitor which component is the leader.