Kubeadm to have an object in Kubeadm config for the Node Specific kubelet configurations

[kumaravanish]

Hi,

This is a FEATURE REQUEST for Kubeadm to have an object in Kubeadm config for the Node Specific kubelet configuration.

During kubeadm init, kubeadm uploads the ClusterConfiguration object to K8s cluster in a ConfigMap called kubeadm-config in the kube-system namespace. This configuration is then read during kubeadm join, kubeadm reset and kubeadm upgrade.
So is the object KubeletConfiguration where kubelet values to be used by kubeadm init and kubeadm join can be provided which are applicable cluster wide. But there is no such Object or configuration available in kubeadm config to specify the Node Specific kubelet configurations.

Versions

• kubeadm version (use kubeadm version):
  kubeadm version: &version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.0", GitCommit:"6abac1505370282a9583329dd35622792b60e449", GitTreeState:"clean", BuildDate:"2022-01-10T12:53:28Z", GoVersion:"go1.17.3", Compiler:"gc", Platform:"linux/amd64"}

• Environment: Multi Node Kubernetes cluster.

• Kubernetes version (use kubectl version):
  Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.0", GitCommit:"6abac1505370282a9583329dd35622792b60e449", GitTreeState:"clean", BuildDate:"2022-01-10T12:53:58Z", GoVersion:"go1.17.3", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.0", GitCommit:"6abac1505370282a9583329dd35622792b60e449", GitTreeState:"clean", BuildDate:"2022-01-10T12:44:17Z", GoVersion:"go1.17.3", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration: Openstack VM

• OS (e.g. from /etc/os-release):
  NAME="SLES"
  VERSION="15-SP2"
  VERSION_ID="15.2"
  PRETTY_NAME="SUSE Linux Enterprise Server 15 SP2"
  ID="sles"
  ID_LIKE="suse"
  ANSI_COLOR="0;32"
  CPE_NAME="cpe:/o:suse:sles:15:sp2"

• Kernel (e.g. uname -a): Linux 5.3.18-24.96 x86_64 x86_64 x86_64 GNU/Linux

• Container runtime (CRI) (e.g. containerd, cri-o): containerd

• Container networking plugin (CNI) (e.g. Calico, Cilium): calico

• Others:

What happened?

Intention and kubelet configuration : kubelet service should bind to a specific node network ip instead default configuration of all network.The Kubelet's config.yaml can be set for kubelet's address parameter which binds kubelet service to listen on provided ip address (could be eth0). The parameter should be set via the config file specified by the Kubelet's --config flag because of deprecation as a runtime argument.

Problem arises when the config.yaml is managed by the kubeadm. The parameter can be provided in first master node as a kubeadm config under KubeletConfiguration .The kubelet configuration is marshalled to disk at /var/lib/kubelet/config.yaml, and also uploaded to a ConfigMap in the cluster. The ConfigMap is named kubelet-config-1.X, where X is the minor version of the. The Kubelet service is successful bind to the local ip address of first master.

But, when Kubeadm is executed on the rest of masters & worker nodes, it downloads the kubelet-config-1.X ConfigMap and writes it to /var/lib/kubelet/config.yaml which has the same value for "address" as the first master. The kubelet service failed to bind to the address.

What you expected to happen?

The problem is common for all the node specific parameters and becomes worse where the run time kubelet parameters are deprecated. There should be object under the kubeadm config to provide node specific details. or Node should be allowed top modify and have local config map, of course with restriction of not modifying the cluster level and other nodes details.

How to reproduce it (as minimally and precisely as possible)?

Try to setup multi-node kubernetes cluster with the configuration in the kubeadm config file under the KubeletConfiguration with address:<nodes eth0 ip> .

Anything else we need to know?

This seems to a valid use case, might be possible it has been raised before. Please share if any and plan for the feature.

[neolit123]

this is a known problem and we are already tracking it, so please subscribe to these tickets:
#2367
#1682

admittedly we are a on hold for a number of reasons:

• the kubeletconfiguration is stuck in v1beta1 and the design to split it into instance specific (like the IP that you want) vs global settings did not get progress...i cannot find the link for that, but it's somewhere in kubernetes/enhancements.
• we need to figure you what to do on the kubeadm side, do we wait for the instance vs global kubelet config, do we implement other mechanism like storing multiple config maps for each node, do we use patches to write custom overrides in the config on disk during join...etc the kubeadm maintainers need to find time to discuss / make the decisions.

The problem is common for all the node specific parameters and becomes sever where the run time kublet parameters are deprecated.

note that NONE of the deprecated flags will be removed before the kubelet decides what to do with instance specific options.