kubelet 'cloud provider' extra flags not propagated to joined nodes

[NeilW]

BUG REPORT

Versions

kubeadm version (use kubeadm version):
kubeadm version: &version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"b1b29978270dc22fecc592ac55d903350454310a", GitTreeState:"clean", BuildDate:"2018-07-17T18:50:16Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version):
  Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"b1b29978270dc22fecc592ac55d903350454310a", GitTreeState:"clean", BuildDate:"2018-07-17T18:53:20Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"b1b29978270dc22fecc592ac55d903350454310a", GitTreeState:"clean", BuildDate:"2018-07-17T18:43:26Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration:
  Brightbox

• OS (e.g. from /etc/os-release):
  Ubuntu 18.04 LTS

• Kernel (e.g. uname -a):
  Linux srv-uf8ns 4.15.0-29-generic The product_uuid and the hostname should be unique across nodes #31-Ubuntu SMP Tue Jul 17 15:39:52 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

• Others:

What happened?

After running a kubeadm init with cloudProvider: external in the config (in either the v1alpha1 or v1alpha2 format), the cloud-provider flag is not passed to the kubelet on joined nodes in /var/lib/kubelet/kubeadm-flags.env

This means the node is not tainted and not initialised by the cloud-controller-manager application before pods are scheduled.

The master node is tainted and initialised as expected.

What you expected to happen?

If cloud provider external is specified the kubelet should be configured in external mode on all nodes joined to the master

How to reproduce it (as minimally and precisely as possible)?

apiVersion: kubeadm.k8s.io/v1alpha1
cloudProvider: external
kind: MasterConfiguration
clusterName: kubernetes
api:
  advertiseAddress: <v6 address>
networking:
  dnsDomain: cluster.local
  podSubnet: fdc0:726f:1::/48
  serviceSubnet: fdbf:726f:1::/112
token: 1m60m0.ie38hldi7yrlp2du

or

apiVersion: kubeadm.k8s.io/v1alpha2
apiServerExtraArgs:
  cloud-provider: external
controllerManagerExtraArgs:
  cloud-provider: external
nodeRegistration:
  kubeletExtraArgs:
    cloud-provider: external
kind: MasterConfiguration
clusterName: kubernetes
api:
  advertiseAddress: <v6_address>
networking:
  dnsDomain: cluster.local
  podSubnet: fdc0:726f:1::/48
  serviceSubnet: fdbf:726f:1::/112
bootstrapTokens:
  - token: 1m60m0.ie38hldi7yrlp2du

sudo kubeadm init --config kubeadm.conf

then run the printed join command on the worker node

Anything else we need to know?

The stored kubeadm-config configmap has an empty nodeRegistration block, and there is no config setting for cloud-provider in the kubelet-config-1.11 configmap (AIUI there isn't one possible)

...
kubernetesVersion: v1.11.1
networking:
  dnsDomain: cluster.local
  podSubnet: fdc0:726f:1::/48
  serviceSubnet: fdbf:726f:1::/112
nodeRegistration: {}
unifiedControlPlaneImage: ""

Events:  <none>

The result is a different env file on master and worker

master

ubuntu@srv-uf8ns:~$ more /var/lib/kubelet/kubeadm-flags.env 
KUBELET_KUBEADM_ARGS=--cloud-provider=external --cgroup-driver=cgroupfs --cni-bin-dir=/opt/cni/bin --cni-conf-dir=/etc/cni/net.d --network-plugin=cni --resolv-conf=/run/systemd/resolve/resolv.conf

worker

ubuntu@srv-asl5n:~$ more /var/lib/kubelet/kubeadm-flags.env 
KUBELET_KUBEADM_ARGS=--cgroup-driver=cgroupfs --cni-bin-dir=/opt/cni/bin --cni-conf-dir=/etc/cni/net.d --network-plugin=cni --resolv-conf=/run/systemd/resolve/resolv.conf

[NeilW]

Workaround (tested on 1.11.1) is to set the default file on each worker node before initialising.

cat <<EOF | sudo tee /etc/default/kubelet
KUBELET_EXTRA_ARGS=--cloud-provider=external
EOF

[luxas]

Workaround (tested on 1.11.1) is to set the default file on each worker node before initialising.

This is expected, everything in nodeRegistration is on purpose instance-specific. What you set there is only preserved in the place of execution. Instead, set nodeRegistations in NodeConfiguration or modify /etc/default/kubelet as described.

We might wanna document this for the specific case of cloud providers, as that option isn't available in the config file.

[NeilW]

How does that sit with the idea that the token is all you need to connect to the cluster. Has that concept been abandoned?

[luxas]

@NeilW I'm not totally sure I understand the question. bootstrapTokens is ephemeral, not uploaded to the cluster configmap for security reasons, but used at kubeadm init time to apply the desired state to the cluster so that you can connect with a token if you want. However, tokens really should be used only for bootstrapping

[NeilW]

If I use a bootstrap token with a 'kubeadm join', the cluster won't join properly because the 'cloud-provider: external' isn't propagated to the kubelet config on the Node by the Master.
I have to give the node the cloud provider specification, not just the token. My understanding was that the join workflow was only supposed to require the security details (the token) and everything else would be supplied by the master on request. As it stands it doesn't set the cloud provider properly. Are there moves to make 'cloud-provider: external' the default so we don't need it?

[luxas]

everything else would be supplied by the master on request

Yes, normally, but not in this case as cloud provider information isn't stored in the kubelet ComponentConfig. hmm yeah this is an edge case indeed. For now you just have to feed it to your kubelets.

Are there moves to make 'cloud-provider: external' the default so we don't need it?

I don't think it ever will be the default.

cc @andrewsykim FYI