HA Etcd bootstrapping as documented doesn't obey kubeadm.conf

[danderson]

/kind bug

Versions

kubeadm version (use kubeadm version): kubeadm version: &version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"b1b29978270dc22fecc592ac55d903350454310a", GitTreeState:"clean", BuildDate:"2018-07-17T18:50:16Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version): Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"b1b29978270dc22fecc592ac55d903350454310a", GitTreeState:"clean", BuildDate:"2018-07-17T18:53:20Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
• Cloud provider or hardware configuration: Bare metal, x86_64
• OS (e.g. from /etc/os-release): Debian Buster (aka "testing")
• Kernel (e.g. uname -a): Linux test-01 4.17.0-1-amd64 #1 SMP Debian 4.17.8-1 (2018-07-20) x86_64 GNU/Linux

What happened?

We followed the HA Etcd setup guide on kubernetes.io (https://kubernetes.io/docs/setup/independent/setup-ha-etcd-with-kubeadm/). When you follow it to the letter, the generated etcd configurations don't match the desired intent: instead of being configured to form a quorum, the generated etcd pod manifest just listens on localhost, and is configured to run a single-replica cluster.

In other words, the config settings specified in the kubeadm config don't get applied to the pod manifest.

My guess as to root cause is that the documentation is using an outdated config format, and kubeadm's yaml parsing isn't using the newly introduced strict parsing mode (which complains if it sees fields that don't show up in the Go struct), so kubeadm is silently throwing away bits of configuration.

That, and of course the documentation needs updating (or at least delete it if it's wrong).

What you expected to happen?

We expected the documentation on the official website to work :)

How to reproduce it (as minimally and precisely as possible)?

On some sacrificial machine (you'll be overwriting the etcd pod manifest), follow the HA etcd cluster documentation linked above. In the interest of a minimal repro recipe, the following is a condensed version of the interesting steps.

Copy the kubeadm config generation script into conf.sh and save (no need to customize the values, we're not going to be able to build a cluster anyway).

cat >/tmp/conf.sh <<FEO
# Update HOST0, HOST1, and HOST2 with the IPs or resolvable names of your hosts
export HOST0=10.0.0.6
export HOST1=10.0.0.7
export HOST2=10.0.0.8

# Create temp directories to store files that will end up on other hosts.
mkdir -p /tmp/${HOST0}/ /tmp/${HOST1}/ /tmp/${HOST2}/

ETCDHOSTS=(${HOST0} ${HOST1} ${HOST2})
NAMES=("infra0" "infra1" "infra2")

for i in "${!ETCDHOSTS[@]}"; do
HOST=${ETCDHOSTS[$i]}
NAME=${NAMES[$i]}
cat << EOF > /tmp/${HOST}/kubeadmcfg.yaml
apiVersion: "kubeadm.k8s.io/v1alpha2"
kind: MasterConfiguration
etcd:
    localEtcd:
        serverCertSANs:
        - "${HOST}"
        peerCertSANs:
        - "${HOST}"
        extraArgs:
            initial-cluster: infra0=https://${ETCDHOSTS[0]}:2380,infra1=https://${ETCDHOSTS[1]}:2380,infra2=https://${ETCDHOSTS[2]}:2380
            initial-cluster-state: new
            name: ${NAME}
            listen-peer-urls: https://${HOST}:2380
            listen-client-urls: https://${HOST}:2379
            advertise-client-urls: https://${HOST}:2379
            initial-advertise-peer-urls: https://${HOST}:2380
EOF
done
FEO

Run the script.

bash /tmp/conf.sh

Verify that you have an kubeadm.conf to play with (you have 3, but we'll just need the one)

cat /tmp/10.0.0.6/kubeadmcfg.yaml

Use said kubeadm config to generate an etcd pod manifest (here we skip all the cert generation and copying stuff from the official docs, for brevity)

kubeadm alpha phase etcd local --config=/tmp/10.0.0.6/kubeadmcfg.yaml

Inspect the resulting etcd pod manifest. Note that all the URLs contained in etcd's commandline are pointing to 127.0.0.1, etcd is listening on 127.0.0.1, and the initial cluster config specifies a 1-replica cluster. This is nothing like the config we (apparently) requested in kubeadmcfg.yaml.

cat /etc/kubernetes/manifests/etcd.yaml

[timothysc]

/assign @chuckha @detiber