Current stable CoreOS version may cause wrong CPU metrics in HPA

[guiocavalcanti]

There's a bug in Systemd (>=226, w/ docker configured to use the systemd cgroup driver, and a docker version before 1.12) which causes wrong CPU metrics in cAdvisor and Heapster.

These problems are reported here and here.

As far as I can tell, the bug is fixed in beta channel. My question is: how can I upgrade CoreOS version without recreating the cluster?

cc @yagonobre

[seanknox]

My question is: how can I upgrade CoreOS version without recreating the cluster?

Fair warning, it's a fairly involved process. For each node type (workers, masters/etcd), the steps roughly are:

1. Bring up additional infrastructure with the new CoreOS version
2. New infrastructure joins the cluster
3. Drain connections to the old infrastructure
4. Confirm old infrastructure is idle and retire

E.g. for workers,

• copy ASG/launch configuration of existing workers, change the ami-id to the newer coreos AMI (the one running beta)
• bring up a mirror number of worker nodes running the new CoreOS version
• confirm they've joined the cluster and are schedulable
• drain pods from the old workers
• delete the old ASG/launch configuration for the old workers from Terraform manifests, and apply to terminate the nodes and destroy the old worker instructure.

You'll then repeat for master/etcd. Additional info about upgrading individual k8s components are here: https://coreos.com/kubernetes/docs/latest/kubernetes-upgrade.html