Add weave-kube DaemonSet that can be used with k8s version v1.6

[luxas]

@bboreham @errordeveloper @lukemarsden

ref: #2777

[bboreham]

I ran a test to check what happens if you load this file on k8s 1.5:

unable to decode "STDIN": no kind "ClusterRole" is registered for version "rbac.authorization.k8s.io/v1beta1"
unable to decode "STDIN": no kind "ClusterRoleBinding" is registered for version "rbac.authorization.k8s.io/v1beta1"

So we need to figure out how best to help our users. #2754, perhaps.

[luxas]

@bboreham Totally known issue, this is what's hard with providing manifests; they often have to track the release branches of k8s

In this case; Taints and Tolerations was alpha in v1.5; now beta. The same goes for RBAC, hence the above error. RBAC beta did obviously not exist in v1.5.

We have to provide different manifests for v1.5 and v1.6 users somehow

[bboreham]

Comment copied from #2777:

Note fluxcd/flux#400 (comment):

both alpha and beta APIs exist in 1.6, and a role created with v1alpha1 will also be accessible from the v1beta1 API

so I tried this, and it seemed to work at https://circleci.com/gh/weaveworks/weave/8146 (albeit the 'publish' phase failed on branch naming)

[jbeda]

I'm seeing a couple of problems when I apply this to the latest beta of kubeadm/k8s.

1. The taint needs to be specified as a real member and not an annotation.
2. The kube-dns pod needs to be killed/restarted. I'm seeing the following in the describe there:

  13m	38s	57	kubelet, kubeadm-master		Warning	FailedSync	Error syncing pod, skipping: failed to "KillPodSandbox" for "2b03d2c9-0909-11e7-b38b-42010a8a0004" with KillPodSandboxError: "rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod \"kube-dns-3913472980-x7vp7_kube-system\" network: cni config uninitialized"

  13m	4s	60	kubelet, kubeadm-master		Normal	SandboxChanged	Pod sandbox changed, it will be killed and re-created.
  27s	4s	3	kubelet, kubeadm-master		Warning	FailedSync	Error syncing pod, skipping: failed to "KillPodSandbox" for "2b03d2c9-0909-11e7-b38b-42010a8a0004" with KillPodSandboxError: "rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod \"kube-dns-3913472980-x7vp7_kube-system\" network: required env variables missing"

It looks like there is an error about a missing env variable. Is this the env variable for the kubernetes service?

[bboreham]

"cni config uninitialized" suggests the Weave Net pods never got started, because they write out the config. Do you see anything in their logs?

[jbeda]

The weave pods were definitely up. I suggest you try the latest kubeadm to see if you can repo.

[jbeda]

The kubelet has some errors like this:

Mar 15 00:46:19 kubeadm-master kubelet[2702]: W0315 00:46:19.180301    2702 docker_sandbox.go:234] NetworkPlugin cni failed on the status hook for pod "kube-dns-3913472980-x7vp7_kube-system": Unexpected command output nsenter: cannot open : No
Mar 15 00:46:19 kubeadm-master kubelet[2702]:  with error: exit status 1
Mar 15 00:46:19 kubeadm-master kubelet[2702]: 2017/03/15 00:46:19 CNI_NETNS env variable missing
Mar 15 00:46:19 kubeadm-master kubelet[2702]: E0315 00:46:19.186691    2702 cni.go:273] Error deleting network: required env variables missing
Mar 15 00:46:19 kubeadm-master kubelet[2702]: E0315 00:46:19.187645    2702 remote_runtime.go:109] StopPodSandbox "f2171645cf9f735b38590d5c19f69f4983f056a07180026cc93c2234738d5126" from runtime service failed: rpc error: code = 2 desc = Network
Mar 15 00:46:19 kubeadm-master kubelet[2702]: E0315 00:46:19.187935    2702 kuberuntime_gc.go:138] Failed to stop sandbox "f2171645cf9f735b38590d5c19f69f4983f056a07180026cc93c2234738d5126" before removing: rpc error: code = 2 desc = NetworkPlug
Mar 15 00:46:19 kubeadm-master kubelet[2702]: W0315 00:46:19.195435    2702 docker_sandbox.go:234] NetworkPlugin cni failed on the status hook for pod "kube-dns-3913472980-5j347_kube-system": Unexpected command output nsenter: cannot open : No
Mar 15 00:46:19 kubeadm-master kubelet[2702]:  with error: exit status 1
Mar 15 00:46:19 kubeadm-master kubelet[2702]: 2017/03/15 00:46:19 CNI_NETNS env variable missing
Mar 15 00:46:19 kubeadm-master kubelet[2702]: E0315 00:46:19.201472    2702 cni.go:273] Error deleting network: required env variables missing
Mar 15 00:46:19 kubeadm-master kubelet[2702]: E0315 00:46:19.202615    2702 remote_runtime.go:109] StopPodSandbox "b11a00b7adb3e17f6e1551c70f01d9066b525589feb1cd0822f7aa5271fc5124" from runtime service failed: rpc error: code = 2 desc = Network
Mar 15 00:46:19 kubeadm-master kubelet[2702]: E0315 00:46:19.202781    2702 kuberuntime_gc.go:138] Failed to stop sandbox "b11a00b7adb3e17f6e1551c70f01d9066b525589feb1cd0822f7aa5271fc5124" before removing: rpc error: code = 2 desc = NetworkPlug
Mar 15 00:46:19 kubeadm-master kubelet[2702]: W0315 00:46:19.210471    2702 docker_sandbox.go:234] NetworkPlugin cni failed on the status hook for pod "kube-dns-3913472980-520s7_kube-system": Unexpected command output nsenter: cannot open : No
Mar 15 00:46:19 kubeadm-master kubelet[2702]:  with error: exit status 1
Mar 15 00:46:19 kubeadm-master kubelet[2702]: 2017/03/15 00:46:19 CNI_NETNS env variable missing
Mar 15 00:46:19 kubeadm-master kubelet[2702]: E0315 00:46:19.216346    2702 cni.go:273] Error deleting network: required env variables missing

[bboreham]

I managed to achieve error-message parity with you, @jbeda:

Mar 15 12:06:39 brya-0 kubelet[29821]: W0315 12:06:39.393959   29821 docker_sandbox.go:234] NetworkPlugin cni failed on the status hook for pod "kube-dns-3913472980-f0zpk_kube-system": Unexpected command output nsenter: cannot open : No such file or directory
Mar 15 12:06:39 brya-0 kubelet[29821]:  with error: exit status 1
Mar 15 12:06:39 brya-0 kubelet[29821]: 2017/03/15 12:06:39 CNI_NETNS env variable missing
Mar 15 12:06:39 brya-0 kubelet[29821]: E0315 12:06:39.399982   29821 cni.go:273] Error deleting network: required env variables missing
Mar 15 12:06:39 brya-0 kubelet[29821]: E0315 12:06:39.400960   29821 remote_runtime.go:109] StopPodSandbox "d8bb85a1bd54056f5698561f951204163423ee376235fd7777c193e7c4b7a77c" from runtime service failed: rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod "kube-dns-3913472980-f0zpk_kube-system" network: required env variables missing"

All of this derives from the kube-dns infrastructure container being dead:

# docker ps -a | grep kube-dns-3913472980-f0zpk_kube-system
d8bb85a1bd54        gcr.io/google_containers/pause-amd64:3.0   "/pause"                 About an hour ago   Exited (0) About an hour ago                       k8s_POD_kube-dns-3913472980-f0zpk_kube-system_7c2bbc2a-096e-11e7-a6df-42010a800004_0

It looks like kubelet is getting hung up on the repeated failure CNI_NETNS env variable missing, when what it should really do is blow the whole thing away and create a new pod.

That environment variable used to be mandated by the CNI spec, but the spec was changed a couple of months back to allow it to be unset, and when I updated the Weave Net plugin to honour that change it all works.

That update to the plugin is in #2850 but is not merged yet.

[bboreham]

@luxas I think I'm in parallel with you - see branch luxas-add-head-manifest

I think the file should be named '1.6' rather than 'HEAD'

[bboreham]

Merged via the 1.9 branch at ca22ecd
I then renamed the file at 9da5cee to emphasise this is the new file for Kubernetes 1.6

Thanks!