metrics-server: enable authenticationTokenWebhook

[sdarwin]

Per kubernetes-sigs/metrics-server#133 , the metrics-server will not function on kops without these fixes.

1. --kubelet-insecure-tls added to metrics-server
2. authenticationTokenWebhook set on the kubelet

kops edit cluster

kubelet:
     anonymousAuth: false
     authenticationTokenWebhook: true
     authorizationMode: Webhook

Although, minimalistically this is sufficient:

kubelet:
     anonymousAuth: false
     authenticationTokenWebhook: true

Since running a metrics server is pretty common, eliminating the extra steps would be convenient.

Could the settings of "authenticationTokenWebhook: true" and "authorizationMode: Webhook" be configured by default on kops, so the end-user doesn't encounter unexpected errors when deploying metrics?

[sdarwin]

Investigating the issue a bit more, a question could be "Do other k8s installers also have this same problem? How do they solve it?"

1. Kubespray

Kubespray enables both Webhook authentication and authorization for the kubelet. Here's the kubelet-config.

root@node1:~# cat /etc/kubernetes/kubelet-config.yaml

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
nodeStatusUpdateFrequency: "10s"
failSwapOn: True
authentication:
  anonymous:
    enabled: false
  webhook:
    enabled: True
  x509:
    clientCAFile: /etc/kubernetes/ssl/ca.crt
authorization:
  mode: Webhook
staticPodPath: /etc/kubernetes/manifests
cgroupDriver: cgroupfs
maxPods: 110
address: 172.31.6.69
readOnlyPort: 0
kubeletCgroups: /systemd/system.slice
clusterDomain: cluster.local
clusterDNS:
- 169.254.25.10
kubeReserved:
  cpu: 200m
  memory: 512M

2. eksctl

eksctl also enables Webhook authentication and authorization for the kubelet. Here's the kubelet-config.

root@node1:~# cat /etc/eksctl/kubelet.yaml

address: 0.0.0.0
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /etc/eksctl/ca.crt
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
cgroupDriver: cgroupfs
clusterDNS:
- 10.100.0.10
clusterDomain: cluster.local
featureGates:
  RotateKubeletServerCertificate: true
kind: KubeletConfiguration
serverTLSBootstrap: true

---

It might be kops is aiming for a generic kubernetes installation, which means keeping standard kubernetes default settings. That's potentially a reason to not enable Webhook. However, if --anonymous-auth=false is set, it's indicating "You can't connect anonymously, you must supply authentication credentials". And how will that be done? Most often with service tokens. So, tokens should be allowed. Otherwise, an alternative could be to set --anonymous-auth back to the default of "true".

[richard-scott]

Wow, I had assumed that this would have been fixed in kops v1.13.0 but alas it is not, well not for me on AWS.

This "feature" nearly made me change from using kops to something else. I wanted to use the Horizontal Pod Autoscaler and Cluster Autoscaling but could not get the metrics to work and it constantly said ""!

For anyone else wanting an easier way to build a new cluster and add these options to get the metrics server up and running then you can do the following during your build:

$ kops create cluster [...]
$ kops [...] set cluster spec.kubelet.authenticationTokenWebhook=true
$ kops [...] set cluster spec.kubelet.authorizationMode=Webhook
$ kops [...] update cluster --yes

Adding them at cluster creation time means you do not have to do a rolling update to activate them across the cluster. It took me a few hours to figure that one out ;-)

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[sdarwin]

/remove-lifecycle stale

[tsahiduek]

Can anyone respond whether the defaults for authenticationTokenWebhook and authorizationMode should be changed according to @richard-scott / @sdarwin suggestions --> #7200 (comment) ?

I've experienced this in kops 1.15.1

[zarqman]

I can confirm that changing the value of those two settings is still required for either metrics-server or kubernetes-dashboard to function on kubernetes 1.17 (and kops 1.17.0-beta.1). I definitely recommend changing the default values as suggested by OP.

[rafilkmp3]

How i can create cluster with enabled ?

authenticationTokenWebhook: true
authorizationMode: Webhook

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[sdarwin]

/remove-lifecycle stale