Skip to content

Commit

Permalink
kubeadm: add guide for kubelet serving certs and metrics-server
Browse files Browse the repository at this point in the history
Include a new section of the kubeadm certificate management page
to talk about kubelet serving certificates and how to make
them signed. Also include a note about using secure connection
with the metrics-server.
  • Loading branch information
neolit123 committed Apr 3, 2021
1 parent 7f2b282 commit 1434983
Show file tree
Hide file tree
Showing 2 changed files with 84 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -408,7 +408,7 @@ be advised that this is modifying a design principle of the Linux distribution.

## `kubeadm upgrade plan` prints out `context deadline exceeded` error message

This error message is shown when upgrading a Kubernetes cluster with `kubeadm` in the case of running an external etcd. This is not a critical bug and happens because older versions of kubeadm perform a version check on the external etcd cluster. You can proceed with `kubeadm upgrade apply ...`.
This error message is shown when upgrading a Kubernetes cluster with `kubeadm` in the case of running an external etcd. This is not a critical bug and happens because older versions of kubeadm perform a version check on the external etcd cluster. You can proceed with `kubeadm upgrade apply ...`.

This issue is fixed as of version 1.19.

Expand All @@ -420,3 +420,20 @@ To workaround the issue, re-mount the `/var/lib/kubelet` directory after perform

This is a regression introduced in kubeadm 1.15. The issue is fixed in 1.20.

## Cannot use the metrics-server securely in a kubeadm cluster

In a kubeadm cluster, the [metrics-server](https://github.com/kubernetes-sigs/metrics-server)
can be used insecurely by passing the `--kubelet-insecure-tls` to it. This is not recommended for production clusters.

If you want to use TLS between the metrics-server and the kubelet there is a problem,
since kubeadm deploys a self-signed serving certificate for the kubelet. This can cause the following errors
on the side of the metrics-server:
```
x509: certificate signed by unknown authority
x509: certificate is valid for IP-foo not IP-bar
```

See [Enabling signed kubelet serving certificates](/docs/tasks/administer-cluster/kubeadm/kubeadm-certs#kubelet-serving-certs)
to understand how to configure the kubelets in a kubeadm cluster to have properly signed serving certificates.

Also see [How to run the metrics-server securely](https://github.com/kubernetes-sigs/metrics-server/blob/master/FAQ.md#how-to-run-metrics-server-securely).
66 changes: 66 additions & 0 deletions content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-certs.md
Original file line number Diff line number Diff line change
Expand Up @@ -207,3 +207,69 @@ After a certificate is signed using your preferred method, the certificate and t
Kubeadm does not support rotation or replacement of CA certificates out of the box.

For more information about manual rotation or replacement of CA, see [manual rotation of CA certificates](/docs/tasks/tls/manual-rotation-of-ca-certificates/).

## Enabling signed kubelet serving certificates {#kubelet-serving-certs}

By default the kubelet serving certificate deployed by kubeadm is self-signed.
This means a connection from external services like the
[metrics-server](https://github.com/kubernetes-sigs/metrics-server) to a
kubelet cannot be secured with TLS.

To configure the kubelets in a new kubeadm cluster to obtain properly signed serving
certificates you must pass the following minimal configuration to `kubeadm init`:

```yaml
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
serverTLSBootstrap: true
```

If you have already created the cluster you must adapt it by doing the following:
- Find and edit the `kubelet-config-x.yy` ConfigMap under `kube-system` to include
the field `serverTLSBootstrap: true` under the `config` key
- On each node, add the `serverTLSBootstrap: true` field in `/var/lib/kubelet/config.yaml`
and restart the kubelet with `systemctl restart kubelet`

The field `serverTLSBootstrap: true` will enable the bootstrap of kubelet serving
certificates by requesting them from the `certificates.k8s.io` API. One known limitation
is that the CSRs (Certificate Signing Requests) for these certificates cannot be automatically
approved by the default signer in the kube-controller-manager -
[`kubernetes.io/kubelet-serving`](https://kubernetes.io/docs/reference/access-authn-authz/certificate-signing-requests/#kubernetes-signers).
This will require action from the user or a third party controller.

These CSRs can be viewed using:

```shell
kubectl get csr
NAME AGE SIGNERNAME REQUESTOR CONDITION
csr-9wvgt 112s kubernetes.io/kubelet-serving system:node:worker-1 Pending
csr-lz97v 1m58s kubernetes.io/kubelet-serving system:node:control-plane-1 Pending
```

To approve them you can do the following:
```shell
kubectl certificate approve <CSR-name>
```

By default, these serving certificate will expire after one year. Kubeadm sets the
`KubeletConfiguration` field `rotateCertificates` to `true`, which means that close
to expiration a new set of CSRs for the serving certificates will be created and must
be approved to complete the rotation. To understand more see
[Certificate Rotation](/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/#certificate-rotation).

If you are looking for a solution for automatic approval of these CSRs it is recommended
that you contact your cloud provider and ask if they have a CSR signer that verifies
the node identity with an out of band mechanism.

{{% thirdparty-content %}}

Third party custom controllers can be used:
- [kubelet-rubber-stamp](https://github.com/kontena/kubelet-rubber-stamp)

Such a controller is not a secure mechanism unless it not only verifies the CommonName
in the CSR but also verifies the requested IPs and domain names. This would prevent
a malicious actor that has access to a kubelet client certificate to create
CSRs requesting serving certificates for any IP or domain name.

0 comments on commit 1434983

Please sign in to comment.