diff --git a/content/en/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm.md b/content/en/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm.md index 58f4f4652a907..b1de684f28994 100644 --- a/content/en/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm.md +++ b/content/en/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm.md @@ -408,7 +408,7 @@ be advised that this is modifying a design principle of the Linux distribution. ## `kubeadm upgrade plan` prints out `context deadline exceeded` error message -This error message is shown when upgrading a Kubernetes cluster with `kubeadm` in the case of running an external etcd. This is not a critical bug and happens because older versions of kubeadm perform a version check on the external etcd cluster. You can proceed with `kubeadm upgrade apply ...`. +This error message is shown when upgrading a Kubernetes cluster with `kubeadm` in the case of running an external etcd. This is not a critical bug and happens because older versions of kubeadm perform a version check on the external etcd cluster. You can proceed with `kubeadm upgrade apply ...`. This issue is fixed as of version 1.19. @@ -420,3 +420,20 @@ To workaround the issue, re-mount the `/var/lib/kubelet` directory after perform This is a regression introduced in kubeadm 1.15. The issue is fixed in 1.20. +## Cannot use the metrics-server securely in a kubeadm cluster + +In a kubeadm cluster, the [metrics-server](https://github.com/kubernetes-sigs/metrics-server) +can be used insecurely by passing the `--kubelet-insecure-tls` to it. This is not recommended for production clusters. + +If you want to use TLS between the metrics-server and the kubelet there is a problem, +since kubeadm deploys a self-signed serving certificate for the kubelet. This can cause the following errors +on the side of the metrics-server: +``` +x509: certificate signed by unknown authority +x509: certificate is valid for IP-foo not IP-bar +``` + +See [Enabling signed kubelet serving certificates](/docs/tasks/administer-cluster/kubeadm/kubeadm-certs#kubelet-serving-certs) +to understand how to configure the kubelets in a kubeadm cluster to have properly signed serving certificates. + +Also see [How to run the metrics-server securely](https://github.com/kubernetes-sigs/metrics-server/blob/master/FAQ.md#how-to-run-metrics-server-securely). diff --git a/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-certs.md b/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-certs.md index dc7af4a32941b..60329db71b57f 100644 --- a/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-certs.md +++ b/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-certs.md @@ -207,3 +207,69 @@ After a certificate is signed using your preferred method, the certificate and t Kubeadm does not support rotation or replacement of CA certificates out of the box. For more information about manual rotation or replacement of CA, see [manual rotation of CA certificates](/docs/tasks/tls/manual-rotation-of-ca-certificates/). + +## Enabling signed kubelet serving certificates {#kubelet-serving-certs} + +By default the kubelet serving certificate deployed by kubeadm is self-signed. +This means a connection from external services like the +[metrics-server](https://github.com/kubernetes-sigs/metrics-server) to a +kubelet cannot be secured with TLS. + +To configure the kubelets in a new kubeadm cluster to obtain properly signed serving +certificates you must pass the following minimal configuration to `kubeadm init`: + +```yaml +apiVersion: kubeadm.k8s.io/v1beta2 +kind: ClusterConfiguration +--- +apiVersion: kubelet.config.k8s.io/v1beta1 +kind: KubeletConfiguration +serverTLSBootstrap: true +``` + +If you have already created the cluster you must adapt it by doing the following: +- Find and edit the `kubelet-config-x.yy` ConfigMap under `kube-system` to include +the field `serverTLSBootstrap: true` under the `config` key +- On each node, add the `serverTLSBootstrap: true` field in `/var/lib/kubelet/config.yaml` +and restart the kubelet with `systemctl restart kubelet` + +The field `serverTLSBootstrap: true` will enable the bootstrap of kubelet serving +certificates by requesting them from the `certificates.k8s.io` API. One known limitation +is that the CSRs (Certificate Signing Requests) for these certificates cannot be automatically +approved by the default signer in the kube-controller-manager - +[`kubernetes.io/kubelet-serving`](https://kubernetes.io/docs/reference/access-authn-authz/certificate-signing-requests/#kubernetes-signers). +This will require action from the user or a third party controller. + +These CSRs can be viewed using: + +```shell +kubectl get csr +NAME AGE SIGNERNAME REQUESTOR CONDITION +csr-9wvgt 112s kubernetes.io/kubelet-serving system:node:worker-1 Pending +csr-lz97v 1m58s kubernetes.io/kubelet-serving system:node:control-plane-1 Pending +``` + +To approve them you can do the following: +```shell +kubectl certificate approve +``` + +By default, these serving certificate will expire after one year. Kubeadm sets the +`KubeletConfiguration` field `rotateCertificates` to `true`, which means that close +to expiration a new set of CSRs for the serving certificates will be created and must +be approved to complete the rotation. To understand more see +[Certificate Rotation](/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/#certificate-rotation). + +If you are looking for a solution for automatic approval of these CSRs it is recommended +that you contact your cloud provider and ask if they have a CSR signer that verifies +the node identity with an out of band mechanism. + +{{% thirdparty-content %}} + +Third party custom controllers can be used: +- [kubelet-rubber-stamp](https://github.com/kontena/kubelet-rubber-stamp) + +Such a controller is not a secure mechanism unless it not only verifies the CommonName +in the CSR but also verifies the requested IPs and domain names. This would prevent +a malicious actor that has access to a kubelet client certificate to create +CSRs requesting serving certificates for any IP or domain name.