-
Notifications
You must be signed in to change notification settings - Fork 717
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
/var/lib/kubelet/pki/kubelet.crt not renewed with kubeadm alpha certs renew all
#2186
Comments
kubeadm alpha certs renew all
hello, please see https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/
this certificate is rotated out of band and automatically by the kubelet process. i believe it should happen around 80% of the total cert validity duration. also see this issue that we closed a couple of days ago: i'm going to close this but let me know if you have further questions. /close |
@neolit123: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
@neolit123 Thanks for your answer!
Does that mean, if I changed the |
that is not what the docs describe. depends on what you want to change, but the docs instruct you to link your kubelet.conf to the https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/ |
yes, I got it. Just put the The link in kubelet.conf is added since the changelog described that ;) Thanks again. |
Hello In my opinion, /var/lib/kubelet/pki/kubelet.crt is not an obsolete file. It seems as a bundle PEM that contains Self signed CA and associated Server certificate that is used by kubelet to serve "contents". It is typically used by metrics-server for example. It is actually a real problem that kubelet uses self signed certificates to serve "metrics" as it enforces the usage of --kubelet-insecure-tls in metrics-server deployment. see https://github.com/kubernetes/kubeadm/issues/1223 |
@blepoutr Thank for explaining about /var/lib/kubelet/pki/kubelet.crt, Can you just verify that the kubelet.crt is not used by the kubelet to talk to the API server, but used when metrics are served? Is there documentation? |
@pmmalinov01, I solved my issue, by forcing kubelet not to use selfsigned certificate as Serving Certificate (as per doc : https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/#client-and-serving-certificates) Indeed, I have activated the RotateKubeletServerCertificate=true in order to have kubelet requesting a Serving certificate to K8S itself. Besides, it will automatically do it, when a new node is installed, or if the previous existing Serving certificate is going to expire. The good news, is that it allows me to have CA check between Metrics server and kubelet and so I can stop using --kubelet-insecure-tls. The only drawback is that I (as K8S adminstrator) needs to approve the CSR manually ( by kubectl csr approve ...), before the certificate is generated and retrieved by kubelet. I amy consider using such tools to overcome that : https://github.com/kontena/kubelet-rubber-stamp |
I feel like people are talking about two different certificates in /var/lib/kubelet/pki directory. In my 1.18 k8s system I have the following in that directory: ll /var/lib/kubelet/pki/ The kubelet-client-current.pem is what kubelet.conf pointing to. But I just couldn't find anybody else is using kubelet.crt/key (it's self signed non-CA cert). Maybe they are used initially and then later on rotated to use the kubelet-client-current.pem? |
Dig a little bit more ... kubelet-client-current.pem contains a client cert, seems to be used for kubelet to securely connect to apiserver etc. kubelet.crt is a server cert, seems to be for components that try to securely connect to kubelet to verify kubelet's identity. So they are for different purposes. In my system, there seems to be no components are talking to kubelet (only kubelet talks to other components). That's probably why I don't find other components are using kubelet.crt/key. |
|
both the kublet serving and client certs can be auto-rotated. for more info: |
I have similar problem. My kubelet.crt is expired. i don't know a suitable way to renew it. kl get no Is there a way to renew the kubelet.crt/key? I read above conversation but still didn't get any concrete answer to renew my kubelet.crt/key |
Did you found a solution? I still have this problem |
there is a guide for recovering the kubelet failed state due to outdated certs here: the documentation there is currently being edited and will be available here: |
this is not working, removed kubelet.conf and certs from /var/lib/kubelet/pki/*, and replace with kubelet.conf generated as mentioned in above link and restarted the kubelet service, but still it is failing and not creating the certs. |
Anyway I resolved with (in each node)
|
BUG REPORT
Versions
kubeadm version (use
kubeadm version
):kubeadm version: &version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-11T18:12:12Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}
Environment:
kubectl version
):kubectl version
Cloud provider or hardware configuration:
Hardware
OS (e.g. from /etc/os-release):
Ubuntu 16.04..6 LTS
Kernel (e.g.
uname -a
):Linux eins 4.15.0-88-generic #88~16.04.1-Ubuntu SMP Wed Feb 12 04:19:15 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Others:
What happened?
When running
kubeadm alpha certs renew all
, the cert in/var/lib/kubelet/pki/kubelet.crt
is not renewed. See the monitoring screenshot below.However, I am not sure if this file is really used in the cluster.
If this is no bug and work as designed, I suggest to delete the file or at least have a documentation, which certs (in which path) are used.
Maybe this is a side effect of #1223
The text was updated successfully, but these errors were encountered: