Kubelet can't running after renew certificates #2054

pytimer · 2020-03-05T06:25:46Z

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

kubeadm version (use kubeadm version):

kubeadm version: &version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.2", GitCommit:"59603c6e503c87169aea6106f57b9f242f64df89", GitTreeState:"clean", BuildDate:"2020-01-18T23:27:49Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.2", GitCommit:"59603c6e503c87169aea6106f57b9f242f64df89", GitTreeState:"clean", BuildDate:"2020-01-18T23:30:10Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}
The connection to the server localhost:8080 was refused - did you specify the right host or port?

Cloud provider or hardware configuration:
bare-metal
OS (e.g. from /etc/os-release):

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

Kernel (e.g. uname -a):

Linux k8s-236 3.10.0-957.12.2.el7.x86_64 #1 SMP Mon May 20 08:41:20 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux

Others:

What happened?

I using the kubeadm command to renew the cluster certificates, the kubeadm alpha certs renew all works well, and i found the certificates have been changed. But when restart kubelet, kubelet can't running, it exited. I don't know why?
How to do renew control plane certficates and kubelet certficates, which docs can i reference.

What you expected to happen?

Renew all cluster certificates successfully and kubelet running.

How to reproduce it (as minimally and precisely as possible)?

create cluster via kubeadm.
enable kubelet dynamic config.
update the host datetime, make the certificates expiration.
run kubeadm alpha certs renew all command
systemctl restart kubelet

Anything else we need to know?

kubelet log:

server.go:821] Client rotation is on, will bootstrap in background
bootstrap.go:265] part of the existing bootstrap client certificate is expired: 2021-03-04 14:19:56 +0000 UTC
systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a
server.go:273] failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory

The text was updated successfully, but these errors were encountered:

SataQiu · 2020-03-05T11:35:53Z

/cc @fabriziopandini
I have reproduced this problem. After the certificates expire, execute kubeadm alpha certs renew all, and kubelet will not be able to restart normally.

neolit123 · 2020-03-05T12:12:29Z

systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a
server.go:273] failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory

the bootstrap-kubelet.conf missing should not be an issue as long as there is a kubelet.conf, see the kubelet flags:

--bootstrap-kubeconfig string
--
| Path to a kubeconfig file that will be used to get client certificate for kubelet. If the file specified by --kubeconfig does not exist, the bootstrap kubeconfig is used to request a client certificate from the API server. On success, a kubeconfig file referencing the generated client certificate and key is written to the path specified by --kubeconfig. The client certificate and key file will be stored in the directory pointed by --cert-dir.

i'm going to test this as well.

neolit123 · 2020-03-05T12:21:10Z

enable kubelet dynamic config.

please note that this feature is not really supported by kubeadm anymore and we don't have e2e tests for it...

pytimer · 2020-03-05T13:08:43Z

please note that this feature is not really supported by kubeadm anymore and we don't have e2e tests for it...

I enable kubelet dynamic config manually, not use kubeadm command.

pytimer · 2020-03-05T13:10:10Z

systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a
server.go:273] failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory

the bootstrap-kubelet.conf missing should not be an issue as long as there is a kubelet.conf, see the kubelet flags:

Did kubelet use kubelet.conf first, if error it use bootstrap-kubelet.conf retry?

neolit123 · 2020-03-05T13:28:48Z

Did kubelet use kubelet.conf first, if error it use bootstrap-kubelet.conf retry?

there should not be an error if the bootstrap-kubelet.conf is missing, but the kubelet.conf is present.

neolit123 · 2020-03-05T13:52:27Z

@pytimer @SataQiu i cannot reproduce the issue. here are my steps:

kubeadm init ... --kubernetes-version=v1.17.3
kubeadm alpha certs renew all
systemctl restart kubelet

if you are doing the steps exactly like so:

update the host datetime, make the certificates expiration.
run kubeadm alpha certs renew all command
systemctl restart kubelet

by running certs renew you are generating certs with the "future date".
i don't think this is correct.

/priority awaiting-more-evidence

pytimer · 2020-03-05T15:01:18Z

@pytimer @SataQiu i cannot reproduce the issue. here are my steps:
kubeadm init ... --kubernetes-version=v1.17.3
kubeadm alpha certs renew all
systemctl restart kubelet
if you are doing the steps exactly like so:
update the host datetime, make the certificates expiration.
run kubeadm alpha certs renew all command
systemctl restart kubelet
by running certs renew you are generating certs with the "future date".
i don't think this is correct.

Update the host datetime to assume certificate expired.Why did you think this is wrong, can you explain it, i don't understand it. If this is wrong, which way to test certs renew command works well? Thanks.

neolit123 · 2020-03-05T16:25:03Z

Update the host datetime to assume certificate expired.

ok, i was able to understand better what you are trying to do.

update the host datetime, make the certificates expiration.

on a running cluster this means the kubelet now has an invalid client certificate for the API server stored in the file kubelet.conf.

run kubeadm alpha certs renew all command

it's important to note that this command does not update kubelet.conf. the client credentials in there are managed by the kubelet.

systemctl restart kubelet

if the year has changed and if you run this command the kubelet will find an outdated certificate in the kubelet.conf file and it will start looking for bootstrap-kubelet.conf. but nowadays we delete this file after TLS bootstrap for security reasons.

so the kubelet will fail because it has not credentials to connect to the API.

normally this will not happen because the kubelet certificate manager has logic to monitor for certificate expiration and it will auto-rotate your client certificates once ~70% of the expiration period is reached. by forcing a date you are bypassing this mechanism and you end up with a certificate which was not rotated.

If this is wrong, which way to test certs renew command works well?

i suggest you don't touch your system date.
modify the kubeadm source code to issue certificates that expire after 10 minutes and try to renew them after expiration. this will let you verify the certs renew works.
but again, kubelet.conf is not managed by kubeadm.

hope this explains the issue.

pytimer · 2020-03-06T06:49:15Z

Thanks @neolit123

modify the kubeadm source code to issue certificates that expire after 10 minutes and try to renew them after expiration.

Kubeadm certs renew command works well.

normally this will not happen because the kubelet certificate manager has logic to monitor for certificate expiration and it will auto-rotate your client certificates once ~70% of the expiration period is reached.

@neolit123 Update the host time make kubelet client certificates expriation time between 70%~100%, i found kubelet can't rotate certificates, the new certificates created until kubelet restarted. Do you know reason? I can't find some documents about it.

neolit123 · 2020-03-06T13:01:45Z

@neolit123 Update the host time make kubelet client certificates expriation time between 70%~100%, i found kubelet can't rotate certificates, the new certificates created until kubelet restarted. Do you know reason? I can't find some documents about it.

i don't think that updating the host time is a valid approach for testing. this could be tripping the kubelet client cert rotation. but yes, the exact process is not documented fully:
https://kubernetes.io/docs/tasks/tls/certificate-rotation/

neolit123 · 2020-03-06T13:23:50Z

if the kubelet client certificate expired for some reason you can renew it manually. look at the contents of openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -text and search online how to use the /etc/kubernetes/pki/ca.{key|crt} to sign a new one.

dimm0 · 2020-03-31T22:04:38Z

I've hit this in 1.17.1, after master node rebooted
/var/lib/kubelet/pki/kubelet-client-current.pem is renewed. kubeadm alpha certs check-expiration says all are renewed. Is there anything else to renew?

Please help!

dimm0 · 2020-03-31T22:25:34Z

I manually copied the renewed cert and key from /var/lib/kubelet/pki/kubelet-client-current.pem to /etc/kubernetes/kubelet.conf, and it started. Phew!

neolit123 · 2020-03-31T22:27:00Z

instead of copying the cert/key inside the kubeconfig, see what is suggested here:
https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/#check-certificate-expiration

Warning:
On nodes created with kubeadm init, prior to kubeadm version 1.17...

dimm0 · 2020-03-31T22:29:50Z

Ahh! I see, thanks!

wfsly · 2021-08-09T13:46:38Z

I got a cluster, all certs are expired except ca.crt, so control plane are also down.
I fix it with renew certs and recreate bootstrap-kubelet.conf.

kubeadm alpha certs renew all on each node. and restart kubelet, it doesn't work in my case
kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/pki/ca.crt --embed-certs=true --server=https://%v:%v --kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf
on master node, export BOOTSTRAP_TOKEN=$(echo {{ <node_ip> }} | md5sum | awk '{print $1}' ). on worker node, export BOOTSTRAP_TOKEN=$(kubeadm token create --description kubelet-bootstrap-token --groups system:bootstrappers:$(echo <nodeIP> | md5sum | awk '{print $1}' ) --kubeconfig /root/.kube/config)
kubectl config set-credentials kubelet-bootstrap --token=${BOOTSTRAP_TOKEN} --kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf
kubectl config set-context default --cluster=kubernetes --user=kubelet-bootstrap --kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf
kubectl config use-context default --kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf
systemctl restart kubelet

nodesocket · 2022-02-04T04:03:44Z

Just ran into this and blocked. Cluster is currently down. Running Kubernetes v1.15.12.

I ran the following on the master/control plane:

ubuntu@kubernetes-master:~/.kube$ sudo kubeadm alpha certs renew all

certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healtcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed

Then rebooted the master/control plane.

However, kubelet is failing to start still:

Feb  4 04:01:52 kubernetes-master systemd[1]: Started Kubernetes systemd probe.
Feb  4 04:01:52 kubernetes-master kubelet[7251]: I0204 04:01:52.544916    7251 server.go:425] Version: v1.15.12
Feb  4 04:01:52 kubernetes-master kubelet[7251]: I0204 04:01:52.545090    7251 plugins.go:103] No cloud provider specified.
Feb  4 04:01:52 kubernetes-master kubelet[7251]: I0204 04:01:52.545103    7251 server.go:789] Client rotation is on, will bootstrap in background
Feb  4 04:01:52 kubernetes-master kubelet[7251]: E0204 04:01:52.546511    7251 bootstrap.go:263] Part of the existing bootstrap client certificate is expired: 2022-01-26 02:08:58 +0000 UTC
Feb  4 04:01:52 kubernetes-master kubelet[7251]: F0204 04:01:52.546545    7251 server.go:273] failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory
Feb  4 04:01:52 kubernetes-master systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Feb  4 04:01:52 kubernetes-master systemd[1]: kubelet.service: Failed with result 'exit-code'

All the certificates are showing as valid:

ubuntu@kubernetes-master:~/.kube$ sudo kubeadm alpha certs check-expiration
CERTIFICATE                EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
admin.conf                 Feb 04, 2023 04:00 UTC   364d            no
apiserver                  Feb 04, 2023 04:00 UTC   364d            no
apiserver-etcd-client      Feb 04, 2023 04:00 UTC   364d            no
apiserver-kubelet-client   Feb 04, 2023 04:00 UTC   364d            no
controller-manager.conf    Feb 04, 2023 04:00 UTC   364d            no
etcd-healthcheck-client    Feb 04, 2023 04:00 UTC   364d            no
etcd-peer                  Feb 04, 2023 04:00 UTC   364d            no
etcd-server                Feb 04, 2023 04:00 UTC   364d            no
front-proxy-client         Feb 04, 2023 04:00 UTC   364d            no
scheduler.conf             Feb 04, 2023 04:00 UTC   364d            no

What am I missing? Do I need to manually update a configuration file?

neolit123 · 2022-02-04T10:39:19Z

Kubeadm does not manage rotation of kubelet.conf. This might help you https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm/#kubelet-client-cert

nodesocket · 2022-02-04T23:00:51Z

@neolit123 thanks for the reply.

The fix for me was to copy the contents of /etc/kubernetes/admin.conf specifically the keys client-certificate-data and client-key-data and paste those new strings into the file /etc/kubernetes/kubelet.conf under the same keys. Then just a sudo service kubelet restart.

Why is this not more obvious when invoking kubeadm alpha certs renew all? If it's not going to update kubelet.conf there should be a warning message or something. This costs me a few hours last night of time and downtime.

neolit123 · 2022-02-05T00:26:38Z

The fix for me was to copy the contents of /etc/kubernetes/admin.conf specifically the keys client-certificate-data and client-key-data and paste those new strings into the file /etc/kubernetes/kubelet.conf under the same keys. Then just a sudo service kubelet restart.

this is not ideal, as it grants the kubelet client super admin credentials....you are also hardcoding non-rotatable credentials.
the link i've mentioned has steps to generate better scoped credentials:
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm/#kubelet-client-cert

Why is this not more obvious when invoking kubeadm alpha certs renew all? If it's not going to update kubelet.conf there should be a warning message or something. This costs me a few hours last night of time and downtime.

this is documented:
https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/

kubelet.conf is not included in the list above because kubeadm configures kubelet for automatic certificate renewal with rotatable certificates under /var/lib/kubelet/pki. To repair an expired kubelet client certificate see Kubelet client certificate rotation fails.

but back to the original problem, for the credentials to expire something must have gotten wrong on the node...or you had hardcoded kubelet.conf credentials instead of pointing to the rotatable symlink (explained in the docs).
the kubelet rotates the symlinked files every ~8 months.

EDIT: also check this warning in the docs:

On nodes created with kubeadm init, prior to kubeadm version 1.17, there is a bug where you manually have to modify the contents of kubelet.conf. After kubeadm init finishes, you should update kubelet.conf to point to the rotated kubelet client certificates, by replacing client-certificate-data and client-key-data with:

nodesocket · 2022-02-05T01:24:32Z

@neolit123 thanks for the detailed reply.

I think the core problem was the original Kubernetes cluster was really old and upgraded via kubeadm a few times. I'm wanting to completely rebuild from scratch using 1.21.5. I assume doing this will use the correct rotatable symlinks and update kubelet.conf to point to the those symlinks?

This is of-course an on-prem cluster (not cloud managed) and created using kubeadm.

sudo kubeadm init --config /tmp/kubernetes-master-config.yaml | sudo tee /tmp/kubeadm-init.log

neolit123 · 2022-02-05T13:09:08Z

Yes, it should.

…

On Feb 5, 2022 03:24, "Justin Keller" ***@***.***> wrote: @neolit123 <https://github.com/neolit123> thanks for the detailed reply. I think the core problem was the original Kubernetes cluster was really old and upgraded via kubeadm a few times. I'm wanting to completely rebuild from scratch using 1.21.5. I assume doing this will use the correct rotatable symlinks and update kubelet.conf to point to the those symlinks? — Reply to this email directly, view it on GitHub <#2054 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACRATG4FVKIIGTL5ZU2OR3UZR35XANCNFSM4LCC2CQQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

nikunj-diwan · 2023-08-09T05:05:44Z

Hi @neolit123,

Can you please help me.

was faced certificate expiry issue and renewed it properly but /var/lib/kubelet/pki/kubelet-client-current.pem certificate doesn't updated so, as per the steps mentioned on below Kubernetes link I did the steps to regenerated kubelet.conf file and I made a mistake were I have changed node name in newly generated kubelet.conf file due to that not it's giving permission related error s and node doesn't coming into ready state.

Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.1"
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.2"
kubeadm version: &version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.1"
Kubernetes v1.27.1
Operating system: Red Hat Enterprise Linux Server release 7.9 (Maipo)
crictl version v1.26.0
Docker version: 23.0.3

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm/#kubelet-client-cert

kubelet error.txt

Thanks,
Nikunj

k8s-ci-robot added the priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. label Mar 5, 2020

neolit123 closed this as completed Mar 5, 2020

pacoxu mentioned this issue Jun 3, 2021

systemd on Ubuntu - Current command vanished from the unit file kubernetes/kubernetes#102563

Closed

spowelljr mentioned this issue Apr 7, 2023

Renew expired kubeadm certs kubernetes/minikube#16249

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kubelet can't running after renew certificates #2054

Kubelet can't running after renew certificates #2054

pytimer commented Mar 5, 2020

SataQiu commented Mar 5, 2020 •

edited

Loading

neolit123 commented Mar 5, 2020

neolit123 commented Mar 5, 2020

pytimer commented Mar 5, 2020

pytimer commented Mar 5, 2020

neolit123 commented Mar 5, 2020 •

edited

Loading

neolit123 commented Mar 5, 2020

pytimer commented Mar 5, 2020

neolit123 commented Mar 5, 2020

pytimer commented Mar 6, 2020

neolit123 commented Mar 6, 2020

neolit123 commented Mar 6, 2020 •

edited

Loading

dimm0 commented Mar 31, 2020

dimm0 commented Mar 31, 2020

neolit123 commented Mar 31, 2020

dimm0 commented Mar 31, 2020

wfsly commented Aug 9, 2021 •

edited

Loading

nodesocket commented Feb 4, 2022 •

edited

Loading

neolit123 commented Feb 4, 2022 via email

nodesocket commented Feb 4, 2022

neolit123 commented Feb 5, 2022 •

edited

Loading

nodesocket commented Feb 5, 2022 •

edited

Loading

neolit123 commented Feb 5, 2022 via email

nikunj-diwan commented Aug 9, 2023 •

edited

Loading

Kubelet can't running after renew certificates #2054

Kubelet can't running after renew certificates #2054

Comments

pytimer commented Mar 5, 2020

Is this a BUG REPORT or FEATURE REQUEST?

Versions

What happened?

What you expected to happen?

How to reproduce it (as minimally and precisely as possible)?

Anything else we need to know?

SataQiu commented Mar 5, 2020 • edited Loading

neolit123 commented Mar 5, 2020

neolit123 commented Mar 5, 2020

pytimer commented Mar 5, 2020

pytimer commented Mar 5, 2020

neolit123 commented Mar 5, 2020 • edited Loading

neolit123 commented Mar 5, 2020

pytimer commented Mar 5, 2020

neolit123 commented Mar 5, 2020

pytimer commented Mar 6, 2020

neolit123 commented Mar 6, 2020

neolit123 commented Mar 6, 2020 • edited Loading

dimm0 commented Mar 31, 2020

dimm0 commented Mar 31, 2020

neolit123 commented Mar 31, 2020

dimm0 commented Mar 31, 2020

wfsly commented Aug 9, 2021 • edited Loading

nodesocket commented Feb 4, 2022 • edited Loading

neolit123 commented Feb 4, 2022 via email

nodesocket commented Feb 4, 2022

neolit123 commented Feb 5, 2022 • edited Loading

nodesocket commented Feb 5, 2022 • edited Loading

neolit123 commented Feb 5, 2022 via email

nikunj-diwan commented Aug 9, 2023 • edited Loading

SataQiu commented Mar 5, 2020 •

edited

Loading

neolit123 commented Mar 5, 2020 •

edited

Loading

neolit123 commented Mar 6, 2020 •

edited

Loading

wfsly commented Aug 9, 2021 •

edited

Loading

nodesocket commented Feb 4, 2022 •

edited

Loading

neolit123 commented Feb 5, 2022 •

edited

Loading

nodesocket commented Feb 5, 2022 •

edited

Loading

nikunj-diwan commented Aug 9, 2023 •

edited

Loading