[Release-1.30] - K3s fails to start after running k3s certificate rotate-ca

[brandond]

Backport fix for K3s fails to start after running k3s certificate rotate-ca

• K3s fails to start after running k3s certificate rotate-ca #11014

[endawkins]

Validated on release-1.30 using commit e9bb624 | version v1.30

Environment Details:

Node(s) CPU architecture, OS, and Version:

Linux ip-172-31-8-108 5.14.21-150500.55.44-default #1 SMP PREEMPT_DYNAMIC Mon Jan 15 10:03:40 UTC 2024 (cc7d8b6) x86_64 x86_64 x86_64 GNU/Linux
NAME="SLES"
VERSION="15-SP5"
VERSION_ID="15.5"
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP5"
ID="sles"
ID_LIKE="suse"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:15:sp5"
DOCUMENTATION_URL="https://documentation.suse.com/"

Cluster Configuration:

1 server (configuration does not matter)

Files:

• config.yaml

cluster-init: true
write-kubeconfig-mode: 644

• rotate-default-ca-certs.sh
  https://github.com/k3s-io/k3s/blob/release-1.28/contrib/util/rotate-default-ca-certs.sh

Steps:

1. Install K3s
2. Update Certificates using script
3. Rotate ca-certs k3s certificate rotate-ca
4. Restart k3s sudo systemctl restart k3s
5. Check status of k3s sudo systemctl status k3s

Reproduction of the Issue:
#11014 (comment)

Validation of the Issue:

- Observations:

k3s -v
k3s version v1.30.5+k3s-e9bb624c (e9bb624c)
go version go1.22.6

$ ./rotate-default-ca-certs.sh
To update certificates, you may now run:
    k3s certificate rotate-ca --path=/var/lib/rancher/k3s/server/rotate-ca

$ k3s certificate rotate-ca --path=/var/lib/rancher/k3s/server/rotate-ca
certificates saved to datastore

$ sudo systemctl restart k3s
$ sudo systemctl status k3s
● k3s.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: disabled)
     Active: active (running) since Mon 2024-10-14 21:06:53 UTC; 4min 51s ago
       Docs: https://k3s.io
    Process: 29252 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service 2>/dev/null (code=exited, status=0/SUCCESS)
    Process: 29254 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 29255 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
   Main PID: 29256 (k3s-server)

$ kubectl get nodes,pods -A -o wide
NAME                                              STATUS   ROLES                       AGE    VERSION                INTERNAL-IP    EXTERNAL-IP      OS-IMAGE                              KERNEL-VERSION                 CONTAINER-RUNTIME
node/ip-172-31-8-108.us-east-2.compute.internal   Ready    control-plane,etcd,master   144m   v1.30.5+k3s-e9bb624c   172.31.8.108   [REDACTED]       SUSE Linux Enterprise Server 15 SP5   5.14.21-150500.55.44-default   containerd://1.7.22-k3s1

NAMESPACE           NAME                                              READY   STATUS      RESTARTS   AGE    IP           NODE                                         NOMINATED NODE   READINESS GATES
kube-system         pod/coredns-7b98449c4-rllrq                       1/1     Running     0          143m   10.42.0.6    ip-172-31-8-108.us-east-2.compute.internal   <none>           <none>
kube-system         pod/helm-install-traefik-crd-wjxk6                0/1     Completed   0          143m   <none>       ip-172-31-8-108.us-east-2.compute.internal   <none>           <none>
kube-system         pod/helm-install-traefik-tmdvr                    0/1     Completed   1          143m   <none>       ip-172-31-8-108.us-east-2.compute.internal   <none>           <none>
kube-system         pod/local-path-provisioner-595dcfc56f-x6w85       1/1     Running     0          143m   10.42.0.4    ip-172-31-8-108.us-east-2.compute.internal   <none>           <none>
kube-system         pod/metrics-server-cdcc87586-4fhhn                1/1     Running     0          143m   10.42.0.5    ip-172-31-8-108.us-east-2.compute.internal   <none>           <none>
kube-system         pod/svclb-nginx-loadbalancer-svc-9da0d400-6swdb   1/1     Running     0          138m   10.42.0.21   ip-172-31-8-108.us-east-2.compute.internal   <none>           <none>
kube-system         pod/svclb-traefik-2d271cec-qd2xk                  2/2     Running     0          143m   10.42.0.7    ip-172-31-8-108.us-east-2.compute.internal   <none>           <none>
kube-system         pod/traefik-d7c9c5778-lcnsn                       1/1     Running     0          143m   10.42.0.8    ip-172-31-8-108.us-east-2.compute.internal   <none>           <none>
test-ingressroute   pod/whoami-57b48994d9-hsvzx                       1/1     Running     0          138m   10.42.0.25   ip-172-31-8-108.us-east-2.compute.internal   <none>           <none>
test-ingressroute   pod/whoami-57b48994d9-vmg6q                       1/1     Running     0          138m   10.42.0.24   ip-172-31-8-108.us-east-2.compute.internal   <none>           <none>
test-loadbalancer   pod/test-loadbalancer-6dc4cfd864-42nq4            1/1     Running     0          138m   10.42.0.22   ip-172-31-8-108.us-east-2.compute.internal   <none>           <none>
test-loadbalancer   pod/test-loadbalancer-6dc4cfd864-s2pkv            1/1     Running     0          138m   10.42.0.23   ip-172-31-8-108.us-east-2.compute.internal   <none>           <none>