[Release-1.31] - K3s fails to start after running k3s certificate rotate-ca

[brandond]

Backport fix for K3s fails to start after running k3s certificate rotate-ca

• K3s fails to start after running k3s certificate rotate-ca #11014

[endawkins]

Validated on release-1.31 using commit 46cfd2c | version v1.31

Environment Details:

Node(s) CPU architecture, OS, and Version:

Linux ip-172-31-10-157 5.14.21-150500.55.44-default #1 SMP PREEMPT_DYNAMIC Mon Jan 15 10:03:40 UTC 2024 (cc7d8b6) x86_64 x86_64 x86_64 GNU/Linux
NAME="SLES"
VERSION="15-SP5"
VERSION_ID="15.5"
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP5"
ID="sles"
ID_LIKE="suse"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:15:sp5"
DOCUMENTATION_URL="https://documentation.suse.com/"

Cluster Configuration:

1 server (configuration does not matter)

Files:

• config.yaml

cluster-init: true
write-kubeconfig-mode: 644

• rotate-default-ca-certs.sh
  https://github.com/k3s-io/k3s/blob/release-1.28/contrib/util/rotate-default-ca-certs.sh

Steps:

1. Install K3s
2. Update Certificates using script
3. Rotate ca-certs k3s certificate rotate-ca
4. Restart k3s sudo systemctl restart k3s
5. Check status of k3s sudo systemctl status k3s

Reproduction of the Issue:
#11014 (comment)

Validation of the Issue:

- Observations:

k3s -v
k3s version v1.31.1+k3s-46cfd2cf (46cfd2cf)
go version go1.22.6

$ ./rotate-default-ca-certs.sh
To update certificates, you may now run:
    k3s certificate rotate-ca --path=/var/lib/rancher/k3s/server/rotate-ca

$ k3s certificate rotate-ca --path=/var/lib/rancher/k3s/server/rotate-ca
certificates saved to datastore

$ sudo systemctl restart k3s
$ sudo systemctl status k3s
● k3s.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: disabled)
     Active: active (running) since Mon 2024-10-14 21:11:50 UTC; 34s ago
       Docs: https://k3s.io
    Process: 29485 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service 2>/dev/null (code=exited, status=0/SUCCESS)
    Process: 29487 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 29488 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
   Main PID: 29489 (k3s-server)

$ kubectl get nodes,pods -A -o wide
NAME                                               STATUS   ROLES                       AGE    VERSION                INTERNAL-IP     EXTERNAL-IP     OS-IMAGE                              KERNEL-VERSION                 CONTAINER-RUNTIME
node/ip-172-31-10-157.us-east-2.compute.internal   Ready    control-plane,etcd,master   140m   v1.31.1+k3s-46cfd2cf   172.31.10.157   [REDACTED]      SUSE Linux Enterprise Server 15 SP5   5.14.21-150500.55.44-default   containerd://1.7.22-k3s1

NAMESPACE           NAME                                              READY   STATUS      RESTARTS   AGE    IP           NODE                                          NOMINATED NODE   READINESS GATES
kube-system         pod/coredns-56f6fc8fd7-cvm7h                      1/1     Running     0          140m   10.42.0.2    ip-172-31-10-157.us-east-2.compute.internal   <none>           <none>
kube-system         pod/helm-install-traefik-7zf6d                    0/1     Completed   1          140m   <none>       ip-172-31-10-157.us-east-2.compute.internal   <none>           <none>
kube-system         pod/helm-install-traefik-crd-8vslp                0/1     Completed   0          140m   <none>       ip-172-31-10-157.us-east-2.compute.internal   <none>           <none>
kube-system         pod/local-path-provisioner-5cf85fd84d-kbp6z       1/1     Running     0          140m   10.42.0.5    ip-172-31-10-157.us-east-2.compute.internal   <none>           <none>
kube-system         pod/metrics-server-5985cbc9d7-sj2s8               1/1     Running     0          140m   10.42.0.3    ip-172-31-10-157.us-east-2.compute.internal   <none>           <none>
kube-system         pod/svclb-nginx-loadbalancer-svc-e424a0da-scv2d   1/1     Running     0          135m   10.42.0.21   ip-172-31-10-157.us-east-2.compute.internal   <none>           <none>
kube-system         pod/svclb-traefik-2a4636bb-8tvdj                  2/2     Running     0          139m   10.42.0.7    ip-172-31-10-157.us-east-2.compute.internal   <none>           <none>
kube-system         pod/traefik-57b79cf995-fwb5z                      1/1     Running     0          139m   10.42.0.8    ip-172-31-10-157.us-east-2.compute.internal   <none>           <none>
test-ingressroute   pod/whoami-86c8d79cf4-97s4j                       1/1     Running     0          134m   10.42.0.24   ip-172-31-10-157.us-east-2.compute.internal   <none>           <none>
test-ingressroute   pod/whoami-86c8d79cf4-czldv                       1/1     Running     0          134m   10.42.0.25   ip-172-31-10-157.us-east-2.compute.internal   <none>           <none>
test-loadbalancer   pod/test-loadbalancer-6c774b8bb9-62s5b            1/1     Running     0          135m   10.42.0.22   ip-172-31-10-157.us-east-2.compute.internal   <none>           <none>
test-loadbalancer   pod/test-loadbalancer-6c774b8bb9-9xvxq            1/1     Running     0          135m   10.42.0.23   ip-172-31-10-157.us-east-2.compute.internal   <none>           <none>