Update RKE2 v1.24.12+rke2r1

[troyanskiy]

Version
0.11.0

** RKE2 Version **

[root@kube-master certs]# rke2 --version
rke2 version v1.24.12+rke2r1 (1cbcfe3c873df5a7555cde3211a144055312b2a5)
go version go1.19.7 X:boringcrypto

Installed by manual provided here https://docs.rke2.io/upgrade/automated_upgrade just version was changed from v0.9.1 to v0.11.0
Also tested with v0.9.1 -> same errors in the log

Plan was also taken from the example, just version field was change with channel

Having errors in logs of system-upgrade-controller pod

kubectl logs  system-upgrade-controller-5f4b7d8cc6-jjrk6 -n system-upgrade
W0601 13:11:54.016496       1 client_config.go:615] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
time="2023-06-01T13:11:54Z" level=info msg="Applying CRD plans.upgrade.cattle.io"
time="2023-06-01T13:11:54Z" level=info msg="Starting /v1, Kind=Node controller"
time="2023-06-01T13:11:54Z" level=info msg="Starting /v1, Kind=Secret controller"
time="2023-06-01T13:11:54Z" level=info msg="Starting batch/v1, Kind=Job controller"
time="2023-06-01T13:11:54Z" level=info msg="Starting upgrade.cattle.io/v1, Kind=Plan controller"
time="2023-06-01T13:11:54Z" level=error msg="error syncing 'system-upgrade/agent-plan': handler system-upgrade-controller: Get \"https://update.rke2.io/v1-release/channels/stable\": x509: failed to load system roots and no roots provided; open /etc/ssl/certs/ca-certificates.crt: permission denied, requeuing"

Checked in host master system /etc/ssl/certs/ca-certificates.crt does not exists

[root@kube-master certs]# ls -la /etc/ssl/
total 12
drwxr-xr-x.  2 root root   77 Apr 24 15:15 .
drwxr-xr-x. 87 root root 8192 Jun  1 09:00 ..
lrwxrwxrwx.  1 root root   49 Sep 20  2022 cert.pem -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
lrwxrwxrwx.  1 root root   18 Sep 20  2022 certs -> /etc/pki/tls/certs
lrwxrwxrwx.  1 root root   28 Sep 20  2022 ct_log_list.cnf -> /etc/pki/tls/ct_log_list.cnf
lrwxrwxrwx.  1 root root   24 Sep 20  2022 openssl.cnf -> /etc/pki/tls/openssl.cnf

[root@kube-master certs]# ls -la /etc/ssl/certs
lrwxrwxrwx. 1 root root 18 Sep 20  2022 /etc/ssl/certs -> /etc/pki/tls/certs

[root@kube-master certs]# ls -la /etc/ssl/certs/
total 0
drwxr-xr-x. 2 root root  54 Jun  1 16:01 .
drwxr-xr-x. 5 root root 104 Apr 24 15:20 ..
lrwxrwxrwx. 1 root root  49 Sep 20  2022 ca-bundle.crt -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
lrwxrwxrwx. 1 root root  55 Sep 20  2022 ca-bundle.trust.crt -> /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt

UPD 1: Host linux Rocky 9.1

[NickHes]

Same on Rocky 9.

[troyanskiy]

Yes. I forgot to mention. I have it on Rocky 9.1 as well.

[kphunter]

I'm also seeing this on CentOS 7.9... certificate bundle is called ca-bundle.crt, not ca-certificates.crt

[brandond]

Unless you added a volume to the pod, it's loading that file out of the system-upgrade-controller image, not the host...
Is this perhaps being blocked by selinux or some other security module?

[kphunter]

I'm using the Deployment spec in this repo, but I believe the issue is that the underlying certificate trust store is different between Debian-based systems and Redhat-based ones (Golang lookup example)? I've also tried a symlink to ca-certificates.crt on the host systems, but that doesn't seem to work.

If the system-upgrade-controller image could accommodate different bundle names/locations, that would make it possible to mount an appropriate hostPath volume.