[Release-1.31] - S3 snapshots missing from etcdsnapshotfile list / k3s-etcd-snapshots configmap when multiple nodes save at the same time

[brandond]

Backport fix for S3 snapshots missing from etcdsnapshotfile list / k3s-etcd-snapshots configmap when multiple nodes save at the same time

• S3 snapshots missing from etcdsnapshotfile list / k3s-etcd-snapshots configmap when multiple nodes save at the same time #10989

[VestigeJ]

Previous behavior established here - lots of noise around etcd-snapshots taken from Rancher rancher/rke2#7027 (comment)

$ sudo kubectl get nodes

NAME                            STATUS   ROLES                              AGE     VERSION
v1312rc2222-pool1-qnsps-6tz6t   Ready    control-plane,etcd,master,worker   4h27m   v1.31.2-rc2+k3s1
v1312rc2222-pool1-qnsps-6wxlv   Ready    control-plane,etcd,master,worker   4h27m   v1.31.2-rc2+k3s1
v1312rc2222-pool1-qnsps-jdhll   Ready    control-plane,etcd,master,worker   4h29m   v1.31.2-rc2+k3s1
v1312rc2222-pool2-cxjqq-69nl6   Ready    worker                             4h27m   v1.31.2-rc2+k3s1

$ sudo kubectl get cm -n kube-system k3s-etcd-snapshots

NAME                 DATA   AGE
k3s-etcd-snapshots   12     146m

$ sudo k3s etcd-snapshot ls

Name                                                   Location                                                                                               Size     Created
etcd-snapshot-v1312rc2222-pool1-qnsps-6tz6t-1729886402 file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-v1312rc2222-pool1-qnsps-6tz6t-1729886402 10240032 2024-10-25T20:00:02Z
on-demand-v1312rc2222-pool1-qnsps-6tz6t-1729894289     file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-v1312rc2222-pool1-qnsps-6tz6t-1729894289     10240032 2024-10-25T22:11:29Z

$ sudo kubectl get etcdsnapshotfile

NAME                                                                  SNAPSHOTNAME                                             NODE                            LOCATION                                                                                                 SIZE       CREATIONTIME
local-etcd-snapshot-v1312rc2222-pool1-qnsps-6tz6t-1729886402-fa98d2   etcd-snapshot-v1312rc2222-pool1-qnsps-6tz6t-1729886402   v1312rc2222-pool1-qnsps-6tz6t   file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-v1312rc2222-pool1-qnsps-6tz6t-1729886402   10240032   2024-10-25T20:00:02Z
local-etcd-snapshot-v1312rc2222-pool1-qnsps-6wxlv-1729886405-8f9c68   etcd-snapshot-v1312rc2222-pool1-qnsps-6wxlv-1729886405   v1312rc2222-pool1-qnsps-6wxlv   file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-v1312rc2222-pool1-qnsps-6wxlv-1729886405   10268704   2024-10-25T20:00:05Z
local-etcd-snapshot-v1312rc2222-pool1-qnsps-jdhll-1729886400-c0cab1   etcd-snapshot-v1312rc2222-pool1-qnsps-jdhll-1729886400   v1312rc2222-pool1-qnsps-jdhll   file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-v1312rc2222-pool1-qnsps-jdhll-1729886400   10375200   2024-10-25T20:00:00Z
local-on-demand-v1312rc2222-pool1-qnsps-6tz6t-1729894289-3ff983       on-demand-v1312rc2222-pool1-qnsps-6tz6t-1729894289       v1312rc2222-pool1-qnsps-6tz6t   file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-v1312rc2222-pool1-qnsps-6tz6t-1729894289       10240032   2024-10-25T22:11:29Z
local-on-demand-v1312rc2222-pool1-qnsps-6wxlv-1729894289-bea64f       on-demand-v1312rc2222-pool1-qnsps-6wxlv-1729894289       v1312rc2222-pool1-qnsps-6wxlv   file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-v1312rc2222-pool1-qnsps-6wxlv-1729894289       10268704   2024-10-25T22:11:29Z
local-on-demand-v1312rc2222-pool1-qnsps-jdhll-1729894289-f41106       on-demand-v1312rc2222-pool1-qnsps-jdhll-1729894289       v1312rc2222-pool1-qnsps-jdhll   file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-v1312rc2222-pool1-qnsps-jdhll-1729894289       10375200   2024-10-25T22:11:29Z
s3-etcd-snapshot-v1312rc2222-pool1-qnsps-6tz6t-1729886402-5daa13      etcd-snapshot-v1312rc2222-pool1-qnsps-6tz6t-1729886402   s3                              s3://justin-snapshot-buckets/v1312rc2/etcd-snapshot-v1312rc2222-pool1-qnsps-6tz6t-1729886402             10240032   2024-10-25T20:00:02Z
s3-etcd-snapshot-v1312rc2222-pool1-qnsps-6wxlv-1729886405-53323d      etcd-snapshot-v1312rc2222-pool1-qnsps-6wxlv-1729886405   s3                              s3://justin-snapshot-buckets/v1312rc2/etcd-snapshot-v1312rc2222-pool1-qnsps-6wxlv-1729886405             10268704   2024-10-25T20:00:05Z
s3-etcd-snapshot-v1312rc2222-pool1-qnsps-jdhll-1729886400-da171e      etcd-snapshot-v1312rc2222-pool1-qnsps-jdhll-1729886400   s3                              s3://justin-snapshot-buckets/v1312rc2/etcd-snapshot-v1312rc2222-pool1-qnsps-jdhll-1729886400             10375200   2024-10-25T20:00:00Z
s3-on-demand-v1312rc2222-pool1-qnsps-6tz6t-1729894289-3b0fed          on-demand-v1312rc2222-pool1-qnsps-6tz6t-1729894289       s3                              s3://justin-snapshot-buckets/v1312rc2/on-demand-v1312rc2222-pool1-qnsps-6tz6t-1729894289                 10240032   2024-10-25T22:11:29Z
s3-on-demand-v1312rc2222-pool1-qnsps-6wxlv-1729894289-53d324          on-demand-v1312rc2222-pool1-qnsps-6wxlv-1729894289       s3                              s3://justin-snapshot-buckets/v1312rc2/on-demand-v1312rc2222-pool1-qnsps-6wxlv-1729894289                 10268704   2024-10-25T22:11:29Z
s3-on-demand-v1312rc2222-pool1-qnsps-jdhll-1729894289-ebe884          on-demand-v1312rc2222-pool1-qnsps-jdhll-1729894289       s3                              s3://justin-snapshot-buckets/v1312rc2/on-demand-v1312rc2222-pool1-qnsps-jdhll-1729894289                 10375200   2024-10-25T22:11:29Z