[Release-1.29] - S3 snapshots missing from etcdsnapshotfile list / k3s-etcd-snapshots configmap when multiple nodes save at the same time

[brandond]

Backport fix for S3 snapshots missing from etcdsnapshotfile list / k3s-etcd-snapshots configmap when multiple nodes save at the same time

• S3 snapshots missing from etcdsnapshotfile list / k3s-etcd-snapshots configmap when multiple nodes save at the same time #10989

[VestigeJ]

Previous behavior established here - lots of noise around etcd-snapshots taken from Rancher rancher/rke2#7027 (comment)

$ sudo kubectl get nodes

NAME                          STATUS   ROLES                              AGE     VERSION
v12910rc1-pool1-6jpzh-2brnz   Ready    control-plane,etcd,master,worker   4h19m   v1.29.10-rc1+k3s1
v12910rc1-pool1-6jpzh-k4n9c   Ready    control-plane,etcd,master,worker   4h21m   v1.29.10-rc1+k3s1
v12910rc1-pool1-6jpzh-xhqfn   Ready    control-plane,etcd,master,worker   4h18m   v1.29.10-rc1+k3s1
v12910rc1-pool2-xd58n-4dj4q   Ready    worker                             4h18m   v1.29.10-rc1+k3s1

$ sudo kubectl get cm -n kube-system k3s-etcd-snapshots
NAME DATA AGE
k3s-etcd-snapshots 12 132m

$ sudo k3s etcd-snapshot ls

Name Location Size Created
etcd-snapshot-v12910rc1-pool1-6jpzh-k4n9c-1729886403 s3://justin-snapshotsz-buckets/v12910rc1/etcd-snapshot-v12910rc1-pool1-6jpzh-k4n9c-1729886403 10588192 2024-10-25T20:00:03Z
etcd-snapshot-v12910rc1-pool1-6jpzh-xhqfn-1729886403 s3://justin-snapshotsz-buckets/v12910rc1/etcd-snapshot-v12910rc1-pool1-6jpzh-xhqfn-1729886403 10510368 2024-10-25T20:00:03Z
etcd-snapshot-v12910rc1-pool1-6jpzh-2brnz-1729886404 s3://justin-snapshotsz-buckets/v12910rc1/etcd-snapshot-v12910rc1-pool1-6jpzh-2brnz-1729886404 10547232 2024-10-25T20:00:04Z
etcd-snapshot-v12910rc1-pool1-6jpzh-2brnz-1729886404 file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-v12910rc1-pool1-6jpzh-2brnz-1729886404 10547232 2024-10-25T20:00:04Z
on-demand-v12910rc1-pool1-6jpzh-k4n9c-1729894288 s3://justin-snapshotsz-buckets/v12910rc1/on-demand-v12910rc1-pool1-6jpzh-k4n9c-1729894288 10588192 2024-10-25T22:11:28Z
on-demand-v12910rc1-pool1-6jpzh-2brnz-1729894289 s3://justin-snapshotsz-buckets/v12910rc1/on-demand-v12910rc1-pool1-6jpzh-2brnz-1729894289 10547232 2024-10-25T22:11:29Z
on-demand-v12910rc1-pool1-6jpzh-2brnz-1729894289 file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-v12910rc1-pool1-6jpzh-2brnz-1729894289 10547232 2024-10-25T22:11:29Z
on-demand-v12910rc1-pool1-6jpzh-xhqfn-1729894289 s3://justin-snapshotsz-buckets/v12910rc1/on-demand-v12910rc1-pool1-6jpzh-xhqfn-1729894289 10510368 2024-10-25T22:11:29Z

$ sudo kubectl get etcdsnapshotfile

NAME SNAPSHOTNAME NODE LOCATION SIZE CREATIONTIME
local-etcd-snapshot-v12910rc1-pool1-6jpzh-2brnz-1729886404-feee68 etcd-snapshot-v12910rc1-pool1-6jpzh-2brnz-1729886404 v12910rc1-pool1-6jpzh-2brnz file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-v12910rc1-pool1-6jpzh-2brnz-1729886404 10547232 2024-10-25T20:00:04Z
local-etcd-snapshot-v12910rc1-pool1-6jpzh-k4n9c-1729886403-b6386b etcd-snapshot-v12910rc1-pool1-6jpzh-k4n9c-1729886403 v12910rc1-pool1-6jpzh-k4n9c file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-v12910rc1-pool1-6jpzh-k4n9c-1729886403 10588192 2024-10-25T20:00:03Z
local-etcd-snapshot-v12910rc1-pool1-6jpzh-xhqfn-1729886403-d5f1ef etcd-snapshot-v12910rc1-pool1-6jpzh-xhqfn-1729886403 v12910rc1-pool1-6jpzh-xhqfn file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-v12910rc1-pool1-6jpzh-xhqfn-1729886403 10510368 2024-10-25T20:00:03Z
local-on-demand-v12910rc1-pool1-6jpzh-2brnz-1729894289-41c532 on-demand-v12910rc1-pool1-6jpzh-2brnz-1729894289 v12910rc1-pool1-6jpzh-2brnz file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-v12910rc1-pool1-6jpzh-2brnz-1729894289 10547232 2024-10-25T22:11:29Z
local-on-demand-v12910rc1-pool1-6jpzh-k4n9c-1729894288-2dcd37 on-demand-v12910rc1-pool1-6jpzh-k4n9c-1729894288 v12910rc1-pool1-6jpzh-k4n9c file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-v12910rc1-pool1-6jpzh-k4n9c-1729894288 10588192 2024-10-25T22:11:28Z
local-on-demand-v12910rc1-pool1-6jpzh-xhqfn-1729894289-d5b1bb on-demand-v12910rc1-pool1-6jpzh-xhqfn-1729894289 v12910rc1-pool1-6jpzh-xhqfn file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-v12910rc1-pool1-6jpzh-xhqfn-1729894289 10510368 2024-10-25T22:11:29Z
s3-etcd-snapshot-v12910rc1-pool1-6jpzh-2brnz-1729886404-d11558 etcd-snapshot-v12910rc1-pool1-6jpzh-2brnz-1729886404 s3 s3://justin-snapshot-buckets/v12910rc1/etcd-snapshot-v12910rc1-pool1-6jpzh-2brnz-1729886404 10547232 2024-10-25T20:00:04Z
s3-etcd-snapshot-v12910rc1-pool1-6jpzh-k4n9c-1729886403-38d804 etcd-snapshot-v12910rc1-pool1-6jpzh-k4n9c-1729886403 s3 s3://justin-snapshot-buckets/v12910rc1/etcd-snapshot-v12910rc1-pool1-6jpzh-k4n9c-1729886403 10588192 2024-10-25T20:00:03Z
s3-etcd-snapshot-v12910rc1-pool1-6jpzh-xhqfn-1729886403-3d4998 etcd-snapshot-v12910rc1-pool1-6jpzh-xhqfn-1729886403 s3 s3://justin-snapshot-buckets/v12910rc1/etcd-snapshot-v12910rc1-pool1-6jpzh-xhqfn-1729886403 10510368 2024-10-25T20:00:03Z
s3-on-demand-v12910rc1-pool1-6jpzh-2brnz-1729894289-7f0c1f on-demand-v12910rc1-pool1-6jpzh-2brnz-1729894289 s3 s3://justin-snapshot-buckets/v12910rc1/on-demand-v12910rc1-pool1-6jpzh-2brnz-1729894289 10547232 2024-10-25T22:11:29Z
s3-on-demand-v12910rc1-pool1-6jpzh-k4n9c-1729894288-5c747f on-demand-v12910rc1-pool1-6jpzh-k4n9c-1729894288 s3 s3://justin-snapshot-buckets/v12910rc1/on-demand-v12910rc1-pool1-6jpzh-k4n9c-1729894288 10588192 2024-10-25T22:11:28Z
s3-on-demand-v12910rc1-pool1-6jpzh-xhqfn-1729894289-5db748 on-demand-v12910rc1-pool1-6jpzh-xhqfn-1729894289 s3 s3://justin-snapshot-buckets/v12910rc1/on-demand-v12910rc1-pool1-6jpzh-xhqfn-1729894289 10510368 2024-10-25T22:11:29Z

![Screenshot 2024-10-25 at 3 19 46 PM](https://github.com/user-attachments/assets/d3c05381-ea9d-482f-8758-e6e66bded1f5)