S3 snapshots missing from etcdsnapshotfile list / k3s-etcd-snapshots configmap when multiple nodes save at the same time #10989
Comments
This was referenced Oct 9, 2024
|
Waiting for RC to attempt to test Yeah this isn't very good - I feel like we should be doing more cleanup here than we presently do - even with the file retention limit on S3 it gets blasted by all nodes - which is fine as long as it gets cleaned up....
|
|
Yeah, I don't really like how all the nodes independently reconcile S3 since they have individual access to it and are responsible for uploading their own files. I need to think about possible better ways to do this. In the mean time, this PR should at least prevent the S3 snapshot list from being so flakey when they all upload at once. |
aganesh-suse
added
the
waiting-for-RC
|
closing based on #11050 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
K3s tracking issue for:
This is extremely difficult to trigger outside of Rancher. When the user triggers an etcd snapshot via Rancher, all etcd nodes simultaneously save snapshots and attempt to reconcile their local snapshots, and the snapshots they see on S3, with ETCDSnapshotFile resources. Since there is a slight time gap in between when objects are listed from S3, and when ETCDSnapshotFiles resources are listed, not all cluster members will have a consistent view of objects in both stores, and may end up deleting ETCDSnapshotFile resources for snapshots that other nodes just finished uploading. The resources will be recreated next time S3 is reconciled (the next time snapshots are saved/deleted/pruned) but this does cause a temporary mismatch between what shows up in
kubectl get etcdsnapshotfileandkubectl get configmap -n kube-system k3s-etcd-snapshotsand what shows up ink3s etcd-snapshot ls(and what's actually in S3).The text was updated successfully, but these errors were encountered: