Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Release-1.31] - S3 snapshots missing from etcdsnapshotfile list / rke2-etcd-snapshots configmap when multiple nodes save at the same time #7027

Closed
brandond opened this issue Oct 11, 2024 · 1 comment
Closed

[Release-1.31] - S3 snapshots missing from etcdsnapshotfile list / rke2-etcd-snapshots configmap when multiple nodes save at the same time #7027

brandond opened this issue Oct 11, 2024 · 1 comment
Assignees
@brandond @VestigeJ
Milestone
v1.31.2+rke2r1

Comments

@brandond
Copy link
Member

Backport fix for S3 snapshots missing from etcdsnapshotfile list / rke2-etcd-snapshots configmap when multiple nodes save at the same time
@brandond brandond self-assigned this Oct 11, 2024
@brandond brandond mentioned this issue Oct 11, 2024
Merged
@caroline-suse-rancher caroline-suse-rancher added this to the v1.31.2+rke2r1 milestone Oct 14, 2024
@VestigeJ
Copy link
Contributor

Reproduced existing release behavior over here - k3s-io/k3s#10989 (comment)

Looking much better from the cluster and S3 perspectives

$ kubectl get nodes

NAME                              STATUS   ROLES                              AGE     VERSION
justrke2snaps-pool1-pd8kl-76cxp   Ready    control-plane,etcd,master,worker   12m     v1.31.1+rke2r2
justrke2snaps-pool1-pd8kl-f7b8x   Ready    control-plane,etcd,master,worker   7m31s   v1.31.1+rke2r2
justrke2snaps-pool1-pd8kl-m88r7   Ready    control-plane,etcd,master,worker   7m1s    v1.31.1+rke2r2
justrke2snaps-pool2-p6fs2-b27h2   Ready    worker                             5m42s   v1.31.1+rke2r2

> kubectl get cm -n kube-system rke2-etcd-snapshots
NAME                  DATA   AGE
rke2-etcd-snapshots   26     15m

sudo rke2 etcd-snapshot ls
Name                                                 Location                                                                                              Size     Created
on-demand-justrke2snaps-pool1-pd8kl-f7b8x-1729187500 s3://justin-snapshot-buckets/on-demand-justrke2snaps-pool1-pd8kl-f7b8x-1729187500                     23785504 2024-10-17T17:51:40Z
on-demand-justrke2snaps-pool1-pd8kl-m88r7-1729187500 s3://justin-snapshot-buckets/on-demand-justrke2snaps-pool1-pd8kl-m88r7-1729187500                     23789600 2024-10-17T17:51:40Z
on-demand-justrke2snaps-pool1-pd8kl-76cxp-1729187502 s3://justin-snapshot-buckets/on-demand-justrke2snaps-pool1-pd8kl-76cxp-1729187502                     24657952 2024-10-17T17:51:42Z
on-demand-justrke2snaps-pool1-pd8kl-76cxp-1729187502 file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-justrke2snaps-pool1-pd8kl-76cxp-1729187502 24657952 2024-10-17T17:51:42Z

kubectl get etcdsnapshotfile
NAME                                                                    SNAPSHOTNAME                                                  NODE                              LOCATION                                                                                                SIZE       CREATIONTIME
local-on-demand-justrke2snaps-pool1-pd8kl-76cxp-1729187502-d73573       on-demand-justrke2snaps-pool1-pd8kl-76cxp-1729187502          justrke2snaps-pool1-pd8kl-76cxp   file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-justrke2snaps-pool1-pd8kl-76cxp-1729187502   24657952   2024-10-17T17:51:42Z
local-on-demand-justrke2snaps-pool1-pd8kl-f7b8x-1729187500-df923c       on-demand-justrke2snaps-pool1-pd8kl-f7b8x-1729187500          justrke2snaps-pool1-pd8kl-f7b8x   file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-justrke2snaps-pool1-pd8kl-f7b8x-1729187500   23785504   2024-10-17T17:51:40Z
local-on-demand-justrke2snaps-pool1-pd8kl-m88r7-1729187500-27eb52       on-demand-justrke2snaps-pool1-pd8kl-m88r7-1729187500          justrke2snaps-pool1-pd8kl-m88r7   file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-justrke2snaps-pool1-pd8kl-m88r7-1729187500   23789600   2024-10-17T17:51:40Z
s3-etcd-snapshot-rke2-just-racing-pool1-7v2s4-5hbmj-1728511204-6eed9f   etcd-snapshot-rke2-just-racing-pool1-7v2s4-5hbmj-1728511204   s3                                s3://justin-snapshot-buckets/needed/etcd-snapshot-rke2-just-racing-pool1-7v2s4-5hbmj-1728511204         10145824   2024-10-09T22:00:04Z
s3-etcd-snapshot-rke2-just-racing-pool1-7v2s4-5hbmj-1728514803-8625e3   etcd-snapshot-rke2-just-racing-pool1-7v2s4-5hbmj-1728514803   s3                                s3://justin-snapshot-buckets/needed/etcd-snapshot-rke2-just-racing-pool1-7v2s4-5hbmj-1728514803         13365280   2024-10-09T23:00:03Z
s3-etcd-snapshot-rke2-just-racing-pool1-7v2s4-jlqxm-1728514803-c1c85c   etcd-snapshot-rke2-just-racing-pool1-7v2s4-jlqxm-1728514803   s3                                s3://justin-snapshot-buckets/needed/etcd-snapshot-rke2-just-racing-pool1-7v2s4-jlqxm-1728514803         13598752   2024-10-09T23:00:03Z
s3-etcd-snapshot-rke2-just-racing-pool1-7v2s4-m7zpf-1728511202-d439cb   etcd-snapshot-rke2-just-racing-pool1-7v2s4-m7zpf-1728511202   s3                                s3://justin-snapshot-buckets/needed/etcd-snapshot-rke2-just-racing-pool1-7v2s4-m7zpf-1728511202         9715744    2024-10-09T22:00:02Z
s3-etcd-snapshot-rke2-just-racing-pool1-7v2s4-m7zpf-1728514801-ef815c   etcd-snapshot-rke2-just-racing-pool1-7v2s4-m7zpf-1728514801   s3                                s3://justin-snapshot-buckets/needed/etcd-snapshot-rke2-just-racing-pool1-7v2s4-m7zpf-1728514801         13135904   2024-10-09T23:00:01Z
s3-on-demand-justrke2snaps-pool1-pd8kl-76cxp-1729187502-cc987a          on-demand-justrke2snaps-pool1-pd8kl-76cxp-1729187502          s3                                s3://justin-snapshot-buckets/on-demand-justrke2snaps-pool1-pd8kl-76cxp-1729187502                       24657952   2024-10-17T17:51:42Z
s3-on-demand-justrke2snaps-pool1-pd8kl-f7b8x-1729187500-9423c6          on-demand-justrke2snaps-pool1-pd8kl-f7b8x-1729187500          s3                                s3://justin-snapshot-buckets/on-demand-justrke2snaps-pool1-pd8kl-f7b8x-1729187500                       23785504   2024-10-17T17:51:40Z
s3-on-demand-justrke2snaps-pool1-pd8kl-m88r7-1729187500-916a6a          on-demand-justrke2snaps-pool1-pd8kl-m88r7-1729187500          s3                                s3://justin-snapshot-buckets/on-demand-justrke2snaps-pool1-pd8kl-m88r7-1729187500                       23789600   2024-10-17T17:51:40Z
s3-on-demand-rke2-just-racing-pool1-7v2s4-5hbmj-1728511066-587732       on-demand-rke2-just-racing-pool1-7v2s4-5hbmj-1728511066       s3                                s3://justin-snapshot-buckets/needed/on-demand-rke2-just-racing-pool1-7v2s4-5hbmj-1728511066             7213088    2024-10-09T21:57:46Z
s3-on-demand-rke2-just-racing-pool1-7v2s4-5hbmj-1728511251-98681c       on-demand-rke2-just-racing-pool1-7v2s4-5hbmj-1728511251       s3                                s3://justin-snapshot-buckets/needed/on-demand-rke2-just-racing-pool1-7v2s4-5hbmj-1728511251             11026464   2024-10-09T22:00:51Z
s3-on-demand-rke2-just-racing-pool1-7v2s4-5hbmj-1728511422-e91f20       on-demand-rke2-just-racing-pool1-7v2s4-5hbmj-1728511422       s3                                s3://justin-snapshot-buckets/needed/on-demand-rke2-just-racing-pool1-7v2s4-5hbmj-1728511422             12083232   2024-10-09T22:03:42Z
s3-on-demand-rke2-just-racing-pool1-7v2s4-5hbmj-1728511844-24c276       on-demand-rke2-just-racing-pool1-7v2s4-5hbmj-1728511844       s3                                s3://justin-snapshot-buckets/needed/on-demand-rke2-just-racing-pool1-7v2s4-5hbmj-1728511844             13365280   2024-10-09T22:10:44Z
s3-on-demand-rke2-just-racing-pool1-7v2s4-5hbmj-1728512268-ca04a6       on-demand-rke2-just-racing-pool1-7v2s4-5hbmj-1728512268       s3                                s3://justin-snapshot-buckets/needed/on-demand-rke2-just-racing-pool1-7v2s4-5hbmj-1728512268             13365280   2024-10-09T22:17:48Z
s3-on-demand-rke2-just-racing-pool1-7v2s4-jlqxm-1728511035-b239ba       on-demand-rke2-just-racing-pool1-7v2s4-jlqxm-1728511035       s3                                s3://justin-snapshot-buckets/needed/on-demand-rke2-just-racing-pool1-7v2s4-jlqxm-1728511035             22904864   2024-10-09T21:57:15Z
s3-on-demand-rke2-just-racing-pool1-7v2s4-jlqxm-1728511250-86f1a6       on-demand-rke2-just-racing-pool1-7v2s4-jlqxm-1728511250       s3                                s3://justin-snapshot-buckets/needed/on-demand-rke2-just-racing-pool1-7v2s4-jlqxm-1728511250             10481696   2024-10-09T22:00:50Z
s3-on-demand-rke2-just-racing-pool1-7v2s4-jlqxm-1728511421-79d842       on-demand-rke2-just-racing-pool1-7v2s4-jlqxm-1728511421       s3                                s3://justin-snapshot-buckets/needed/on-demand-rke2-just-racing-pool1-7v2s4-jlqxm-1728511421             11726880   2024-10-09T22:03:41Z
s3-on-demand-rke2-just-racing-pool1-7v2s4-jlqxm-1728511842-61e23a       on-demand-rke2-just-racing-pool1-7v2s4-jlqxm-1728511842       s3                                s3://justin-snapshot-buckets/needed/on-demand-rke2-just-racing-pool1-7v2s4-jlqxm-1728511842             13140k     2024-10-09T22:10:42Z
s3-on-demand-rke2-just-racing-pool1-7v2s4-jlqxm-1728512267-fc9856       on-demand-rke2-just-racing-pool1-7v2s4-jlqxm-1728512267       s3                                s3://justin-snapshot-buckets/needed/on-demand-rke2-just-racing-pool1-7v2s4-jlqxm-1728512267             13140k     2024-10-09T22:17:47Z
s3-on-demand-rke2-just-racing-pool1-7v2s4-m7zpf-1728511034-563fa9       on-demand-rke2-just-racing-pool1-7v2s4-m7zpf-1728511034       s3                                s3://justin-snapshot-buckets/needed/on-demand-rke2-just-racing-pool1-7v2s4-m7zpf-1728511034             22622240   2024-10-09T21:57:14Z
s3-on-demand-rke2-just-racing-pool1-7v2s4-m7zpf-1728511250-aeb6ad       on-demand-rke2-just-racing-pool1-7v2s4-m7zpf-1728511250       s3                                s3://justin-snapshot-buckets/needed/on-demand-rke2-just-racing-pool1-7v2s4-m7zpf-1728511250             10469408   2024-10-09T22:00:50Z
s3-on-demand-rke2-just-racing-pool1-7v2s4-m7zpf-1728511421-73c1f7       on-demand-rke2-just-racing-pool1-7v2s4-m7zpf-1728511421       s3                                s3://justin-snapshot-buckets/needed/on-demand-rke2-just-racing-pool1-7v2s4-m7zpf-1728511421             11661344   2024-10-09T22:03:41Z
s3-on-demand-rke2-just-racing-pool1-7v2s4-m7zpf-1728511842-4a4acb       on-demand-rke2-just-racing-pool1-7v2s4-m7zpf-1728511842       s3                                s3://justin-snapshot-buckets/needed/on-demand-rke2-just-racing-pool1-7v2s4-m7zpf-1728511842             13135904   2024-10-09T22:10:42Z
s3-on-demand-rke2-just-racing-pool1-7v2s4-m7zpf-1728512267-bd5715       on-demand-rke2-just-racing-pool1-7v2s4-m7zpf-1728512267       s3                                s3://justin-snapshot-buckets/needed/on-demand-rke2-just-racing-pool1-7v2s4-m7zpf-1728512267             13135904   2024-10-09T22:17:47Z

Screenshot 2024-10-17 at 10 56 50 AM
Screenshot 2024-10-17 at 10 56 05 AM

This was referenced Oct 25, 2024
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@brandond brandond

@VestigeJ VestigeJ

Labels
None yet
Projects
None yet
Milestone
v1.31.2+rke2r1
Development

No branches or pull requests
3 participants
@brandond @VestigeJ @caroline-suse-rancher