s3 snapshots do not show in the UI

[davidnuzik]

Rancher Server Setup

• Rancher version: v2.6-head 246fbd5fa 8/5/21 (single-node docker)
• Installation option (Docker install/Helm Chart): Docker 19.03

Downstream cluster: rke2 provisioned by rancher, 3 etcd, 2 cp, 3 worker
Downstream k8s version: v1.21.3-rc3+rke2r2

Describe the bug

1. Create a cluster. Submit valid s3 information.
2. Take a snapshot. This appears to be successful.

2021/08/05 22:35:57 [INFO] rkecluster fleet-default/dave-rke2-s3test: refreshing etcd create state
2021/08/05 22:35:58 [INFO] rkecluster fleet-default/dave-rke2-s3test: starting etcd snapshot
2021/08/05 22:35:58 [INFO] rkecluster fleet-default/dave-rke2-s3test: waiting for etcd snapshot
2021/08/05 22:35:58 [INFO] rkecluster fleet-default/dave-rke2-s3test: waiting for etcd snapshot
2021/08/05 22:35:58 [INFO] rkecluster fleet-default/dave-rke2-s3test: waiting for etcd snapshot
2021/08/05 22:35:58 [INFO] rkecluster fleet-default/dave-rke2-s3test: waiting for etcd snapshot
2021/08/05 22:35:58 [INFO] rkecluster fleet-default/dave-rke2-s3test: waiting for etcd snapshot
2021/08/05 22:36:04 [INFO] rkecluster fleet-default/dave-rke2-s3test: waiting for etcd snapshot
2021/08/05 22:36:04 [INFO] rkecluster fleet-default/dave-rke2-s3test: refreshing etcd create state
2021/08/05 22:36:04 [INFO] rkecluster fleet-default/dave-rke2-s3test: provisioning bootstrap node(s) dave-rke2-s3test-e-59bc958d8-bvkrg
2021/08/05 22:36:04 [INFO] rkecluster fleet-default/dave-rke2-s3test: provisioning bootstrap node(s) dave-rke2-s3test-e-59bc958d8-bvkrg
2021/08/05 22:36:04 [INFO] rkecluster fleet-default/dave-rke2-s3test: provisioning bootstrap node(s) dave-rke2-s3test-e-59bc958d8-bvkrg: waiting for plan to be applied
2021/08/05 22:36:04 [INFO] rkecluster fleet-default/dave-rke2-s3test: provisioning bootstrap node(s) dave-rke2-s3test-e-59bc958d8-bvkrg: waiting for plan to be applied
2021/08/05 22:36:22 [INFO] rkecluster fleet-default/dave-rke2-s3test: provisioning bootstrap node(s) dave-rke2-s3test-e-59bc958d8-bvkrg: waiting for plan to be applied

3. Inspect the s3 bucket - everything is there it looks like the snapshot was successful and I can retake snapshots and they will work and upload to my bucket.
   

4. Inspect the snapshots tab - none of the snapshots show.
   

Note: Refreshing the UI does not fix the issue. Cleaning browser cache and launching the UI again also does not fix.

Result
The UI does not show any of the (apparently) successfully-taken s3 snapshots.

Expected Result

The UI should show the s3 snapshots, just like it does for local snapshots.

Additional Info
Blocks testing/closure of rancher/rancher#32391

[thedadams]

Rancher looks at the configmap rke2-etcd-snapshots in the downstream RKE2 cluster to get the information for the snapshots. In this case, the configmap exists, but it is empty.

> kubectl -n kube-system get configmap rke2-etcd-snapshots -o yaml
apiVersion: v1
kind: ConfigMap
metadata:
  creationTimestamp: "2021-08-05T22:30:30Z"
  name: rke2-etcd-snapshots
  namespace: kube-system
  resourceVersion: "378"
  uid: 118aca17-9b4d-428b-af38-32043e65e8ac

[richard-cox]

@thedadams Is that the correct location to look at, if so is this a backend issue?

[thedadams]

@richard-cox I believe it is an RKE2 issue. It seems RKE2 is responsible for populating the configmap.

[Jono-SUSE-Rancher]

This is being worked on by the RKE2 team via rancher/rke2#1551, once they have a fix in place I will move this back over to test.

[richard-cox]

I've tagged this as waiting for backend (we don't have a waiting for rke column/label)

[davidnuzik]

Yes, I am working with the RKE2 team via rancher/rke2#1551 as Jono mentioned. Once there is a fix/solution there then I'll post an update here.

[davidnuzik]

Will be fixed by rancher/rke2#1551 probably today or tomorrow depending on how things go. We know the nature of the issue now after troubleshooting/debugging.

[briandowns]

The linked issue above is now in "to test" with the resolved PR's merged and backported.

[davidnuzik]

Need rke2 rc first; will put this on pause until have that and 2.6 KDM points to the new RC.

[davidnuzik]

rancher/rancher#34088 is changing how s3 works so waiting on that now.

[davidnuzik]

So this was working and it's not again, but the nature of the issue seems to be different. The ConfigMap is indeed populated now but the snapshots don't show in the UI. Bit difficult to reproduce though -- I think this may be caused (but do not know why) by a failed snapshot restoration - local snapshots do not work now when attempt to restore. I think if this happens and one cluster gets stuck in Error state all other clusters won't list out their snapshots now even though they were successfully taken and show in the ConfigMap.

Not quite sure if I should open a new bug or just re-use this one - the cause seems different now, but I technically can't close out this issue because snapshots don't show in the UI. hmm

[davidnuzik]

This appears to be working now. Restore does not work but that is a separate issue.

My checks PASSED

Reproduction Steps:

Not required.

Validation Environment:

Rancher version: v2.6-head 0935a90 pulled at 8/24/21 11:09 am Pacific
Rancher cluster type: single-node docker
Docker version: 20.10

Downstream cluster type: rke2
Downstream K8s version: v1.21.3+rke2r2

Validation steps:

1. Create an RKE2 cluster, specify valid s3 config during cluster creation (this creates an s3-compatible cloud credential for us).
2. Once the cluster is ready, take a snapshot. The snapshot is listed in the UI now (the ConfigMap is also populated).