Unable to upgrade Keycloak after Velero Backup / Restore

[tworcester]

Environment

Device and OS: RHEL 8
App version: [v0.24.1]
Kubernetes distro being used: zarf k3s 1.28
Other:

Steps to reproduce

1. Deploy UDS-core (With velero configured for EnableCSI, in this case I was using aws-ebs-csi-driver)
2. trigger backup k -n velero exec -it deploy/velero -- velero backup create --from-schedule=velero-udsbackup
3. stand up another cluster
4. Deploy UDS-core (Ideally you would just install Velero, but I didn't have access to just that zarf package)
5. trigger restore k -n velero exec -it deploy/velero -- velero restore create --from-backup=<reference> --existing-resource-policy=update --include-namespaces=keycloak
6. try to install UDS-core to a newer version (or the same)

Expected result

Successful upgrade of UDS-core on a cluster with velero restored UDS-core components.

Actual Result

PVCs that are restored have these fields in the spec due to the restoration options:

spec:
  dataSource:
    apiGroup: snapshot.storage.k8s.io
    kind: VolumeSnapshot
    name: velero-keycloak-data-12345
  dataSourceRef:
    apiGroup: snapshot.storage.k8s.io
    kind: VolumeSnapshot
    name: velero-keycloak-data-12345

When you try to re-install or upgrade to a new version, the Keycloak deployment fails saying that these fields are immutable.

Visual Proof (screenshots, videos, text, etc)

N/A

Severity/Priority

High - I vote this is pretty high since Velero is supposed to be the disaster recovery mechanism if an entire cluster goes down. This bug would prevent you from ever doing updates/maintenance on UDS-core

Additional Context

N/A

[mjnagel]

I was able to reproduce this issue today. Still need to identify where this limitation comes from, but it appears that restored PVCs end up with a 1Gi size. When re-deploying core it tries to shrink the PVC to the default size of 512Mi, which fails since PVCs cannot be shrunk.

Updating the default size should be a small, safe change that will fix this issue - will have a PR up by tomorrow to resolve this.

[mjnagel]

Also wanted to note, we would generally recommend an external DB (i.e. RDS or even an in cluster postgres) when running in production. This allows scalability and doesn't rely on the PVCs, which can simplify the backup/restore (can leverage other backup/restore capabilities in tools like RDS). We have a short doc/example on Keycloak HA with a database here: https://uds.defenseunicorns.com/core/configuration/resource-configuration-and-ha/#keycloak

[mjnagel]

@tworcester in further digging we discovered that this is a limitation specific to EBS. EBS volumes have a minimum size of 1Gi, so when the snapshots are created and used to restore Velero only has the snapshot size to go off of, not the original PVC size. I don't think we're going to update the default PVC size in this case since it could cause issues on the upgrade if someone's storage class does not allow volume expansion, but you should have a few options here still:

• Use bundle overrides to set a larger PVC size (1Gi+) which will ensure that the restored snapshots are the same size
• Switch to RDS/external DB for Keycloak's storage: This is definitely the ideal if this is for a more persistent staging/prod environment

We're likely going to make some slight changes to our pre-req notes on storage class to also help here.

[tworcester]

Awesome, thank you for tracking this down!