After enlarge database persistence disk size, no change applied to the database pod

[ShuangMen]

Is your feature request related to a problem? Please describe.
After kubecf deployed, try to enlarge the database persistence disk size with below change:
update values.yaml, change the database disk size from 20G to 40G

  database:
    instances: ~
    persistence:
      size: 20Gi

==>

  database:
    instances: ~
    persistence:
      size: 40Gi

then run helm upgrade, then pod database-0 terminated and start as a new one.

Then perform below check:

$k get qsts database  -n kubecf -o yaml
...
      volumeClaimTemplates:
      - metadata:
          name: pxc-data
        spec:
          accessModes:
          - ReadWriteOnce
          resources:
            requests:
              storage: 40Gi
          storageClassName: new-ibmc-vpc-block-10iops-tier
...

...
$k get sts database  -n kubecf -o yaml

  volumeClaimTemplates:
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      creationTimestamp: null
      name: pxc-data
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 20Gi
      storageClassName: new-ibmc-vpc-block-10iops-tier
      volumeMode: Filesystem
    status:
      phase: Pending
...

login the pod database-0 and check the disk size, no change applied to the disk, still 20G.

$ k exec -it database-0 sh -n kubecf
Defaulting container name to database.
Use 'kubectl describe pod/database-0 -n kubecf' to see all of the containers in this pod.
sh-4.4# df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay          99G   47G   48G  50% /
tmpfs            64M     0   64M   0% /dev
tmpfs           7.3G     0  7.3G   0% /sys/fs/cgroup
/dev/vda2        99G   47G   48G  50% /root
shm              64M     0   64M   0% /dev/shm
/dev/vdf         20G  430M   20G   3% /var/lib/mysql
tmpfs           7.3G   16K  7.3G   1% /etc/mysql/tls/certs
tmpfs           7.3G   16K  7.3G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs           7.3G     0  7.3G   0% /proc/acpi
tmpfs           7.3G     0  7.3G   0% /proc/scsi
tmpfs           7.3G     0  7.3G   0% /sys/firmware

Describe the solution you'd like
After enlarging the disk size and then applying the change with helm upgrade, there should be a smooth data migration from the original database disk to a new bigger disk and then have new disk mounted to the database pod.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/174568809

The labels on this github issue will be updated when the story is started.

[manno]

We discussed the issue and updated the story in trackers. Can you check if k8s can handle this automatically: kubernetes/kubernetes#35941 (comment)

Some storage classes do support this: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#expanding-persistent-volumes-claims

[JimmyMa]

@manno Below is the storageclass, it doesn't have allowVolumeExpansion that is mentioned in https://kubernetes.io/docs/concepts/storage/persistent-volumes/#expanding-persistent-volumes-claims , is that the reason the persistent disk resizing failed?

root@shuck1:~# kubectl get storageclass ibmc-vpc-block-10iops-tier -o yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    armada-service: addon-vpc-block-csi-driver
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{"armada-service":"addon-vpc-block-csi-driver","storageclass.beta.kubernetes.io/is-default-class":"true","version":"2.0.3_375"},"labels":{"addonmanager.kubernetes.io/mode":"Reconcile","app":"ibm-vpc-block-csi-driver","razee/force-apply":"true"},"name":"ibmc-vpc-block-10iops-tier"},"parameters":{"billingType":"hourly","classVersion":"1","csi.storage.k8s.io/fstype":"ext4","encrypted":"false","encryptionKey":"","profile":"10iops-tier","region":"","resourceGroup":"","sizeRange":"[10-2000]GiB","tags":"","zone":""},"provisioner":"vpc.block.csi.ibm.io","reclaimPolicy":"Delete"}
    storageclass.beta.kubernetes.io/is-default-class: "true"
    version: 2.0.3_375
  creationTimestamp: "2020-08-26T02:56:08Z"
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
    app: ibm-vpc-block-csi-driver
    razee/force-apply: "true"
  name: ibmc-vpc-block-10iops-tier
  resourceVersion: "13304267"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/ibmc-vpc-block-10iops-tier
  uid: 967f5315-a65c-4a61-b436-2ac68c2fe085
parameters:
  billingType: hourly
  classVersion: "1"
  csi.storage.k8s.io/fstype: ext4
  encrypted: "false"
  encryptionKey: ""
  profile: 10iops-tier
  region: ""
  resourceGroup: ""
  sizeRange: '[10-2000]GiB'
  tags: ""
  zone: ""
provisioner: vpc.block.csi.ibm.io
reclaimPolicy: Delete
volumeBindingMode: Immediate

After changing the size of persistent disk of database to 40G, the database statefulset still has 20G in volumeClaimTemplates as below, shouldn't the storage be 40G after the change?

$k get sts database  -n kubecf -o yaml

  volumeClaimTemplates:
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      creationTimestamp: null
      name: pxc-data
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 20Gi
      storageClassName: new-ibmc-vpc-block-10iops-tier
      volumeMode: Filesystem
    status:
      phase: Pending

[manno]

Looking at the code I can see the quarks statefulset reconciler cannot update the spec.resources.requests fields directly.

Forbidden: updates to statefulset spec for fields other than ‘replicas’, ‘template’, and ‘updateStrategy’ are forbidden”

The process seems to be more complicated, than just updating the sts spec.

@JimmyMa

Our CI is not using a storage class that supports this. Can you check if you can manually change the size in the PVC, kubectl edit -n kubecf pvc pxc-data?

Using the local-path storageclass, I get error: persistentvolumeclaims "pvc-example-quarks-statefulset-0" could not be patched: persistentvolumeclaims "pvc-example-quarks-statefulset-0" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize.

Same error with 'ibmc-file-gold'.

[ShuangMen]

I just edit the pvc, change it to request 40Gi.

$ k get pvc pxc-data-database-0 -n kubecf -o yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: vpc.block.csi.ibm.io
    volume.kubernetes.io/selected-node: 10.240.1.13
  creationTimestamp: "2020-08-31T03:53:59Z"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    quarks.cloudfoundry.org/az-index: "0"
    quarks.cloudfoundry.org/quarks-statefulset-name: database
  name: pxc-data-database-0
  namespace: kubecf
  resourceVersion: "13728907"
  selfLink: /api/v1/namespaces/kubecf/persistentvolumeclaims/pxc-data-database-0
  uid: ba41790c-1537-4944-81d7-2f101b5d6333
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 40Gi
  storageClassName: new-ibmc-vpc-block-10iops-tier
  volumeMode: Filesystem
  volumeName: pvc-ba41790c-1537-4944-81d7-2f101b5d6333
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 20Gi
  phase: Bound

But after the above update, nothing changed.

$ k get pv -n kubecf
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                                  STORAGECLASS                     REASON   AGE
pvc-74b3326d-85d6-48e3-a0b1-efc38b303124   100Gi      RWO            Delete           Bound    kubecf/singleton-blobstore-pvc-singleton-blobstore-0   new-ibmc-vpc-block-10iops-tier            2d4h
pvc-7e04c4bc-3f51-4413-ab39-e618ed4ec4ea   40Gi       RWO            Delete           Bound    kubecf/diego-cell-ephemeral-diego-cell-0               new-ibmc-vpc-block-10iops-tier            2d4h
pvc-ba41790c-1537-4944-81d7-2f101b5d6333   20Gi       RWO            Delete           Bound    kubecf/pxc-data-database-0                             new-ibmc-vpc-block-10iops-tier            2d4h

$ k get pvc -n kubecf
NAME                                            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                     AGE
diego-cell-ephemeral-diego-cell-0               Bound    pvc-7e04c4bc-3f51-4413-ab39-e618ed4ec4ea   40Gi       RWO            new-ibmc-vpc-block-10iops-tier   2d4h
pxc-data-database-0                             Bound    pvc-ba41790c-1537-4944-81d7-2f101b5d6333   20Gi       RWO            new-ibmc-vpc-block-10iops-tier   2d4h
singleton-blobstore-pvc-singleton-blobstore-0   Bound    pvc-74b3326d-85d6-48e3-a0b1-efc38b303124   100Gi      RWO            new-ibmc-vpc-block-10iops-tier   2d4h

[manno]

I guess k8s doesn't automatically resize, because the storageclass is missing the allowVolumeExpansion.

We resized an underlying volume manually in the past, then shelled (k9s, ...) into the container and ran resize2fs.