Restore of PVC fail with error - plugin panicked: runtime error: invalid memory address or nil pointer dereference

[rohansadale]

What steps did you take and what happened:
I was trying to backup and restore a namespace. A busybox pod is running in the namespace and has a PVC attached.
PVC was provisioned by following the steps on https://github.com/openebs/cstor-operators/blob/master/docs/tutorial/volumes/snapshot.md

Steps followed for backup/restore

Create a namespace and deploy a pod.

root@rohan-virtual-machine:/home/rohan/openebs/cstor-operators# kubectl create ns test-backup
namespace/test-backup created

root@rohan-virtual-machine:/home/rohan/openebs/cstor-operators# kubectl get pvc -n test-backup
NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS       AGE
demo-cstor-vol   Bound    pvc-d85191ca-0b87-4dbd-a8ff-57503d4ef8d4   5Gi        RWO            cstor-csi-stripe   52s

root@rohan-virtual-machine:/home/rohan/openebs/cstor-operators# kubectl get all -n test-backup
NAME          READY   STATUS    RESTARTS   AGE
pod/busybox   1/1     Running   0          14s

apiVersion: velero.io/v1
kind: VolumeSnapshotLocation
metadata:
  # name -- volumeSnapshotLocation Name (local-default...)
  name: openebs-local
  namespace: velero
spec:
  provider: openebs.io/cstor-blockstore
  config:
    namespace: openebs
    local: "true"

Create backup job

root@rohan-virtual-machine:/home/rohan/openebs/cstor-operators# velero backup create test-localbackup --include-namespaces=test-backup --snapshot-volumes --volume-snapshot-locations=openebs-local --storage-location k8s-view-01
Backup request "test-localbackup" submitted successfully.
Run `velero backup describe test-localbackup` or `velero backup logs test-localbackup` for more details.


root@rohan-virtual-machine:/home/rohan/openebs/cstor-operators# velero backup describe test-localbackup --details
Name:         test-localbackup
Namespace:    velero
Labels:       velero.io/storage-location=k8s-view-01
Annotations:  velero.io/source-cluster-k8s-gitversion=v1.18.3
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=18

Phase:  Completed

Namespaces:
  Included:  test-backup
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Storage Location:  k8s-view-01

Velero-Native Snapshot PVs:  true

TTL:  720h0m0s

Hooks:  <none>

Backup Format Version:  1

Started:    2020-08-20 00:49:46 -0700 PDT
Completed:  2020-08-20 00:49:49 -0700 PDT

Expiration:  2020-09-19 00:49:46 -0700 PDT

Total items to be backed up:  15
Items backed up:              15

Resource List:
  v1/Event:
    - test-backup/busybox.162cea6ae5e0cb59
    - test-backup/busybox.162cea6aec12f771
    - test-backup/busybox.162cea6ceb3cdf70
    - test-backup/busybox.162cea6d3bd1e3cc
    - test-backup/busybox.162cea6d40cbb003
    - test-backup/busybox.162cea6d4dc38d2f
    - test-backup/demo-cstor-vol.162cea641ca39795
    - test-backup/demo-cstor-vol.162cea641ce77755
    - test-backup/demo-cstor-vol.162cea641f5590fd
  v1/Namespace:
    - test-backup
  v1/PersistentVolume:
    - pvc-d85191ca-0b87-4dbd-a8ff-57503d4ef8d4
  v1/PersistentVolumeClaim:
    - test-backup/demo-cstor-vol
  v1/Pod:
    - test-backup/busybox
  v1/Secret:
    - test-backup/default-token-rwzlf
  v1/ServiceAccount:
    - test-backup/default

Velero-Native Snapshots:
  pvc-d85191ca-0b87-4dbd-a8ff-57503d4ef8d4:
    Snapshot ID:        pvc-d85191ca-0b87-4dbd-a8ff-57503d4ef8d4-velero-bkp-test-localbackup
    Type:               cstor-snapshot
    Availability Zone:
    IOPS:               <N/A>

Restore from that backup

root@rohan-virtual-machine:/home/rohan/openebs/cstor-operators# velero restore create --from-backup test-localbackup --restore-volumes=true --namespace-mappings test-backup:restored-test-backup
Restore request "test-localbackup-20200820005458" submitted successfully.
Run `velero restore describe test-localbackup-20200820005458` or `velero restore logs test-localbackup-20200820005458` for more details.


root@rohan-virtual-machine:/home/rohan/openebs/cstor-operators# velero restore describe test-localbackup-20200820005458
Name:         test-localbackup-20200820005458
Namespace:    velero
Labels:       <none>
Annotations:  <none>

Phase:  PartiallyFailed (run 'velero restore logs test-localbackup-20200820005458' for more information)

Errors:
  Velero:     <none>
  Cluster:  error executing PVAction for persistentvolumes/pvc-d85191ca-0b87-4dbd-a8ff-57503d4ef8d4: rpc error: code = Aborted desc = plugin panicked: runtime error: invalid memory address or nil pointer dereference
  Namespaces: <none>

Backup:  test-localbackup

Namespaces:
  Included:  all namespaces found in the backup
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io
  Cluster-scoped:  auto

Namespace mappings:  test-backup=restored-test-backup

Label selector:  <none>

Restore PVs:  true

PVC details

root@rohan-virtual-machine:/home/rohan/openebs/cstor-operators# kubectl get pvc -A
NAMESPACE              NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS       AGE
restored-test-backup   demo-cstor-vol   Lost     pvc-d85191ca-0b87-4dbd-a8ff-57503d4ef8d4   0                         cstor-csi-stripe   36s
test-backup            demo-cstor-vol   Bound    pvc-d85191ca-0b87-4dbd-a8ff-57503d4ef8d4   5Gi        RWO            cstor-csi-stripe   7m33s


root@rohan-virtual-machine:/home/rohan/openebs/cstor-operators# kubectl -n restored-test-backup describe pvc demo-cstor-vol
Name:          demo-cstor-vol
Namespace:     restored-test-backup
StorageClass:  cstor-csi-stripe
Status:        Lost
Volume:        pvc-d85191ca-0b87-4dbd-a8ff-57503d4ef8d4
Labels:        velero.io/backup-name=test-localbackup
               velero.io/restore-name=test-localbackup-20200820005458
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: cstor.csi.openebs.io
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      0
Access Modes:
VolumeMode:    Filesystem
Mounted By:    busybox
Events:
  Type     Reason         Age   From                         Message
  ----     ------         ----  ----                         -------
  Warning  ClaimMisbound  113s  persistentvolume-controller  Two claims are bound to the same volume, this one is bound incorrectly

What did you expect to happen:

Restore should have been successful and PVC should have been attached to the pod

Environment:

• Velero version (use velero version): v1.4.0
• Velero features (use velero client config get features):
• Velero-plugin version
• OpenEBS version 1.11
• Kubernetes version (use kubectl version): 1.18
• Kubernetes installer & version:
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):

NAME="Ubuntu"
VERSION="20.04 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

[mittachaitu]

@rohansadale able to reproduce in local setups and we will raise a PR shortly.