Volumes with csi- prefix stoped working (volume not present in storage backend) #7

kvaps · 2019-03-27T00:35:32Z

Hi, I've just updated csi-provisioner, which is probably implements this fix #3.

Now when I can create new PVCs, volumes are creating for them with the name like: pvc-bbd4682a-5024-11e9-82fd-c2b1dc5386a1 - this is working fine.

But old volumes with the name like csi-0964269a-b154-43a1-876c-d15989446479 are stopped working at all:

csi-attacher log:

I0327 00:31:03.233747       1 controller.go:173] Started VA processing "csi-c6daec176b656194f5c0e70768581463229899fa56b274fb062b27de39ee64ec"
I0327 00:31:03.233778       1 csi_handler.go:93] CSIHandler: processing VA "csi-c6daec176b656194f5c0e70768581463229899fa56b274fb062b27de39ee64ec"
I0327 00:31:03.233787       1 csi_handler.go:120] Attaching "csi-c6daec176b656194f5c0e70768581463229899fa56b274fb062b27de39ee64ec"
I0327 00:31:03.233793       1 csi_handler.go:259] Starting attach operation for "csi-c6daec176b656194f5c0e70768581463229899fa56b274fb062b27de39ee64ec"
I0327 00:31:03.233846       1 csi_handler.go:214] PV finalizer is already set on "pvc-86e46c8e-3ab0-11e9-baf1-22b1a44f6c4f"
I0327 00:31:03.236093       1 csi_handler.go:524] Can't get CSINodeInfo pve1: csinodeinfos.csi.storage.k8s.io "pve1" not found
I0327 00:31:03.236158       1 csi_handler.go:173] VA finalizer is already set on "csi-c6daec176b656194f5c0e70768581463229899fa56b274fb062b27de39ee64ec"
I0327 00:31:03.236166       1 csi_handler.go:187] NodeID annotation is already set on "csi-c6daec176b656194f5c0e70768581463229899fa56b274fb062b27de39ee64ec"
I0327 00:31:03.236180       1 connection.go:242] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I0327 00:31:03.236187       1 connection.go:243] GRPC request: {"node_id":"pve1","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"storage.kubernetes.io/csiProvisionerIdentity":"1551278453900-8081-io.drbd.linstor-csi"},"volume_id":"deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23"}
I0327 00:31:03.330688       1 connection.go:245] GRPC response: {}
I0327 00:31:03.331017       1 connection.go:246] GRPC error: rpc error: code = NotFound desc = ControllerPublishVolume failed for deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23: volume not present in storage backend
I0327 00:31:03.331030       1 csi_handler.go:388] Saving attach error to "csi-c6daec176b656194f5c0e70768581463229899fa56b274fb062b27de39ee64ec"
I0327 00:31:03.333783       1 controller.go:139] Ignoring VolumeAttachment "csi-c6daec176b656194f5c0e70768581463229899fa56b274fb062b27de39ee64ec" change
I0327 00:31:03.333827       1 csi_handler.go:398] Saved attach error to "csi-c6daec176b656194f5c0e70768581463229899fa56b274fb062b27de39ee64ec"
I0327 00:31:03.333844       1 csi_handler.go:103] Error processing "csi-c6daec176b656194f5c0e70768581463229899fa56b274fb062b27de39ee64ec": failed to attach: rpc error: code = NotFound desc = ControllerPublishVolume failed for deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23: volume not present in storage backend

Example PV:

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: io.drbd.linstor-csi
  creationTimestamp: "2019-02-27T16:57:07Z"
  finalizers:
  - kubernetes.io/pv-protection
  - external-attacher/io-drbd-linstor-csi
  name: pvc-86e46c8e-3ab0-11e9-baf1-22b1a44f6c4f
  resourceVersion: "40803092"
  selfLink: /api/v1/persistentvolumes/pvc-86e46c8e-3ab0-11e9-baf1-22b1a44f6c4f
  uid: b7316819-3ab0-11e9-8fc7-f6cd6192cc92
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 50Mi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: eig
    namespace: default
    resourceVersion: "29068182"
    uid: 86e46c8e-3ab0-11e9-baf1-22b1a44f6c4f
  csi:
    driver: io.drbd.linstor-csi
    fsType: ext4
    volumeAttributes:
      storage.kubernetes.io/csiProvisionerIdentity: 1551278453900-8081-io.drbd.linstor-csi
    volumeHandle: deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23
  persistentVolumeReclaimPolicy: Delete
  storageClassName: linstor-basic-storage
  volumeMode: Filesystem
status:
  phase: Bound

Linstor does not provides any opportunity for rename already created resources, and Kubernetes does not provides to change spec.csi.volumeHandle for the already created PV.

Does it mean, that this situation have no other solution than manually edit linstor database for remove csi- prefix?

The text was updated successfully, but these errors were encountered:

kvaps · 2019-03-27T13:33:43Z

I've more investigated in this, I have the next behavior:

If spec.csi.volumeHandle equals deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23 (this is PV crerated by old linstor-csi-provisioner), I have an error volume not present in storage backend:

time="2019-03-27T13:27:52Z" level=debug msg="looking up resource by CSI volume ID" func="github.com/LINBIT/linstor-csi/pkg/client.(*Linstor).GetByID" file="/home/hayley/projects/linstor-csi/pkg/client/linstor.go:257" annotationsKey=csi-volume-annotations controllers="localhost:3376" csiVolumeID=deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23 linstorCSIComponent=client
time="2019-03-27T13:27:52Z" level=info msg="running external command" func=github.com/LINBIT/golinstor.ResourceDeployment.traceCombinedOutput file="/home/hayley/go/pkg/mod/github.com/!l!i!n!b!i!t/[email protected]/linstor.go:337" command="linstor -m --controllers localhost:3376 resource-definition list" package=golinstor resourceName=CSIGetByID
time="2019-03-27T13:27:52Z" level=error msg="method failed" func=github.com/LINBIT/linstor-csi/pkg/driver.Driver.Run.func1 file="/home/hayley/projects/linstor-csi/pkg/driver/driver.go:714" error="rpc error: code = NotFound desc = ControllerPublishVolume failed for deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23: volume not present in storage backend" linstorCSIComponent=driver method=/csi.v1.Controller/ControllerPublishVolume nodeID=pve2 provisioner=io.drbd.linstor-csi req="volume_id:\"deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23\" node_id:\"pve1\" volume_capability:<mount:<fs_type:\"ext4\" > access_mode:<mode:SINGLE_NODE_WRITER > > volume_context:<key:\"storage.kubernetes.io/csiProvisionerIdentity\" value:\"1551278453900-8081-io.drbd.linstor-csi\" > " resp="<nil>" version=v0.3.2

Otherwise if spec.csi.volumeHandle equals csi-deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23, I have an another error:

time="2019-03-27T13:29:01Z" level=debug msg="looking up resource by CSI volume ID" func="github.com/LINBIT/linstor-csi/pkg/client.(*Linstor).GetByID" file="/home/hayley/projects/linstor-csi/pkg/client/linstor.go:257" annotationsKey=csi-volume-annotations controllers="localhost:3376" csiVolumeID=csi-deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23 linstorCSIComponent=client
time="2019-03-27T13:29:01Z" level=info msg="running external command" func=github.com/LINBIT/golinstor.ResourceDeployment.traceCombinedOutput file="/home/hayley/go/pkg/mod/github.com/!l!i!n!b!i!t/[email protected]/linstor.go:337" command="linstor -m --controllers localhost:3376 resource-definition list" package=golinstor resourceName=CSIGetByID
time="2019-03-27T13:29:01Z" level=debug msg="converted resource definition to volume" func="github.com/LINBIT/linstor-csi/pkg/client.(*Linstor).resDefToVolume" file="/home/hayley/projects/linstor-csi/pkg/client/linstor.go:154" annotationsKey=csi-volume-annotations controllers="localhost:3376" linstorCSIComponent=client resourceDefinition="{VlmDfns:[{VlmDfnUUID:067622a9-4af7-408a-9869-2065cb138046 VlmMinor:1007 VlmNr:0 VlmSize:51201}] RscDfnSecret:c7qk4vl6mfuCbKFCE4EW RscDfnUUID:6f530b5f-a63c-441c-b606-d9496764db62 RscName:csi-deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23 RscDfnPort:7007 RscDfnProps:[{Value:{\"name\":\"pvc-86e46c8e-3ab0-11e9-baf1-22b1a44f6c4f\",\"id\":\"deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23\",\"createdBy\":\"io.drbd.linstor-csi\",\"creationTime\":\"2019-02-27T16:57:06.125301711Z\",\"sizeBytes\":52428800,\"readonly\":false,\"parameters\":{\"autoPlace\":\"2\",\"storagePool\":\"pve\"}} Key:Aux/csi-volume-annotations} {Value:PVE2 Key:DrbdPrimarySetOn}]}" volume="&{Name:pvc-86e46c8e-3ab0-11e9-baf1-22b1a44f6c4f ID:deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23 CreatedBy:io.drbd.linstor-csi CreationTime:2019-02-27 16:57:06.125301711 +0000 UTC SizeBytes:52428800 Readonly:false Parameters:map[autoPlace:2 storagePool:pve]}"
time="2019-03-27T13:29:01Z" level=info msg="found existing volume" func=github.com/LINBIT/linstor-csi/pkg/driver.Driver.ControllerPublishVolume file="/home/hayley/projects/linstor-csi/pkg/driver/driver.go:514" existingVolume="&{Name:pvc-86e46c8e-3ab0-11e9-baf1-22b1a44f6c4f ID:deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23 CreatedBy:io.drbd.linstor-csi CreationTime:2019-02-27 16:57:06.125301711 +0000 UTC SizeBytes:52428800 Readonly:false Parameters:map[autoPlace:2 storagePool:pve]}" linstorCSIComponent=driver nodeID=pve2 provisioner=io.drbd.linstor-csi version=v0.3.2
time="2019-03-27T13:29:01Z" level=info msg="attaching volume" func="github.com/LINBIT/linstor-csi/pkg/client.(*Linstor).Attach" file="/home/hayley/projects/linstor-csi/pkg/client/linstor.go:310" annotationsKey=csi-volume-annotations controllers="localhost:3376" linstorCSIComponent=client targetNode=pve1 volume="&{Name:pvc-86e46c8e-3ab0-11e9-baf1-22b1a44f6c4f ID:deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23 CreatedBy:io.drbd.linstor-csi CreationTime:2019-02-27 16:57:06.125301711 +0000 UTC SizeBytes:52428800 Readonly:false Parameters:map[autoPlace:2 storagePool:pve]}"
time="2019-03-27T13:29:01Z" level=info msg="running external command" func=github.com/LINBIT/golinstor.ResourceDeployment.traceCombinedOutput file="/home/hayley/go/pkg/mod/github.com/!l!i!n!b!i!t/[email protected]/linstor.go:337" command="linstor -m --controllers localhost:3376 resource list" package=golinstor resourceName=deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23
time="2019-03-27T13:29:01Z" level=info msg="running external command" func=github.com/LINBIT/golinstor.ResourceDeployment.traceCombinedOutput file="/home/hayley/go/pkg/mod/github.com/!l!i!n!b!i!t/[email protected]/linstor.go:337" command="linstor -m --controllers localhost:3376 resource create pve1 deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23 -s DfltDisklessStorPool --diskless" package=golinstor resourceName=deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23
time="2019-03-27T13:29:01Z" level=error msg="method failed" func=github.com/LINBIT/linstor-csi/pkg/driver.Driver.Run.func1 file="/home/hayley/projects/linstor-csi/pkg/driver/driver.go:714" error="rpc error: code = Internal desc = ControllerPublishVolume failed for csi-deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23: failed to run command \"[\\n  {\\n    \\\"ret_code\\\": 13835058055302349101, \\n    \\\"correction\\\": \\\"Create a resource definition with the name 'deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23' first.\\\", \\n    \\\"error_report_ids\\\": [\\n      \\\"5C9ABDCA-00000-000312\\\"\\n    ], \\n    \\\"obj_refs\\\": [\\n      {\\n        \\\"value\\\": \\\"deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23\\\", \\n        \\\"key\\\": \\\"RscDfn\\\"\\n      }\\n    ], \\n    \\\"details\\\": \\\"Node(s): 'pve1', Resource: 'deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23'\\\", \\n    \\\"message\\\": \\\"Resource definition 'deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23' not found.\\\", \\n    \\\"cause\\\": \\\"The specified resource definition 'deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23' could not be found in the database\\\"\\n  }\\n]\\n\": error status from one or more linstor operations: [{\"details_format\":\"\",\"message_format\":\"\",\"obj_refs\":[{\"value\":\"deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23\",\"key\":\"RscDfn\"}],\"variables\":null,\"ret_code\":13835058055302349101}]" linstorCSIComponent=driver method=/csi.v1.Controller/ControllerPublishVolume nodeID=pve2 provisioner=io.drbd.linstor-csi req="volume_id:\"csi-deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23\" node_id:\"pve1\" volume_capability:<mount:<fs_type:\"ext4\" > access_mode:<mode:SINGLE_NODE_WRITER > > volume_context:<key:\"storage.kubernetes.io/csiProvisionerIdentity\" value:\"1551278453900-8081-io.drbd.linstor-csi\" > " resp="<nil>" version=v0.3.2

It tries to call:

linstor -m --controllers localhost:3376 resource create pve1 deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23 -s DfltDisklessStorPool --diskless

that's weird because linstor know this resource as csi-deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23:

linstor r l -r csi-deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23
+----------------------------------------------------------------------------+
| ResourceName                             | Node | Port | Usage  |    State |
|----------------------------------------------------------------------------|
| csi-deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23 | pve2 | 7007 | Unused | UpToDate |
| csi-deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23 | pve3 | 7007 | Unused | UpToDate |
+----------------------------------------------------------------------------+

deleteriousEffect · 2019-03-27T14:09:00Z

can I get the output of linstor -m resource-definition list

kvaps · 2019-03-27T14:17:32Z

@haySwim sure:

linstor-rd-list.json.txt

deleteriousEffect · 2019-03-27T15:01:39Z

Alright, I understand the issue. Give me a day or so to think about how to solve it in an intelligent way.

kvaps · 2019-03-29T00:21:29Z

Hi @haySwim,

I solved this problem by modifying metadata of my old volumes, what have been done:

Every resource in linstor was updated property Aux/csi-volume-annotations like:

{
   "name": "pvc-a4303601-3ab0-11e9-baf1-22b1a44f6c4f",
-  "id": "9b527570-bedb-438b-9745-3bada7bbc83e",
+  "id": "csi-9b527570-bedb-438b-9745-3bada7bbc83e",
   "createdBy": "io.drbd.linstor-csi",
   "creationTime": "2019-02-27T16:57:52.909364881Z",
   "sizeBytes": 52428800,
   "readonly": false,
   "parameters": {
     "ReplicasOnDifferen": "side",
     "autoPlace": "2",
     "storagePool": "pve"
   }
 }

Every kubernetes PV was updated like:

 apiVersion: v1
 kind: PersistentVolume
 metadata:
   annotations:
     pv.kubernetes.io/provisioned-by: io.drbd.linstor-csi
   creationTimestamp: "2019-03-29T00:00:49Z"
   finalizers:
   - kubernetes.io/pv-protection
   name: pvc-a4303601-3ab0-11e9-baf1-22b1a44f6c4f
   resourceVersion: "41688434"
   selfLink: /api/v1/persistentvolumes/pvc-a4303601-3ab0-11e9-baf1-22b1a44f6c4f
   uid: b61f757e-51b5-11e9-b606-52b5c95dfe5c
 spec:
   accessModes:
   - ReadWriteOnce
   capacity:
     storage: 50Mi
   claimRef:
     apiVersion: v1
     kind: PersistentVolumeClaim
     name: foo
     namespace: default
     resourceVersion: "29068421"
     uid: a4303601-3ab0-11e9-baf1-22b1a44f6c4f
   csi:
     driver: io.drbd.linstor-csi
     fsType: ext4
     volumeAttributes:
       storage.kubernetes.io/csiProvisionerIdentity: 1551278453900-8081-io.drbd.linstor-csi
-    volumeHandle: 9b527570-bedb-438b-9745-3bada7bbc83e
+    volumeHandle: csi-9b527570-bedb-438b-9745-3bada7bbc83e
   persistentVolumeReclaimPolicy: Delete
   storageClassName: linstor-basic-storage
   volumeMode: Filesystem
 status:
   phase: Bound

After these changes old csi volumes attachment started working properly. For automate process I've prepared small script for generate right commands: https://github.com/kvaps/linstor-csi-migrator

I want to know your opinion: What do you think, is it correct way for solve this problem?
Thanks!

deleteriousEffect · 2019-03-29T17:24:12Z

I think this is a good solution and I'm happy that you found a workaround.

Right now we're at major version 0 and development is pretty active, so we're not trying especially hard to maintain backwards compatibility. I think ultimately the way that this metadata is stored is going to get changed in breaking way soon-ish, but should stabilize afterwards.

kvaps · 2019-03-29T17:36:27Z

OK, great! Hopefully next changes will be more portable to existing volumes.

deleteriousEffect closed this as completed Mar 29, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Volumes with csi- prefix stoped working (volume not present in storage backend) #7

Volumes with csi- prefix stoped working (volume not present in storage backend) #7

kvaps commented Mar 27, 2019 •

edited

Loading

kvaps commented Mar 27, 2019 •

edited

Loading

deleteriousEffect commented Mar 27, 2019

kvaps commented Mar 27, 2019

deleteriousEffect commented Mar 27, 2019

kvaps commented Mar 29, 2019

deleteriousEffect commented Mar 29, 2019

kvaps commented Mar 29, 2019

Volumes with csi- prefix stoped working (volume not present in storage backend) #7

Volumes with csi- prefix stoped working (volume not present in storage backend) #7

Comments

kvaps commented Mar 27, 2019 • edited Loading

kvaps commented Mar 27, 2019 • edited Loading

deleteriousEffect commented Mar 27, 2019

kvaps commented Mar 27, 2019

deleteriousEffect commented Mar 27, 2019

kvaps commented Mar 29, 2019

deleteriousEffect commented Mar 29, 2019

kvaps commented Mar 29, 2019

kvaps commented Mar 27, 2019 •

edited

Loading

kvaps commented Mar 27, 2019 •

edited

Loading