Unable to manage PVCs with a user different from client.admin

[rascasoft]

Describe the bug

Following with success every step of the official procedure [1] to configure everything, I'm unable to use a user different from client.admin to create PVCs on the configured ceph storage.

[1] https://docs.ceph.com/docs/master/rbd/rbd-kubernetes/

Environment details

• Image/version of Ceph CSI driver: canary (as from today's latest commit [2])
• No helm chart used
• Kubernetes cluster version: 1.18.0 (but the same happens with 1.17)
• Logs: see Additional Context below

[2] 2cc59ca

Steps to reproduce

Steps to reproduce the behavior:

With a 3 node Ceph cluster (nautilus) accessible from the Kubernetes cluster, I've created a pool, initiated and created a user to access it, like described in [1]:

[root@ceph-1 ~]# ceph osd pool create kubepool 64 64 replicated 
pool 'kubepool' created

[root@ceph-1 ~]# rbd pool init kubepool

[root@ceph-1 ~]# ceph auth get-or-create client.kubernetes mon 'profile rbd' osd 'profile rbd pool=kubepool' mgr 'profile rbd pool=kubepool'
[client.kubernetes]
	key = <KUBERNETESKEY>

Everything on the Kubernetes side is fine:

NAME                                             READY   STATUS    RESTARTS   AGE
pod/csi-rbdplugin-dqr2h                          3/3     Running   0          26m
pod/csi-rbdplugin-ll75l                          3/3     Running   0          26m
pod/csi-rbdplugin-provisioner-65787c77d7-6pknw   6/6     Running   0          26m
pod/csi-rbdplugin-provisioner-65787c77d7-br9b8   6/6     Running   0          26m
pod/csi-rbdplugin-provisioner-65787c77d7-gwv27   6/6     Running   0          26m
pod/csi-rbdplugin-q8ncq                          3/3     Running   0          26m

NAME                                TYPE        CLUSTER-IP	EXTERNAL-IP   PORT(S)    AGE
service/csi-metrics-rbdplugin       ClusterIP   10.97.170.165   <none>        8080/TCP   26m
service/csi-rbdplugin-provisioner   ClusterIP   10.96.219.128   <none>        8080/TCP   26m
service/kubernetes                  ClusterIP   10.96.0.1	<none>        443/TCP    23h

NAME                           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/csi-rbdplugin   3         3         3	   3            3           <none>          26m

NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/csi-rbdplugin-provisioner   3/3     3            3           26m

NAME                                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/csi-rbdplugin-provisioner-65787c77d7   3         3         3	   26m

The secret is declared as follows:

---
apiVersion: v1
kind: Secret
metadata:
  name: csi-rbd-secret
  namespace: default
stringData:
  userID: kubernetes
  userKey: <KUBERNETESKEY>

So I prepare a PVC related to Ceph in this way:

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rbd-pvc
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 500Mi
  storageClassName: csi-rbd-sc

Actual results

If I try to create the PVC it stays in Pending state, and looking at the status I see:

Events:
  Type     Reason                Age                From                                                                                              Message
  ----     ------                ----               ----                                                                                              -------
  Normal   Provisioning          12s (x6 over 30s)  rbd.csi.ceph.com_csi-rbdplugin-provisioner-65787c77d7-br9b8_44b3f95f-1bcf-48a0-8278-c3024f7afea4  External provisioner is provisioning volume for claim "default/rbd-pvc"
  Warning  ProvisioningFailed    11s (x6 over 30s)  rbd.csi.ceph.com_csi-rbdplugin-provisioner-65787c77d7-br9b8_44b3f95f-1bcf-48a0-8278-c3024f7afea4  failed to provision volume with StorageClass "csi-rbd-sc": rpc error: code = Internal desc = failed to get IOContext: failed to get connection: connecting failed: rados: ret=1, Operation not permitted
  Normal   ExternalProvisioning  5s (x4 over 30s)   persistentvolume-controller                                                                       waiting for a volume to be created, either by external provisioner "rbd.csi.ceph.com" or manually created by system administrator

So for some reason there's a permission issue.

Expected behavior

It should've worked fine, making the status of the PVC Bound.

Additional context

This do not happens if I change the secret to rely on the admin key, like:

---
apiVersion: v1
kind: Secret
metadata:
  name: csi-rbd-secret
  namespace: default
stringData:
  userID: admin
  userKey: <ADMINKEY>

Replacing the actual secret, result is:

Events:
  Type    Reason                 Age   From                                                                                              Message
  ----    ------                 ----  ----                                                                                              -------
  Normal  ExternalProvisioning   7s    persistentvolume-controller                                                                       waiting for a volume to be created, either by external provisioner "rbd.csi.ceph.com" or manually created by system administrator
  Normal  Provisioning           7s    rbd.csi.ceph.com_csi-rbdplugin-provisioner-65787c77d7-br9b8_44b3f95f-1bcf-48a0-8278-c3024f7afea4  External provisioner is provisioning volume for claim "default/rbd-pvc"
  Normal  ProvisioningSucceeded  6s    rbd.csi.ceph.com_csi-rbdplugin-provisioner-65787c77d7-br9b8_44b3f95f-1bcf-48a0-8278-c3024f7afea4  Successfully provisioned volume pvc-75a71ee7-3c4d-4931-bed7-9e5a26090cdb

and the PVC status is Bound as expected:

[root@kubernetes-1 kubernetes]# kubectl get pvc
NAME      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
rbd-pvc   Bound    pvc-75a71ee7-3c4d-4931-bed7-9e5a26090cdb   500Mi      RWO            csi-rbd-sc     19m

[Madhu-1]

closing this as duplicate of #904

[rascasoft]

closing this as duplicate of #904

Thanks, didn't check the related issue, my fault. Just for future reference (and future comers), it was enough to use the 4 yamls from the specific release, v2.0.1:

kubectl create -f https://raw.githubusercontent.com/ceph/ceph-csi/v2.0.1/deploy/rbd/kubernetes/csi-provisioner-rbac.yaml
kubectl create -f https://raw.githubusercontent.com/ceph/ceph-csi/v2.0.1/deploy/rbd/kubernetes/csi-nodeplugin-rbac.yaml
kubectl create -f https://raw.githubusercontent.com/ceph/ceph-csi/v2.0.1/deploy/rbd/kubernetes/csi-rbdplugin-provisioner.yaml
kubectl create -f https://raw.githubusercontent.com/ceph/ceph-csi/v2.0.1/deploy/rbd/kubernetes/csi-rbdplugin.yaml

and everything worked fine.

Thanks again.

[Madhu-1]

closing this as duplicate of #904

Thanks, didn't check the related issue, my fault. Just for future reference (and future comers), it was enough to use the 4 yamls from the specific release, v2.0.1:

kubectl create -f https://raw.githubusercontent.com/ceph/ceph-csi/v2.0.1/deploy/rbd/kubernetes/csi-provisioner-rbac.yaml
kubectl create -f https://raw.githubusercontent.com/ceph/ceph-csi/v2.0.1/deploy/rbd/kubernetes/csi-nodeplugin-rbac.yaml
kubectl create -f https://raw.githubusercontent.com/ceph/ceph-csi/v2.0.1/deploy/rbd/kubernetes/csi-rbdplugin-provisioner.yaml
kubectl create -f https://raw.githubusercontent.com/ceph/ceph-csi/v2.0.1/deploy/rbd/kubernetes/csi-rbdplugin.yaml

and everything worked fine.

Thanks again.

Yes we have this issue only in canary image (master branch)