Failed to set fsGroup with RWX pvc

[bc185174]

Hi, I have been trying to provision a RWX pvc, where both devices can temporarily be primary (this is based on issue #159).

The issue I am seeing is the fsGroup does not get set for the RWX volume but it does for RWO volume:

kubectl exec -ti fsgroup-test-79d595bb7c-2v92x -- ls -la /
drwxrwsr-x    3 root     107           4096 Nov  7 14:10 data-rwo
drwxr-xr-x    3 root     root          4096 Nov  7 14:10 data-rwx

Reproducing the issue

1. Install Linstor configured with lvm thick pool.

2. Create StorageClass:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: linstor-cluster-storage
  annotations:
    storageclass.kubernetes.io/is-default-class: "false"
provisioner: linstor.csi.linbit.com
parameters:
  autoPlace: "2"
  storagePool: "lvm-thick"
  DrbdOptions/Net/allow-two-primaries: "yes"
allowVolumeExpansion: true

3. Create Test pod:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: fsgroup-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: fsgroup-test
  template:
    metadata:
      labels:
        app: fsgroup-test
    spec:
      securityContext:
        fsGroup: 107
        runAsUser: 107
      containers:
        - name: debug
          image: alpine:3.11
          command: ["/bin/sh"]
          stdin: true
          tty: true
          volumeMounts:
          - mountPath: /data-rwx
            name: linstor-fsgroup-test-rwx
          - mountPath: /data-rwo
            name: linstor-fsgroup-test-rwo
      volumes:
      - name: linstor-fsgroup-test-rwx
        persistentVolumeClaim:
          claimName: linstor-fsgroup-test-rwx
          readOnly: false
      - name: linstor-fsgroup-test-rwo
        persistentVolumeClaim:
          claimName: linstor-fsgroup-test-rwo
          readOnly: false
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: linstor-fsgroup-test-rwx
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  storageClassName: linstor-cluster-storage
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: linstor-fsgroup-test-rwo
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: linstor-cluster-storage

Note: post provisioning, I set the user to root and I could change the fs group manually.

[WanzenBug]

Without looking to far into it: RWX volumes are not supported, and probably never will be.

The "allow-two-primary" option exists only for very specific edge cases, such as:

• The volume is used as a block device by something like KVM during live migration
• You are using something like GPFS (which you don't want to, unless you have to)

Using it with a normal volume is a great way to corrupt your filesystem and lose all data.

I'm honestly not sure why it even let's you provision a volume with these settings, I guess that is the real bug...

[bc185174]

Without looking to far into it: RWX volumes are not supported, and probably never will be.

The "allow-two-primary" option exists only for very specific edge cases, such as:

• The volume is used as a block device by something like KVM during live migration
• You are using something like GPFS (which you don't want to, unless you have to)

Using it with a normal volume is a great way to corrupt your filesystem and lose all data.

I'm honestly not sure why it even let's you provision a volume with these settings, I guess that is the real bug...

Thank you for the reply - its based on kubevirt/containerized-data-importer#2447, so I think my next step is to solve the block-device permissions issue.

Is there a reason not to support RWX volumes for this edge case? I have considered NFS storage (ganesha); however, we have found that there is degraded performance in comparison.

[WanzenBug]

Is there a reason not to support RWX volumes for this edge case?

Because it's hard. For the same reason that ganesha is slow, probably. We configure a normal file system on top of DRBD. This is fast and works well, because DRBD ensures that only one node can have the volume mounted. This means you can have the normal advantages of a local file system (like local caching), while the data is replicated in the background.

For another node to access the data, the first node needs to unmount the volume, which also ensures all cached data is flushed to disk. Only after the first node has unmounted the volume can the other node can mount the volume itself.

As you discovered, there is an escape-hatch with allow-two-primaries: "yes". This is only valid if whatever is accessing the DRBD device has some other synchronization build in. And if you use a normal filesystem, that is simply not the case.

For the issue with block volumes, I suspect it is related to these settings on the driver: https://kubernetes-csi.github.io/docs/support-fsgroup.html#supported-modes We use the ReadWriteOnceWithFSType setting, as that is the "old" behaviour, but I think I tested File once and ran into issues.