MountPropagation=false by default breaks CSI plugins for Kubernetes v1.10+

[yasker]

The Kubernetes standard way of deploying CSI plugins includes a driver pod on each host, which will need to mount the device and propagate back to the host in order to be used by the user pod.

MountProgation=false breaks CSI plugins. And it will do so silently since there is no way CSI plugin can know that the mount cannot be propagated back to the host (somehow the request for MountPropagationBidirectional wasn't denied by the Kubernetes API server). So the user pod will receive an empty directory for the volume which appears to be not suspicous at all at first glance. But the data would be lost once the pod was destroyed since it's not at persistent storage at all.

I believe that's one of the reasons that MountProgation=true is the default value for Kubernetes v1.10, which also enabled CSI by default.

@ibuildthecloud @galal-hussein

[deniseschannon]

Currently, we've set MountProgation=false even though it's enabled to true for k8s 1.10 to allow b2d and vSphere to work in Rancher. We need to update to be able to set it to true and have a different approach to get v2d/vsphere to work.

[dryprogrammer]

I believe this is the reason StorageOS doesn't work with the current Rancher. I hope to try Rancher and StorageOS once this is fixed, as the helm charts(ex. WordPress) with PVC with subPath are not working for me with Local Host Path.
Here is the StorageOS thread:
https://forums.rancher.com/t/storageos-rancher-and-kubelet/10128/7

[moelsayed]

Tested using 0.1.9-rc6. All mounts are defaulting to rprivate as expected.

[yasker]

@moelsayed At least the feature mount propagation is enabled, the bidirectional mount works right?

[fatih]

Hi,

I also received some reports that our CSI plugin for DigitalOcean doesn't work: digitalocean/csi-digitalocean#41 After some research I found that the mounts are all mounted private and not shared as it should be. This can be validated via findmnt:

/ # findmnt -o TARGET,PROPAGATION /dev/disk/by-id/scsi-0DO_Volume_pvc-232d4824-95a0-11e8-a249-4ef7fc9c8e48
TARGET                                                                                                                              PROPAGATION
/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-232d4824-95a0-11e8-a249-4ef7fc9c8e48/globalmount                                  private
/var/lib/kubelet/pods/adcbcc68-95a0-11e8-a249-4ef7fc9c8e48/volumes/kubernetes.io~csi/pvc-232d4824-95a0-11e8-a249-4ef7fc9c8e48/mount private

As you see the mounts are propagated as private

I installed rancher via this command:

sudo docker run -d --restart=unless-stopped -p 80:80 -p 443:443 rancher/rancher

So, based on: #729 adding the following to config.yaml fixes the issue:

services:
  kube-api:
    extra_args:
      feature-gates: MountPropagation=true

  kubelet:
    extra_args:
      feature-gates: MountPropagation=true

Propagation is also fixed and I can now see the correct value (shared):

/ # findmnt -o TARGET,PROPAGATION /dev/disk/by-id/scsi-0DO_Volume_pvc-a021d476-95a3-11e8-bfa3-4ef7fc9c8e48
TARGET                                                                                                                              PROPAGATION
/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-a021d476-95a3-11e8-bfa3-4ef7fc9c8e48/globalmount                                  shared
/var/lib/kubelet/pods/a38437cc-95a3-11e8-bfa3-4ef7fc9c8e48/volumes/kubernetes.io~csi/pvc-a021d476-95a3-11e8-bfa3-4ef7fc9c8e48/mount shared

Having said that, what is the recommended way of using this feature now? I also see this is fixed here: https://github.com/rancher/types/pull/499/files, if that is the case, what's the latest Rancher version we can recommend to our CSI plugin users?

Thanks