Support CreateVolume

[huchengze]

A complete csi plugin needs to finish its controllerserver.go
https://github.com/kubernetes-csi/csi-driver-nfs/blob/master/pkg/nfs/controllerserver.go.

We can use the code in https://github.com/kubernetes-incubator/external-storage/blob/master/nfs-client/cmd/nfs-client-provisioner/provisioner.go as comparison.

[wozniakjan]

Hi @huchengze, for now the development of this plugin is a little bit slower due to no dedicated full time maintainer, only few hobbyists slowly grinding through the unfinished tasks in their free time. No immediate plan for finishing that in next weeks afaik but if you'd like to submit a PR, we appreciate all help and gladly review to successful merge :)

[msau42]

I do have an implementation of the nfs-client equivalent behavior that provisions subdirectories from an existing nfs server. I would be happy to contribute if you think the nfs-client-like provisioner is the way to go.

Another alternative is that the provisioner creates an entire nfs server, like the ganesha-based nfs external provisioner.

I think it would be hard for a single driver to support both. I think we should just pick one mode for simplicity. There was an attempt in the past to support different provisioning modules via go plugins, but I think there were some challenges with that model: #13

cc @kmova for thoughts

[wozniakjan]

that provisions subdirectories from an existing nfs server

that sounds like a solid implementation, I am in favor of supporting this approach. There might be a path to also support multiple NFS servers in cluster but I think it would make sense to have separate StorageClass for each NFS server and the plugin can obtain connection information from the StorageClass. Kind of like the old ganesha provisioner but without spinning up the entire NFS server, only as a reference point to one created independently.

There was an attempt in the past to support different provisioning modules via go plugins, but I think there were some challenges with that model: #13

this is a very good resource, thanks. Let's see if we can resurrect that PR, I could find some spare time in early May. @huchengze also feel free to chime in, every idea is welcome.

[msau42]

/reopen

[k8s-ci-robot]

@msau42: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[msau42]

/kind feature

[msau42]

Here's an example StorageClass for my nfs-client equivalent implementation:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-nfs
provisioner: csi-nfsplugin
parameters:
  server: 10.123.42.242
  base-dir: vol1

You specify the nfs share you want to use as the "base". And with every CreateVolume call, it mounts that share, and creates a subdirectory under it, then unmounts it.

[kmova]

I agree with this approach.

I think it would be hard for a single driver to support both. I think we should just pick one mode for simplicity.

@msau42 @wozniakjan - couple of questions related to the scope/implementation:

• Will csi-driver-nfs be considered a replacement for the in-tree NFS driver? Should it support:

  • Managing PV with NFS spec.
  • Managing PV with NFS spec that includes sub-directories

• What are the advantages of going with CSI driver approach for NFS sub directory volumes as opposed to using external-provisioner? One advantage I see is that we can have node driver potentially support volume metrics. (Though metrics could be done via daemonset, CSI approach seems cleaner) . What else?

[msau42]

I think from a driver point of view, we can't really tell the difference between a top level share and a subdirectory. The "share' in the volumeAttributes can be either:

    volumeAttributes: 
      server: 127.0.0.1
      share: /export

The main advantages of the csi driver over the nfs-client external provisioner is that we can potentially support CSI-only features like snapshots and cloning, although a generic driver will be a bit limited in what we can offer. I think snapshots/cloning would basically just be copying directories to another place in the nfs share.

[msau42]

Supporting volume metrics is another useful feature. Do you know if "df" works if you mount a subdirectory in an nfs share?

[msau42]

/assign

[kmova]

Typically, the df -h of a sub-directory will show the usage details of the main export path. Except for NFS implementations that can treat subdirectory as a separate volume.

df -h is useful when using full block devices. The information it provides can be misleading when using subdirectories. du is safer, but expensive when there are lots of subdirectories and files.

Example:

NFS Server does the following to provide /export.

• Format a block device /dev/sdb with ext4
• Mounts the /dev/sdb as /mnt/nfs-export
• Add the export path saying /mnt/nfs-export => /export

NFS client df -h on /export will match NFS Server df -h on /mnt/nfs-export
NFS client df -h on /export/subdir will also be same as NFS Server df -h on /mnt/nfs-export

Assume a slight variation to the above example, where NFS server uses a hostpath like: /var/export => /export, where /var/export is a subdirectory on OS disk itself. df -h for /export and /export/subdir will be that of the OS disk from the NFS server.

One case where I have seen NFS sub directories showing their own capacity and usage are NFS volumes backed by ZFS datasets.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[andyzhangx]

I think we can do an empty implementation first on CreateVolume func to support storage class first, that's what I have done on SMB CSI driver:
https://github.com/kubernetes-csi/csi-driver-smb/blob/master/deploy/example/e2e_usage.md#option1-storage-class-usage

Shall I do the same implementation on this NFS driver? @msau42
there is already discussion here: kubernetes-csi/csi-driver-smb#61 (comment)

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[msau42]

/remove-lifecycle rotten

I haven't had time to work on this. If anyone is interested, I can push my incomplete workspace and someone can take over.

[andyzhangx]

/remove-lifecycle rotten

I haven't had time to work on this. If anyone is interested, I can push my incomplete workspace and someone can take over.

@msau42 could you push your code somewhere? I have one intern looking at this first.
And what about implement an empty CreateVolume first? that would support storage class first(still need bring your own nfs server first), refer to https://github.com/kubernetes-csi/csi-driver-smb/blob/ca5f7e2fc6ae8cc49e9c47d3311eab0d8486d22e/pkg/smb/controllerserver.go#L28-L40

[msau42]

Here's my branch, happy to help walk through it if needed: msau42@da0e3d6

Re: empty CreateVolume, I think not supporting delete is a downside. I would rather try to go with nfs-client behavior (which I think is close) instead of adding this and having people depend on it, and then removing it in the future.

[andyzhangx]

Here's my branch, happy to help walk through it if needed: msau42@da0e3d6

Re: empty CreateVolume, I think not supporting delete is a downside. I would rather try to go with nfs-client behavior (which I think is close) instead of adding this and having people depend on it, and then removing it in the future.

@msau42 thanks, I see, I will start with an e2e test first and then continue with this PR, I see this PR is close to complete, thanks.