CSI: implement ExpandVolume

[tgross]

Splitting out of #8212, as we're most likely going to ship the create and snapshot workflows in Nomad 1.1.0 without having shipped ExpandVolume. WIP PR is #10281

From our internal design doc:

The Resize workflow tells the Storage Provider to alter the size of an existing volume. Because Nomad does not treat disk resources as a scheduling concern, this doesn’t impact our existing scheduling decisions. Nomad doesn’t implement any of this workflow currently, but most of the RPC/HTTP plumbing can be lifted from the existing Controller and Node RPCs for register/deregister/status.

We do need to validate whether the volume has “online” resizing and potentially return an error if the volume is claimed. The volume resizing operations themselves are dependent on the CSI plugin: cloud provider block storage (ex. AWS EBS, Digital Ocean block store) are less likely to have this option, whereas NFS-like or clustered file systems (gluster, Ceph) are more likely. When a volume resize request is sent, we’ll see the following workflow:
*If the plugin does not support resizing, the user will get an error saying it’s unsupported for the plugin.

• If the plugin supports resizing but not online resizing, and there are live claims, the user will get an error saying the volume can’t be resized while claimed, and identify the allocations that currently claims it.
• If the plugin supports resizing including online resizing, the storage backend will get a resize operation. We’ll block on this request. Once it is complete, we’ll return a success or failure message to the user.

Command line will be: nomad volume resize :volume_id -sizeNG [-secretkey=value]​
Takes the volume ID and a -size flag that accepts the usual size strings (ex ​-size 10G​ for 10GiB).

[tgross]

Rather than having a new CLI option for this, I'd like to update the nomad volume register and nomad volume create commands to automatically detect the case where we're trying to resize and send the ExpandVolume if permitted, returning an error if not.

[taveraluis]

@tgross is this expected to ship in a recent release? Is this actively being worked on?

[tgross]

I have a long-stale PR open for it in #10281 but we're not planning on shipping it in the upcoming Nomad 1.3.0. From a design perspective I want to resolve #11899 first, so that volume commands are more declarative the way that job commands are.

[thatsk]

Hi tim when we are planning to ship in nomad in which version

[tgross]

Hi @thatsk There's an open PR I worked on a long while ago #10281 but I haven't had a chance to get back to it. It's not currently on the roadmap so if I manage to ship it, it'll be unplanned work. In general though, we don't publish a roadmap.

[thatsk]

K but it's good use case where csi volume can resize from volume defination and use iac in storage management to keep up this Strong contender in container orchestration.

[thatsk]

Hoping it should get on roadmap and release soon to use it nomad with CSI

[digi-talo]

+1 for this, bit of a showstopper that you can't resize volumes after creation.

[gulducat]

Quick update: Work on this is underway. The interface for it will likely change however.

The process will be to update the capacity_min and capacity_max values in the volume specification declaratively, then issue a volume create or register (TBD) call with the whole volume spec. Nomad will then reconcile the new values by sending (Controller|Node)ExpandVolume requests to the CSI plugins as needed.

[gulducat]

Alrighty folks, this is feature complete, slated for release with Nomad 1.6.3 when that goes out. If you'd like to pull and build the release/1.6.x branch and try it out ahead of release, I'd be glad to hear how it works for you!