Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CSI Docs for K8s v1.10 #7698

Merged
merged 1 commit into from
Mar 13, 2018
Merged

CSI Docs for K8s v1.10 #7698

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
107 changes: 77 additions & 30 deletions docs/concepts/storage/volumes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,6 @@ Kubernetes supports several types of Volumes:
* `azureFile`
* `cephfs`
* `configMap`
* `csi`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@saad-ali why are we removing this? I can understand the CSI plugin is out-of-tree, but from users' perspective, they are still called csi volumes, right?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For consistency. FlexVolume is not listed here either. I'm not sure this list adds much value, considering the Table of Content above captures the list of volume types already and is easier to keep updated. Any objection to removing this list altogether? Might be a good idea to move "Out of Tree" volumes immediately after the list of in tree volumes?

* `downwardAPI`
* `emptyDir`
* `fc` (fibre channel)
Expand Down Expand Up @@ -972,50 +971,98 @@ specification, and to select the type of media to use, for clusters that have
several media types.

## Out-of-Tree Volume Plugins
In addition to the previously listed volume types, storage vendors may create
custom plugins without adding it to the Kubernetes repository. This can be
achieved by using either the `CSI` plugin or the `FlexVolume` plugin.
The Out-of-tree volume plugins include the Container Storage Interface (`CSI`)
and `FlexVolume`. They enable storage vendors to create custom storage plugins
without adding them to the Kubernetes repository.

For storage vendors looking to create an out-of-tree volume plugin, [please refer to this FAQ](https://github.com/kubernetes/community/blob/master/sig-storage/volume-plugin-faq.md) for choosing between the plugin options.
Before the introduction of `CSI` and `FlexVolume`, all volume plugins (like
volume types listed above) were "in-tree" meaning they were built, linked,
compiled, and shipped with the core Kubernetes binaries and extend the core
Kubernetes API. This meant that adding a new storage system to Kubernetes (a
volume plugin) required checking code into the core Kubernetes code repository.

Both `CSI` and `FlexVolume` allow volume plugins to be developed independent of
the Kubernetes code base, and deployed (installed) on Kubernetes clusters as
extensions.

For storage vendors looking to create an out-of-tree volume plugin, please refer
to [this FAQ](https://github.com/kubernetes/community/blob/master/sig-storage/volume-plugin-faq.md).

### CSI

CSI stands for [Container Storage Interface](https://github.com/container-storage-interface/spec/blob/master/spec.md),
a specification attempting to establish an industry standard interface that
container orchestration systems can use to expose arbitrary storage systems
to their container workloads.
Please read
[CSI design proposal](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/container-storage-interface.md) for further information.
{% assign for_k8s_version="v1.10" %}{% include feature-state-beta.md %}

[Container Storage Interface](https://github.com/container-storage-interface/spec/blob/master/spec.md) (CSI)
defines a standard interface for container orchestration systems (like
Kubernetes) to expose arbitrary storage systems to their container workloads.

<!-- TODO: add link to the kubernetes extension section -->
The `csi` volume type is an in-tree CSI volume plugin for Pods to interact
with external CSI volume drivers running on the same node.
After having deployed a CSI compatible volume driver, users can use `csi` as the
volume type to mount the storage provided by the driver.
Please read the [CSI design proposal](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/container-storage-interface.md) for more information.

CSI persistent volume support is an alpha feature in Kubernetes v1.9 and requires a
cluster administrator to enable it. To enable CSI persistent volume support, the
cluster administrator adds `CSIPersistentVolume=true` to the `--feature-gates` flag
for apiserver, controller-manager, and kubelet.
CSI support was introduced as alpha in Kubernetes v1.9 and moved to beta in
Kubernets v1.10.

Once a CSI compatible volume driver is deployed on a Kubernetes cluster, users
may use the `csi` volume type to attach, mount, etc. the volumes exposed by the
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This means users are still supposed to use csi as the volume type when using volumes provisioned by these out-of-tree plugins ...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct.

CSI driver.

The `csi` volume type does not support direct reference from pod and may only be
referenced in a pod via a `PersistentVolumeClaim` object.

The following fields are available to storage administrators to configure a CSI
persistent volume:

- `driver`: A string value that specifies the name of the volume driver to use.
It has to be less than 63 characters and starts with a character. The driver
name can have '`.`', '`-`', '`_`' or digits in it.
- `volumeHandle`: A string value that uniquely identify the volume name returned
from the CSI volume plugin's `CreateVolume` call. The volume handle is then
used in all subsequent calls to the volume driver for referencing the volume.
This value must corespond to the value returned in the `GetPluginInfoResponse`
by the CSI driver as defined in the [CSI spec](https://github.com/container-storage-interface/spec/blob/master/spec.md#getplugininfo).
It is used by Kubernetes to identify which CSI driver to call out to, and by
CSI driver components to identify which PV objects belong to the CSI driver.
- `volumeHandle`: A string value that uniquely identifies the volume. This value
must correspond to the value returned in the `volume.id` field of the
`CreateVolumeResponse` by the CSI driver as defined in the [CSI spec](https://github.com/container-storage-interface/spec/blob/master/spec.md#createvolume).
The value is passed as `volume_id` on all calls to the CSI volume driver when
referencing the volume.
- `readOnly`: An optional boolean value indicating whether the volume is to be
published as read only. Default is false.
"ControllerPublished" (attached) as read only. Default is false. This value is
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

readOnly can also be checked in NodePublishVolume, when a previously created volume get mounted on the node where a workload is scheduled to run.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, the readonly value passed to NodePublishVolume must come from PodSpec.Containers.VolumeMounts.readonly. I checked and it looks like the value is not set at all, so I opened bug kubernetes/kubernetes#61008 to track that.

passed to the CSI driver via the `readonly` field in the
`ControllerPublishVolumeRequest`.
- `fsType`: If the PV's `VolumeMode` is `Filesystem` then this field may be used
to specify the filesystem that should be used to mount the volume. If the
volume has not been formated and formating is supported, this value will be
used to format the volume. If a value is not specified, `ext4` is assumed.
This value is passed to the CSI driver via the `VolumeCapability` field of
`ControllerPublishVolumeRequest`, `NodeStageVolumeRequest`, and
`NodePublishVolumeRequest`.
- `volumeAttributes`: A map of string to string that specifies static properties
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The way I read is like volumeAttributes are only set during ControllerCreateVolume, but in fact external provisioner can also set/use these attributes. Maybe it is worth to mention.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They are set by external provisioner during volume creation, however if a volume is pre-provisioned and cluster admin manually creates PV objects, then the field will have to be set by cluster admin.

of a volume. This map must corespond to the map returned in the
`volume.attributes` field of the `CreateVolumeResponse` by the CSI driver as
defined in the [CSI spec](https://github.com/container-storage-interface/spec/blob/master/spec.md#createvolume).
The map is passed to the CSI driver via the `volume_attributes` field in the
`ControllerPublishVolumeRequest`, `NodeStageVolumeRequest`, and
`NodePublishVolumeRequest`.
- `controllerPublishSecretRef`: A reference to the secret object containing
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ControllerCreateVolume and ControllerDeleteVolume also use this secret.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's not possible since the PV object should not exist before provisioning. CreateVolume and DeleteVolume should be using the SecretRef from the StorageClass.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, my mistake, got confused with secret in StorageClass.

sensitive information to pass to the CSI driver to complete the CSI
`ControllerPublishVolume` and `ControllerUnpublishVolume` calls. This field is
optional, and may be empty if no secret is required. If the secret object
contains more than one secret, all secrets are passed.
- `nodeStageSecretRef`: A reference to the secret object containing
sensitive information to pass to the CSI driver to complete the CSI
`NodeStageVolume` call. This field is optional, and may be empty if no secret
is required. If the secret object contains more than one secret, all secrets
are passed.
- `nodePublishSecretRef`: A reference to the secret object containing
sensitive information to pass to the CSI driver to complete the CSI
`NodePublishVolume` call. This field is optional, and may be empty if no
secret is required. If the secret object contains more than one secret, all
secrets are passed.

### FlexVolume
`FlexVolume` enables users to mount vendor volumes into a pod. The vendor plugin
is implemented using a driver, an executable supporting a list of volume commands
defined by the `FlexVolume` API. Drivers must be installed in a pre-defined
volume plugin path on each node. Pods interact with FlexVolume drivers through the `flexVolume` in-tree plugin.

`FlexVolume` is an out-of-tree plugin interface that has existed in Kubernetes
since version 1.2 (before CSI). It uses an exec-based model to interface with
drivers. FlexVolume driver binaries must be installed in a pre-defined volume
plugin path on each node (and in some cases master).

Pods interact with FlexVolume drivers through the `flexVolume` in-tree plugin.
More details can be found [here](https://github.com/kubernetes/community/blob/master/contributors/devel/flexvolume.md).

## Mount propagation
Expand Down