Update README to Kubernetes 1.14

It follows external-attacher example.

README.md

@@ -1,50 +1,86 @@
 [![Build Status](https://travis-ci.org/kubernetes-csi/external-provisioner.svg?branch=master)](https://travis-ci.org/kubernetes-csi/external-provisioner)
 
-# Kubernetes external provisioner that works with CSI volumes
+# CSI provisioner
 
-This is an example external provisioner for Kubernetes which provisions using CSI Volume drivers..  It's under heavy development, so at this time README.md is notes for the developers coding.  Once complete this will change to something user friendly.
+The external-provisioner is a sidecar container that dynamically provisions volumes by calling `ControllerCreateVolume` and `ControlerDeleteVolume` functions of CSI drivers. It is necessary because internal persistent volume controller running in Kubernetes controller-manager does not have any direct interfaces to CSI drivers.
 
-# Build
+## Overview
+The external-provisioner is an external controller that monitors `PersistentVolumeClaim` objects created by user and annotatate by controller-manager and creates/deletes volumes for them. Full design can be found at Kubernetes proposal at [container-storage-interface.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/container-storage-interface.md)
 
-```bash
-make csi-provisioner
+## Compatibility
+
+This information reflects the head of this branch.
+
+| Compatible with CSI Version                                                                | Container Image                        | Min K8s Version |
+| ------------------------------------------------------------------------------------------ | ---------------------------------------| --------------- |
+| [CSI Spec v1.0.0](https://github.com/container-storage-interface/spec/releases/tag/v1.0.0) | quay.io/k8scsi/csi-provisioner:v1.0.1  | 1.13            |
+
+## Usage
+
+It is necessary to create a new service account and give it enough privileges to run the external-provisioner, see `deploy/kubernetes/rbac.yaml`. The provisioner is then deployed as single Deployment as illustrated below:
+
+```sh
+kubectl create deploy/kubernetes/deployment.yaml
 ```
 
-# Test
+The external-provisioner may run in the same pod with other external CSI controllers such as the external-attacher, external-snapshotter and/or external-resizer.
 
-## Start Kubernetes
+Note that the external-provisioner does not scale with more replicas. Only one external-provisioner is elected as leader and running. The others are waiting for the leader to die. They re-elect a new active leader in ~15 seconds after death of the old leader.
 
-Run a local kubernetes cluster built from latest master code
+### Command line options
 
-## Run Storage Provider
+#### Important optional arguments that are highly recommended to be used
+* `--csi-address <path to CSI socket>`: This is the path to the CSI driver socket inside the pod that the external-provisioner container will use to issue CSI operations (`/run/csi/socket` is used by default).
 
-### Use HostPath drivers
+* `--enable-leader-election`: Enables leader election. This is useful when there are multiple replicas of the same external-provisioner running for one CSI driver. Only one of them may be active (=leader). A new leader will be re-elected when current leader dies or becomes unresponsive for ~15 seconds.
 
-Go to drivers and run:
+* `--timeout <duration>`: Timeout of all calls to CSI driver. It should be set to value that accommodates majority of `ControllerCreateVolume` and `ControlerDeleteVolume` calls. See [CSI error and timeout handling](#csi-error-and-timeout-handling) for details. 15 seconds is used by default.
 
-```bash
-bin/hostpathplugin --drivername mydriver  --endpoint unix://tmp/csi.sock --nodeid foobar -v=5
-```
+* `--retry-interval-start <duration>` - Initial retry interval of failed provisioning or deletion. It doubles with each failure, up to `--retry-interval-max` and then it stops increasing. Default value is 1 second. [CSI error and timeout handling](#csi-error-and-timeout-handling) for details.
 
-### Start external provisioner
+* `--retry-interval-max <duration>` - Maximum retry interval of failed provisioning or deletion. See [CSI error and timeout handling](#csi-error-and-timeout-handling) for details.
 
-```bash
-bin/csi-provisioner --kubeconfig /var/run/kubernetes/admin.kubeconfig --alsologtostderr
-```
+* `--worker-threads <num>`: Number of simultaneously running `ControllerCreateVolume` and `ControllerDeleteVolume` operations. Default value is `100`.
 
-### Create Storage class, PVC, and Pod
+#### Other recognized arguments
+* `--provisioning-retry-count <num>`: Sets number of retries of failed provisioning of a single volume. The external-provisioner will give up provisioning after given number of retries. It should be used only with special CSI drivers whose provisioning of volumes is extremely slow or expensive so the external-provisioner does not waste resources. Set to `0` to retry forever (which is also the default value).
 
-```bash
-kubectl create -f examples/sc.yaml
-kubectl create -f example/pvc.yaml
-kubectl create -f example/pod.yaml
-```
+* `--deletion-retry-count <num>`: Sets number of retries of failed deletion of a single volume. The external-provisioner will give up deletion of the volume after given number of retries. It should be used only with special CSI drivers whose deletion of volumes is extremely slow or expensive so the external-provisioner does not waste resources. Set to `0` to retry forever (which is also the default value).
 
-### Delete PVC
+* `--feature-gates <gates>`: Enables or disables alpha (disabled by default) or beta features (enabled by default). Recognized gates:
+  * `AllAlpha`: Matches all alpha gates.
+  * `Topology` (alpha): Enables topology aware dynamic provisioning.
 
-```bash
-kubectl delete pvc example-pvc
-```
+* `--kubeconfig <path>`: Path to Kubernetes client configuration that the external-provisioner uses to connect to Kubernetes API server. When omitted, default token provided by Kubernetes will be used. This option is useful only when the external-provisioner does not run as a Kubernetes pod, e.g. for debugging. Either this or `--master` needs to be set if the external-provisioner is being run out of cluster.
+
+* `--master <url>`: Master URL to build a client config from. When omitted, default token provided by Kubernetes will be used. This option is useful only when the external-provisioner does not run as a Kubernetes pod, e.g. for debugging. Either this or `--kubeconfig` needs to be set if the external-provisioner is being run out of cluster.
+
+* `--volume-name-prefix <prefix>`: Prefix of PersistentVolume names created by the external-provisioner. Default value is "pvc", i.e. created PersistentVolume objects will have name "pvc-<uuid>".
+
+* `--volume-name-uuid-length`: Length of UUID to be added to `--volume-name-prefix`. Defaults behavior is to NOT truncate the UUID.
+
+* `--version`: Prints current external-provisioner version and quits.
+
+* All glog / klog arguments are supported, such as `-v <log level>` or `-alsologtostderr`.
+
+#### Deprecated arguments
+* `--connection-timeout <duration>`: This option was used to limit establishing connection to CSI driver. Currently, the option does not have any effect and the external-provisioner tries to connect to CSI driver socket indefinitely. It is recommended to run ReadinessProbe on the driver to ensure that the driver comes up in reasonable time.
+
+
+### CSI error and timeout handling
+The external-provisioner invokes all gRPC calls to CSI driver with timeout provided by `--timeout` command line argument (15 seconds by default).
+
+Correct timeout value and number of worker threads depends on the storage backend and how quickly it is able to processes `ControllerCreateVolume` and `ControllerDeleteVolume` calls. The value should be set to accommodate majority of them. It is fine if some calls time out - such calls will be re-tried after exponential backoff (starting with 1s by default), however, this backoff will introduce delay when the call times out several times for a single volume.
+
+Frequency of `ControllerCreateVolume` and `ControllerDeleteVolume` retries can be configured by `--retry-interval-start` and `--retry-interval-max` parameters. The external-provisioner starts retries with `retry-interval-start` interval (1s by default) and doubles it with each failure until it reaches `retry-interval-max` (5 minutes by default). The external provisioner stops increasing the retry interval when it reaches `retry-interval-max`, however, it still re-tries provisioning/deletion of a volume. The external-provisioner keeps its own number of provisioning/deletion failures for each volume. Number of retries can be configured by `--provisioning-retry-count` and `--deletion-retry-count`, however, these should be used in extreme case where provisioning/deletion of a volume is very expensive in the underlying storage subsystem and/or cluster admin attention is needed to resume failed provisioning or deletion. By default, the external-provisioner retries forever until it provisions or deletes the volume.
+
+The external-provisioner can invoke up to `--worker-threads` (100 by default) `ControllerCreateVolume` **and** up to `--worker-threads` `ControllerDeleteVolume` calls in parallel, i.e. these two calls are counted separately. The external-provisioner assumes that the storage backend can cope with such high number of parallel requests and that the requests are handled in relatively short time (ideally sub-second). Lower value should be used for storage backends that expect slower processing related to newly created / deleted volumes or can handle lower amount of parallel calls.
+
+Details of error handling of individual CSI calls:
+* `ControllerCreateVolume`: The call might have timed out just before the driver provisioned a volume and was sending a response. From that reason, timeouts from `ControllerCreateVolume` is considered as "*volume may be provisioned*" or "*volume is being provisioned in the background*." The external-provisioner will re-try calling `ControllerCreateVolume` after exponential backoff until it gets either successful response or final (non-timeout) error that the volume cannot be created.
+* `ControllerDeleteVolume`: This is similar to `ControllerCreateVolume`, The external-provisioner will re-try calling `ControllerDeleteVolume` with exponential backoff after timeout until it gets either successful response or a final error that the volume cannot be deleted.
+* `Probe`: The external-provisioner re-tries calling Probe until the driver reports it's ready. It re-tries also when it receives timeout from `Probe` call. The external-provisioner has no limit of retries. It is expected that ReadinessProbe on the driver container will catch case when the driver takes too long time to get ready.
+* `GetPluginInfo`, `GetPluginCapabilitiesRequest`, `ControllerGetCapabilities`: The external-provisioner expects that these calls are quick and does not retry them on any error, including timeout. Instead, it assumes that the driver is faulty and exits. Note that Kubernetes will likely start a new provisioner container and it will start with `Probe` call.
 
 ## Community, discussion, contribution, and support