WIP: Save PV with exponential backoff

[jsafrane]

This PR tries to move PV saving to the main claim workqueue. If a PV fails to be saved to API server, controller remembers the PV and enqueues corresponding claim back to its workqueue. On the next syncClaim, the controller finds saved PV and tries to push it to API server again.

Benefits:

• We don't delete volumes if PV is not saved X times.
• We have nice exponential backoff.

Open issues:

• The exponential backoff is the same as for provisioning.
• The controller stops trying to save the PV after failedProvisionThreshold. More complicated refactoring is needed to get rid of it.
• It changes behavior of the provisioner and ignores CreateProvisionedPVRetryCount() and CreateProvisionedPVInterval() options.

All these could be fixed by a new workqueue for failed PVs that would respect previous behavior. Still it's quite ugly and possibly error prone.

Fixes: kubernetes-csi/external-provisioner#94 kubernetes-csi/external-provisioner#131

cc @wongma7, any ideas?

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jsafrane

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [jsafrane]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jsafrane]

/close
in favor of #33

[k8s-ci-robot]

@jsafrane: Closed this PR.

In response to this:

/close
in favor of #33

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.