Increase clusterclient timeout and retry intervals

[karan]

What this PR does / why we need it: Title says it all. Some vsphere images can take a while to spin up. Client side timeout should be much higher to protect against those cases.

Release note:

Increase clusterclient timeout and retry intervals

/kind clusterctl

@kubernetes/kube-deploy-reviewers

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: karan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [karan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[karan]

/assign @krousey

[ingvagabund]

Some vsphere images can take a while to spin up.

Is the need for the increased timeout cloud provider specific? Is there a cloud provider for which we can with high confidence say if a resource is not ready in 5 minutes, it will never be ready?

Maybe instead of increasing the timeout (two times or three times) what about to introduce retries? E.g. if I retry TimeoutResourceReady 2 times and still nothing happens -> fail. For me as a user it is better to see "Timeouted waiting for a resources to be ready after %v, trying again..." every 5 minutes instead of waiting 15 minutes to get a "Timeouted ..." error message.

EDIT: ok, I checked the behaviour of the util.PollImmediate. With the PR merged one will get "Waiting for Cluster v1alpha resources to become available..." printed every 10s instead of 5s. Please, ignore my comment :).

[karan]

The timeout increase is mainly to handle the tail-end of cases. For example one test I'm running would fail without increasing this timeout. The retries we have already make so you don't have to wait for 15 minutes if the machine came up sooner.

[karan]

/uncc kris-nova k4leung4

[dims]

/lgtm