Switch to exponential backoff while creating/deletion machines

[hardikdr]

What would you like to be added:
On failure of machine creation or deletion requests, MCM constantly tries to create or delete the machine-objects. This could cause a heavy load on control-cluster's API-server, and exhaust the API rate-limits of cloud-provider. We should exponentially back-off on the failure of requests.

Why is this needed:

[prashanth26]

/assign @hardikdr @prashanth26
/priority blocker

[hardikdr]

/priority normal
We implemented the constant backoff here #525. We should consider looking at a more sophisticated exponential backoff mechanism, a proposal would be nice.
I mainly see 2 options,

1. Backoff at the queue. An attempt to machine-set queue: Back off failed Machine operations #510
2. Backoff inside the reconcile function.
   • Maybe something similar to https://github.com/gardener/autoscaler/tree/machine-controller-manager-provider/cluster-autoscaler/utils/backoff .

cc @zuzzas

[zuzzas]

Thanks to #525 we can now attach a RateLimitingInterface to the queue, and throttle Machines in CrashLoopBackoff.

1. I'd take the backoff_manager concept from here.
2. Create a throttling-by-CrashLoopBackoff function here.
3. And attach the resulting RateLimitingInterface to the queue here.

Then, there's a matter of replacing Adds with AddRateLimiteds to ensure that our new RateLimiter is being triggered.

[prashanth26]

/title Switch to exponential backoff while creating/deletion machines