chore: Add retryable error to cloud provider

[jigisha620]

Fixes #947

Description
The termination controller calls cloudProvider.Delete(nodeClaim) which tries to terminate the underlying instance. It will remove the finalizer once the cloudProvider.Delete() call succeeds. However, the current implementation in AWS does not wait until the instance state is actually terminated and proceeds to remove the finalizer once the ec2.TerminateInstance() call has succeeded.

The change in this PR adds a retryable error to cloudprovider which will allow Karpenter to parse instance state that has been successfully called terminate, but wasn't fully terminated. This effectively re-enqueues the node into the reconciliation loop of the termination controller.

How was this change tested?

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[k8s-ci-robot]

Hi @jigisha620. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[jonathan-innis]

/lgtm
/approve

[Bryce-Soghigian]

We are also interested in a Retriable cloud provider error.

We were discussing should we drive retries for creating vms through the azure sdk or through the controller? There are some cases where we see 429 throttling errors, and being able to set a retry after on the nodeclaim would be very interesting.

Likely we will be doing some work at some point looking at the retry patterns we are using and comparing single provisioning vs adding a batch style pattern that aws provider does.

[njtran]

/lgtm
/approve

[jonathan-innis]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jigisha620, jonathan-innis, njtran

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [jonathan-innis,njtran]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coveralls]

Pull Request Test Coverage Report for Build 8562595230

Details

• 9 of 17 (52.94%) changed or added relevant lines in 2 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage increased (+0.02%) to 78.697%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
pkg/cloudprovider/types.go	8	16	50.0%

Totals
Change from base Build 8560649553:	0.02%
Covered Lines:	8286
Relevant Lines:	10529

---

💛 - Coveralls