asyncpoller fails when SDK PollUntilDone returns a Done poller

[nojnhuh]

/kind bug

What steps did you take and what happened:

This AKS test run failed to delete an AzureManagedMachinePool: https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api-provider-azure/3932/pull-cluster-api-provider-azure-e2e-aks/1699797265842966528

The following appears in the logs several times:

E0907 16:04:55.072224       1 controller.go:324]  "msg"="Reconciler error" "error"="error deleting AzureManagedMachinePool capz-e2e-8c0xmy/poolspot: failed to delete machine pool poolspot: failed to convert poller to future: failed to get resume token: poller is in a terminal state" "AzureManagedMachinePool"={"name":"poolspot","namespace":"capz-e2e-8c0xmy"} "controller"="azuremanagedmachinepool" "controllerGroup"="infrastructure.cluster.x-k8s.io" "controllerKind"="AzureManagedMachinePool" "name"="poolspot" "namespace"="capz-e2e-8c0xmy" "reconcileID"="bde3f505-a842-41f2-b11a-9bd4058eddcc"

Other runs:

• https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api-provider-azure/3988/pull-cluster-api-provider-azure-e2e-aks/1701703934978035712
• https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api-provider-azure/3973/pull-cluster-api-provider-azure-e2e-aks/1702404407368355840
• https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api-provider-azure/4007/pull-cluster-api-provider-azure-e2e-aks/1703839941588422656

What did you expect to happen:

Anything else you would like to add:

It looks like the call to DeleteAsync in asyncpoller.go is returning a non-nil poller where poller.Done() == true. There are a few things I noticed that might help in coming up with a fix, but it's not obvious to me the best route to take:

• The err returned by the call to DeleteAsync in asyncpoller.go is not checked when poller != nil.
• The poller returned in the same call is not checked if it's .Done() before ultimately it's .ResumeToken() method is called, which fails when the poller is done. This directly correlates to the above error message form the CAPZ logs.
• If the poller is Done, might we find an error in its .Result()?

The same issue likely affects the CreateOrUpdate flow as well.

Environment:

• cluster-api-provider-azure version:
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):

fyi @mboersma

[CecileRobertMichon]

/milestone v1.11

[nojnhuh]

/assign