Add proposal for Async Azure Resource Creation and Deletion

[CecileRobertMichon]

What type of PR is this?

What this PR does / why we need it: This is a proposal to formalize some of the work that was already done in #1067, and to extend it to other Azure PUT/DELETE operations in the codebase. Working on a POC in parallel but opening this up to get some feedback asynchronously (😉 ).

Credit goes to @devigned for kickstarting this effort with AzureMachinePools, and to @shysank for reflections in #1181.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:

Please confirm that if this PR changes any image versions, then that's the sole change this PR makes.

TODOs:

• squashed commits
• includes documentation
• adds unit tests

Release note:

NONE

[CecileRobertMichon]

/cc @devigned @nader-ziada @shysank @alexeldeib

[devigned]

Just have a couple comments and a small nit. I really like the tie between long running operation state and conditions, though it might be useful to provide a little more perscriptive guidance on whether or not a long running operation would tie to a given condition. For example, would a condition state be tied 1 to 1 with a long running operation and would that state change with the state of a long running operation.

Great work! This has the potential to make a significant improvement in the reactivity of the system and the experience of a user interacting with the controller.

[CecileRobertMichon]

though it might be useful to provide a little more perscriptive guidance on whether or not a long running operation would tie to a given condition. For example, would a condition state be tied 1 to 1 with a long running operation and would that state change with the state of a long running operation.

Will add. My current thinking is that conditions should not be tied 1 to 1 with a long running operation, although we might have conditions that summarize the state of 1 or more resources created/deleted via long running operations. So long-running operations conditions might be a subset of all conditions.

I'll think about this further and come up with an initial list of added conditions.

[CecileRobertMichon]

I'm expecting the numbers to become more aggressive with some testing, maybe closer to 2s per call / 15 seconds overall

[devigned]

I like the updates. This is looking really solid. I'm added a couple comments and questions.

[shysank]

Thanks @CecileRobertMichon for formalizing the ideas. Looks good overall with a couple of nits/clarifications.

[CecileRobertMichon]

There haven't been any comments on this for a while, I'd like to start a lazy consensus for end of next week if no other comments get added by then.

/hold for lazy consensus

/cc @jackfrancis @alexeldeib @mboersma in case you haven't had a chance to review this yet

[devigned]

lgtm. Great work!

[nader-ziada]

a little bit late, but also lgtm 🎉

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: devigned

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [devigned]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment