workaround ASO swallowing updates into ongoing operations

[nojnhuh]

What type of PR is this?
/kind bug

What this PR does / why we need it:

This PR works around a bug in ASO (Azure/azure-service-operator#3451) where updating the spec of an ASO resource while another update is already in progress swallows the requeue request from the new update and the new update will not take effect until after the next periodic ASO resync which could be up to an hour later by default.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes tests on #4069

Special notes for your reviewer:

• cherry-pick candidate

TODOs:

• squashed commits
• includes documentation
• adds unit tests

Release note:

Fixed a bug causing some updates to take effect significantly later than before

[nojnhuh]

/assign @nawazkh

[nojnhuh]

/cherry-pick release-1.11

[k8s-infra-cherrypick-robot]

@nojnhuh: once the present PR merges, I will cherry-pick it on top of release-1.11 in a new PR and assign it to you.

In response to this:

/cherry-pick release-1.11

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (9b844c6) 57.87% compared to head (1a82f8c) 57.87%.
Report is 10 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #4158   +/-   ##
=======================================
  Coverage   57.87%   57.87%           
=======================================
  Files         187      187           
  Lines       19212    19216    +4     
=======================================
+ Hits        11118    11122    +4     
  Misses       7466     7466           
  Partials      628      628           

Files	Coverage Δ
azure/services/aso/aso.go	89.95% <100.00%> (+0.17%)	⬆️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[nawazkh]

/lgtm

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: 26d6214421ceaa6f51d0e2948fa864477aa78857

[nojnhuh]

/hold
I'm running this alongside #4069 and seeing the tests fail further down the line. Want to check and see if it's related to this change.

[nojnhuh]

/retest

[nojnhuh]

/hold I'm running this alongside #4069 and seeing the tests fail further down the line. Want to check and see if it's related to this change.

I'm reasonably confident this failure is unrelated to the rest of these changes. I think it's a weird concurrency thing trying to update managed clusters and agent pools at the same time like we do in the tags test. The real bug is likely in AKS and/or ASO, but I think we can work around that by using the plain tags API like we were doing before instead of using ASO to manage those. I'll update #4069 with that change but I think this change is still valuable.

/hold cancel
/assign @CecileRobertMichon

[CecileRobertMichon]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: CecileRobertMichon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [CecileRobertMichon]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[nojnhuh]

/retest

[k8s-infra-cherrypick-robot]

@nojnhuh: new pull request created: #4174

In response to this:

/cherry-pick release-1.11

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.