test deleting all tags, node labels, and node taints in AKS e2e

[nojnhuh]

What type of PR is this?
/kind cleanup

What this PR does / why we need it:

This PR adds tests that remove all defined tags, node labels, and node taints on AKS resources. Deleting all tags had been previously tested but was removed in #4069 due to an ASO limitation at the time that has since been lifted. Deleting all node labels and taints was first made possible in #4122 and has never been tested in e2e before.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #4266

Special notes for your reviewer:

• cherry-pick candidate

TODOs:

• squashed commits
• includes documentation
• adds unit tests

Release note:

NONE

[nojnhuh]

/cherry-pick release-1.12

[k8s-infra-cherrypick-robot]

@nojnhuh: once the present PR merges, I will cherry-pick it on top of release-1.12 in a new PR and assign it to you.

In response to this:

/cherry-pick release-1.12

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (63df74f) 59.73% compared to head (a8a1f0f) 59.73%.
Report is 34 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #4295   +/-   ##
=======================================
  Coverage   59.73%   59.73%           
=======================================
  Files         192      192           
  Lines       19303    19303           
=======================================
  Hits        11531    11531           
  Misses       7142     7142           
  Partials      630      630           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[CecileRobertMichon]

@nojnhuh fyi we found an issue with @jackfrancis last week where the test deletes the creationTimestamp tag on the AzureManagedControlPlane which is synced to the MC_ rg by AKS, so the tag gets deleted from the RG. This causes the test sub RG cleanup job to delete the node resource group mid-test when it happens to be running at the same time, causing flakes (rg cleanup uses tags to know if a rg is old enough to delete and deleted everything without a tag https://github.com/Azure/rg-cleanup).

So we might not be able to remove all tags as-is, we might need to preserve some of the "essential" tags, at least on the RGs.

[nojnhuh]

@CecileRobertMichon Ah, that makes sense and sounds like the real root cause of #3955. Maybe we can skip the "delete all tags" step for the AzureManagedControlPlane and keep it for the AzureManagedMachinePools. I think the mechanism in ASO is similar enough that that would still give us enough confidence that that still works for ASO ManagedClusters if we only test it for ManagedClustersAgentPools.

[CecileRobertMichon]

/lgtm
/assign @jackfrancis

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: b300bcbf62180ee1d3294d26399e64332aa6ec7c

[nojnhuh]

@jackfrancis Could you PTAL?

[jackfrancis]

/lgtm

(added one clarifying question, I don't expect it will result in any changes)

[jackfrancis]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jackfrancis

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [jackfrancis]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-infra-cherrypick-robot]

@nojnhuh: new pull request created: #4319

In response to this:

/cherry-pick release-1.12

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.