wait for provisioning state in AKS e2e tests #4255

nojnhuh · 2023-11-11T01:11:51Z

What type of PR is this?
/kind flake

What this PR does / why we need it:

Many of the e2e test specs for managed clusters make some update to a CAPZ resource and then wait for that change to be reflected in Azure. Until now, this wait hasn't taken into account the provisioningState of the resource in Azure, which changes from "Updating" to "Succeeded" when the operation is done. Anecdotally, I tend to observe updates in the Azure API to apply a noticeable amount of time before operations finish. I have a hypothesis that the current behavior is causing the tests to progress too quickly and cause issues like #3955 and #4069 (comment).

This PR inserts checks that the provisioning state of the AKS resources is Succeeded alongside the existing checks that the other relevant API fields have been updated to ensure the cluster is in a successful state and not actively being updated before moving on.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #3955

Special notes for your reviewer:

cherry-pick candidate

TODOs:

squashed commits
includes documentation
adds unit tests

Release note:

NONE

nojnhuh · 2023-11-11T01:12:22Z

test/e2e/aks_upgrade.go

@@ -65,6 +66,7 @@ func AKSUpgradeSpec(ctx context.Context, inputGetter func() AKSUpgradeSpecInput)
 	Eventually(func(g Gomega) {
 		resp, err := managedClustersClient.Get(ctx, infraControlPlane.Spec.ResourceGroupName, infraControlPlane.Name, nil)
 		g.Expect(err).NotTo(HaveOccurred())
+		g.Expect(resp.Properties.ProvisioningState).To(Equal(ptr.To("Succeeded")))


FYI @willie-yao I think this overlaps with a change you added to #4155.

Looks like my change is accomplishing the same thing although the provisioning state is coming from the managed cluster itself rather than the response properties. Is the behavior any different?

codecov · 2023-11-11T02:14:00Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (8a09a42) 58.16% compared to head (201101f) 58.16%.
Report is 16 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #4255   +/-   ##
=======================================
  Coverage   58.16%   58.16%           
=======================================
  Files         187      187           
  Lines       19351    19351           
=======================================
  Hits        11256    11256           
  Misses       7457     7457           
  Partials      638      638

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

nojnhuh · 2023-11-12T03:23:27Z

/test pull-cluster-api-provider-azure-conformance-custom-builds
/test pull-cluster-api-provider-azure-e2e-aks

nojnhuh · 2023-11-12T16:33:47Z

/test pull-cluster-api-provider-azure-e2e-aks

Flake hunting 🔎

nojnhuh · 2023-11-12T19:29:19Z

/test pull-cluster-api-provider-azure-e2e-aks

nojnhuh · 2023-11-12T21:15:06Z

/test pull-cluster-api-provider-azure-e2e-aks

CecileRobertMichon

/lgtm

k8s-ci-robot · 2023-11-13T21:58:16Z

LGTM label has been added.

Git tree hash: 59448553444bc6247ef8041b3ef87fcc40eab376

willie-yao · 2023-11-14T01:28:40Z

/lgtm

mboersma

/lgtm
/approve

k8s-ci-robot · 2023-11-14T16:58:23Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mboersma

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [mboersma]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

mboersma · 2023-11-14T17:01:12Z

/cherry-pick release-1.11
/cherry-pick release-1.10

k8s-infra-cherrypick-robot · 2023-11-14T17:01:15Z

@mboersma: once the present PR merges, I will cherry-pick it on top of release-1.11 in a new PR and assign it to you.

In response to this:

/cherry-pick release-1.11
/cherry-pick release-1.10

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-infra-cherrypick-robot · 2023-11-14T18:57:17Z

@mboersma: #4255 failed to apply on top of branch "release-1.11":

Applying: wait for provisioning state in AKS e2e tests
Using index info to reconstruct a base tree...
M	test/e2e/aks_autoscaler.go
M	test/e2e/aks_azure_cluster_autoscaler.go
M	test/e2e/aks_node_labels.go
M	test/e2e/aks_node_taints.go
M	test/e2e/aks_tags.go
M	test/e2e/aks_upgrade.go
Falling back to patching base and 3-way merge...
Auto-merging test/e2e/aks_upgrade.go
CONFLICT (content): Merge conflict in test/e2e/aks_upgrade.go
Auto-merging test/e2e/aks_tags.go
CONFLICT (content): Merge conflict in test/e2e/aks_tags.go
Auto-merging test/e2e/aks_node_taints.go
CONFLICT (content): Merge conflict in test/e2e/aks_node_taints.go
Auto-merging test/e2e/aks_node_labels.go
Auto-merging test/e2e/aks_azure_cluster_autoscaler.go
Auto-merging test/e2e/aks_autoscaler.go
CONFLICT (content): Merge conflict in test/e2e/aks_autoscaler.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 wait for provisioning state in AKS e2e tests
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick release-1.11
/cherry-pick release-1.10

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-infra-cherrypick-robot · 2023-11-14T18:57:51Z

@mboersma: #4255 failed to apply on top of branch "release-1.10":

Applying: wait for provisioning state in AKS e2e tests
Using index info to reconstruct a base tree...
M	test/e2e/aks_autoscaler.go
M	test/e2e/aks_azure_cluster_autoscaler.go
M	test/e2e/aks_node_labels.go
M	test/e2e/aks_node_taints.go
M	test/e2e/aks_tags.go
M	test/e2e/aks_upgrade.go
Falling back to patching base and 3-way merge...
Auto-merging test/e2e/aks_upgrade.go
CONFLICT (content): Merge conflict in test/e2e/aks_upgrade.go
Auto-merging test/e2e/aks_tags.go
CONFLICT (content): Merge conflict in test/e2e/aks_tags.go
Auto-merging test/e2e/aks_node_taints.go
CONFLICT (content): Merge conflict in test/e2e/aks_node_taints.go
Auto-merging test/e2e/aks_node_labels.go
Auto-merging test/e2e/aks_azure_cluster_autoscaler.go
CONFLICT (content): Merge conflict in test/e2e/aks_azure_cluster_autoscaler.go
Auto-merging test/e2e/aks_autoscaler.go
CONFLICT (content): Merge conflict in test/e2e/aks_autoscaler.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 wait for provisioning state in AKS e2e tests
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick release-1.11
/cherry-pick release-1.10

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

wait for provisioning state in AKS e2e tests

201101f

nojnhuh commented Nov 11, 2023

View reviewed changes

k8s-ci-robot requested review from jackfrancis and willie-yao November 11, 2023 02:07

CecileRobertMichon reviewed Nov 13, 2023

View reviewed changes

k8s-ci-robot assigned CecileRobertMichon Nov 13, 2023

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 13, 2023

k8s-ci-robot assigned willie-yao Nov 14, 2023

mboersma approved these changes Nov 14, 2023

View reviewed changes

k8s-ci-robot assigned mboersma Nov 14, 2023

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 14, 2023

k8s-ci-robot merged commit fc261f0 into kubernetes-sigs:main Nov 14, 2023
28 checks passed

k8s-ci-robot added this to the v1.12 milestone Nov 14, 2023

mboersma mentioned this pull request Nov 14, 2023

[release-1.11] wait for provisioning state in AKS e2e tests #4271

Merged

3 tasks

mboersma mentioned this pull request Nov 14, 2023

[release-1.10] wait for provisioning state in AKS e2e tests #4272

Merged

3 tasks

nojnhuh deleted the aks-e2e-provstate branch November 22, 2023 02:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

wait for provisioning state in AKS e2e tests #4255

wait for provisioning state in AKS e2e tests #4255

nojnhuh commented Nov 11, 2023

nojnhuh Nov 11, 2023

willie-yao Nov 14, 2023

codecov bot commented Nov 11, 2023 •

edited

Loading

nojnhuh commented Nov 12, 2023

nojnhuh commented Nov 12, 2023

nojnhuh commented Nov 12, 2023

nojnhuh commented Nov 12, 2023

CecileRobertMichon left a comment

k8s-ci-robot commented Nov 13, 2023

willie-yao commented Nov 14, 2023

mboersma left a comment

k8s-ci-robot commented Nov 14, 2023

mboersma commented Nov 14, 2023

k8s-infra-cherrypick-robot commented Nov 14, 2023

k8s-infra-cherrypick-robot commented Nov 14, 2023

k8s-infra-cherrypick-robot commented Nov 14, 2023

wait for provisioning state in AKS e2e tests #4255

wait for provisioning state in AKS e2e tests #4255

Conversation

nojnhuh commented Nov 11, 2023

nojnhuh Nov 11, 2023

Choose a reason for hiding this comment

willie-yao Nov 14, 2023

Choose a reason for hiding this comment

codecov bot commented Nov 11, 2023 • edited Loading

Codecov Report

nojnhuh commented Nov 12, 2023

nojnhuh commented Nov 12, 2023

nojnhuh commented Nov 12, 2023

nojnhuh commented Nov 12, 2023

CecileRobertMichon left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Nov 13, 2023

willie-yao commented Nov 14, 2023

mboersma left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Nov 14, 2023

mboersma commented Nov 14, 2023

k8s-infra-cherrypick-robot commented Nov 14, 2023

k8s-infra-cherrypick-robot commented Nov 14, 2023

k8s-infra-cherrypick-robot commented Nov 14, 2023

codecov bot commented Nov 11, 2023 •

edited

Loading