[donotmerge] debug aks e2e

[alexeldeib]

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:

Please confirm that if this PR changes any image versions, then that's the sole change this PR makes.

TODOs:

• squashed commits
• includes documentation
• adds unit tests

Release note:

[k8s-ci-robot]

@alexeldeib: Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[alexeldeib]

In that one the cluster reached provisioned but both MPs seem to have gotten stuck. Added some debu logs and trying again.

What’s weird is they both seem to have been created in azure — toward the end you can see the logs from the agentpools service where we diff the current/desired agentpool, and both pools are up to date. But we don’t see to reach the end of the reconcile, and yet I don’t see any errors. Could be failing to call VMSS but somehow not logging..

[alexeldeib]

/hold

just for debug

[alexeldeib]

hmm. in that run, cluster failed to provision, but AMCP is fully populated/ready. checking the timestamps...

https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api-provider-azure/1488/pull-cluster-api-provider-azure-e2e-exp/1411107169369067520/artifacts/clusters/bootstrap/resources/capz-e2e-ni5r7m/AzureManagedControlPlane/capz-e2e-ni5r7m.yaml

looks like AMCP was "reconciling kubeconfig" withing 5min

[alexeldeib]

and then we randomly start failing on identity not present, AFTER amcp successfully reconciles?

failed to create scope: failed to retrieve AzureClusterIdentity external object \"capz-e2e-ni5r7m\"/\"multi-tenancy-identity\": AzureClusterIdentity.infrastructure.cluster.x-k8s.io \"multi-tenancy-identity\" not found\nsigs.k8s.io/cluster-api-provider-azure/azure/scope.NewManagedControlPlaneCredentialsProvider\n\t/workspace/azure/scope/identity.go:118\nsigs.k8s.io/cluster-api-provider-azure/azure/scope.NewManagedControlPlaneScope\n\t/workspace/azure/scope/managedcontrolplane.go:69\nsigs.k8s.io/cluster-api-provider-azure/exp/controllers.(*AzureManagedControlPlaneReconciler).Reconcile\n\t/workspace/exp/controllers/azuremanagedcontrolplane_controller.go:157\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:298\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:214\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371\nfailed to init credentials provider\nsigs.k8s.io/cluster-api-provider-azure/azure/scope.NewManagedControlPlaneScope\n\t/workspace/azure/scope/managedcontrolplane.go:71\nsigs.k8s.io/cluster-api-provider-azure/exp/controllers.(*AzureManagedControlPlaneReconciler).Reconcile\n\t/workspace/exp/controllers/azuremanagedcontrolplane_controller.go:157\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:298\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:214

interestingly, I do not see the AzureClusterIdentity in the output artifacts: https://gcsweb.k8s.io/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api-provider-azure/1488/pull-cluster-api-provider-azure-e2e-exp/1411107169369067520/artifacts/clusters/bootstrap/resources/capz-e2e-ni5r7m/

[alexeldeib]

/test pull-cluster-api-provider-azure-e2e-exp

getting some more data

[alexeldeib]

so for the failures waiting for a control plane machine where it looks "hung", appears we get stuck looping here:

cluster-api-provider-azure/exp/controllers/azuremanagedmachinepool_reconciler.go

Line 149 in 03ba941

scope.V(2).Info("No match")

now to find why

update: from a successful run, this is the period during which no vmss are created, so we're looping trying to find one. I don't get why that makes sense though, if we are able to run pods, the VMs are clearly there...something is weird with the amounts of time between different events

[alexeldeib]

/test pull-cluster-api-provider-azure-e2e-exp

[alexeldeib]

in the last run, it looks like pods did come up, but we get no results listing scalesets and eventually time out :/

https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api-provider-azure/1488/pull-cluster-api-provider-azure-e2e-exp/1411150593765937152/artifacts/clusters/bootstrap/controllers/capz-controller-manager/capz-controller-manager-69d5ccc44d-2mgcg/manager.log

probably want to dump kubectl get nodes / equivalent, also adding nmi logs might be good. I don't understand why we get stuck in that loop for so long

[alexeldeib]

So it seems like we're really stuck in that loop, but the nodes from both pools already joined. So i'm trying to understand why we keep listing VMSS and getting zero results from Azure.

[alexeldeib]

/test pull-cluster-api-provider-azure-e2e-exp

[alexeldeib]

/test pull-cluster-api-provider-azure-e2e-exp

[alexeldeib]

/test pull-cluster-api-provider-azure-e2e-exp

[alexeldeib]

/test pull-cluster-api-provider-azure-e2e

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please ask for approval from alexeldeib after the PR has been reviewed.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

@alexeldeib: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[CecileRobertMichon]

/close

[k8s-ci-robot]

@CecileRobertMichon: Closed this PR.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.