nmi failed to refresh token, error: adal: Failed to execute the refresh request.

[nader-ziada]

Have you

• Read Troubleshooting Guide
• Read Known Issues
• Searched on GitHub issues

Describe the bug
When using nmi in managed mode with cluster-api-provider-azure running on Azure, I get the following error when trying to read/create resources from Azure

I0519 14:15:27.829897       1 managed.go:52] clientID in request: 4b8d##### REDACTED #####9086, capz-system/capz-controller-manager-874c54fb4-nslgz has been matched with azure identity capz-system/capz-e2e-c951p2-capz-e2e-c14w7g-cluster-identity
I0519 14:15:27.830129       1 managed.go:89] matched identityType:1 adendpoint: tenantid:b39138ca-**** auxiliaryTenantIDs:[] clientid:4b8d##### REDACTED #####9086 resource:https://management.azure.com/
E0519 14:15:57.833861       1 server.go:378] failed to get service principal token for pod: capz-system/capz-controller-manager-874c54fb4-nslgz, error: failed to refresh token, error: adal: Failed to execute the refresh request. Error = 'Post "https://login.microsoftonline.com/b39138ca-3cee-4b4a-a4d6-cd83d9dd62f0/oauth2/token?api-version=1.0": dial tcp: i/o timeout'

Steps To Reproduce
1- Create cluster-api-provider-azure management cluster on Azure
2- Use this management cluster to create a private cluster in the same resource group and vnet
3- Creation of the management cluster fails and when looking at the cluster-api-provider-azure logs, I see the following

E0519 14:12:43.959382       1 controller.go:302] controller-runtime/manager/controller/azurecluster "msg"="Reconciler error" "error"="failed to reconcile cluster services: failed to get availability zones: failed to get zones for location eastus: failed to refresh resource sku cache: could not list resource skus: azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.azure.com/subscriptions/****/providers/Microsoft.Compute/skus?%24filter=location+eq+%27eastus%27&api-version=2019-04-01: StatusCode=403 -- Original Error: adal: Refresh request failed. Status Code = '403'. Response body: failed to refresh token, error: adal: Failed to execute the refresh request. Error = 'Post \"https://login.microsoftonline.com/b39138ca-3cee-4b4a-a4d6-cd83d9dd62f0/oauth2/token?api-version=1.0\": dial tcp: i/o timeout'\n Endpoint http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&client_id=4b8d33af-0bb7-468a-ad13-72bd5e7c9086&resource=https%3A%2F%2Fmanagement.azure.com%2F" "name"="capz-e2e-c951p2" "namespace"="capz-e2e-c14w7g" "reconciler group"="infrastructure.cluster.x-k8s.io" "reconciler kind"="AzureCluster"
I0519 14:15:27.825974       1 azurecluster_controller.go:187] controllers/AzureCluster "msg"="Reconciling AzureCluster" "azureCluster"="capz-e2e-c951p2" "cluster"="capz-e2e-c951p2" "namespace"="capz-e2e-c14w7g"

4- ssh to the node machine running the nmi pod and was able to execute the following curl command successfully

curl -X GET -H 'Content-Type: application/x-www-form-urlencoded' \
-d 'grant_type=client_credentials&client_id=<client ID>&resource=https://management.azure.com/&client_secret=<client secret>' https://login.microsoftonline.com/<tenant ID>/oauth2/token

Expected behavior
access to azure resources working similar to how the management cluster worked

AAD Pod Identity version
v1.8.0

Kubernetes version

Additional context
this is the PR in cluster-api-porvider-azure when this issues happens:
kubernetes-sigs/cluster-api-provider-azure#1360

[aramase]

Conversation in https://kubernetes.slack.com/archives/CEX9HENG7/p1621362051002500

[nader-ziada]

feel free to close this,
here is the relevant issue in capz tracking fixing the dns issues with calico
kubernetes-sigs/cluster-api-provider-azure#1448

[aramase]

Thank you for the update!

I'll close this one as you're tracking the changes in cluster-api-provider-azure