Dependency issue between role assignments and AKS

[wasfree]

I'm trying to add a user-managed identity to AKS kubelet but this also requires that the identity of the cluster is also user-assigned. Beside of this an additional role assignment is needed by custom role or "Managed Identity Operator" build-in-role.

The following issue results due to the fact terraform is not aware the dependency of the role assignment and ask creation. So the Process should be:

1. Create Managed identity
2. Create role assignment to clusters managed identity
3. Create AKS

Creation of AKS before role assignment will lead to the following issue:

 Error: creating Managed Kubernetes Cluster "k8stest-test-aks-platform" (Resource Group "k8stest-test-rg-common"): containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="CustomKubeletIdentityMissingPermissionError" Message="The cluster user assigned identity must be given permission to assign kubelet identity /subscriptions/00000000-00000-00000-0000-000000000/resourceGroups/k8stest-test-rg-common/providers/Microsoft.ManagedIdentity/userAssignedIdentities/k8stest-test-msi-aks-platform-system-pool-usermsi. Check access result not allowed for action Microsoft.ManagedIdentity/userAssignedIdentities/assign/action."
│ 
│   with module.azure.module.aks_clusters["platform"].azurerm_kubernetes_cluster.aks,
│   on /home/vscode/.terraform/modules/azure/modules/compute/aks/aks.tf line 47, in resource "azurerm_kubernetes_cluster" "aks":
│   47: resource "azurerm_kubernetes_cluster" "aks" {

Example configuration:

[...]

custom_role_definitions = {
  custom_kubelet_identity_permission = {
    name = "custom_kubelet_identity_permission"
    description = "Permissions for AKS MI to set kublet identity"
    permissions = {
      actions     = ["Microsoft.ManagedIdentity/userAssignedIdentities/assign/action"]
    }
  }
}

role_mapping = {
  custom_role_mapping = {
    aks_clusters = {
      platform = {
        "custom_kubelet_identity_permission" = {
          managed_identities = {
            keys = ["aks_platform_cluster_usermsi"]
          }
        }
      }
    }
  }
}

managed_identities = {
  aks_platform_cluster_usermsi = {
    name               = "aks-platform-cluster-usermsi"
    resource_group_key = "common"
  }
  aks_platform_kubelet_usermsi = {
    name               = "aks-platform-kubelet-usermsi"
    resource_group_key = "common"
  }
}

compute = {
  aks_clusters = {
    sample = {
      name                     = "sample"
      resource_group_key       = "sample"
      node_resource_group_name = "nodepools"

     [...]

      identity = {
        type                 = "UserAssigned"
        managed_identity_key = "aks_platform_cluster_usermsi"
      }

      kubelet_identity = {
        managed_identity_key = "aks_platform_kubelet_usermsi"
      }

     [...]
    }
  }
}