Multi-tenancy within one manager instance using Azure User Assigned Identities

[nader-ziada]

/kind feature

Describe the solution you'd like

As described by the multi-tenancy design proposal https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/master/docs/proposals/20200720-single-controller-multitenancy.md (updated in #1089),
the manager instance should also support Azure User Assigned Identities

The current implementation (from pr #977) only added support for using Service Principal using aad-pod-identity

[shysank]

/assign

[nader-ziada]

@shysank there might be some changes need in the aad-pod-library before we can work on this, better to discuss with @devigned

[devigned]

As @nader-ziada was mentioning and if I recall correctly, AAD Pod Identity is configured to work within a resource group when assigning managed identities and is configured through https://github.com/Azure/aad-pod-identity/blob/fb1bdba48ab5c1fa0be64382b6a74acdbcb41089/pkg/config/azureconfig.go#L10. After talking with @aramase, the reasoning behind this was mostly due to performance concerns of having to look up Azure resources outside of the resource group, which can be an expensive operation. I did not get the feeling scoping by resource group was functionally necessary if a resource lookup could be done in an efficient manner.

Since CAPZ resources can be in any resource group a user desires and AAD Pod Identity is intended to be run as a singleton controller within a cluster, the constraint on resource group scoping of the Managed Identity Controller must be removed to unblock Managed Identities for CAPZ.

Happy to setup some time to discuss in adhoc meeting, in thread, or during our next office hours.

[aramase]

The resourcegroup that's configured with MIC (MIC gets that value from azure.json) is for the compute resources only. MIC uses the resource group to list the VMSS instances and assign identity only to the VMSS in the cluster resource group. However the identities can be in any resource group.

@devigned The management cluster is the one that has the identities, right?

[devigned]

The management cluster is the one that has the identities, right?

Yes.

So, are you saying that Azure compute resources hosting the management cluster would not span multiple resource groups, so having resourcegroup in the azure.json makes sense?

[aramase]

So, are you saying that Azure compute resources hosting the management cluster would not span multiple resource groups, so having resourcegroup in the azure.json makes sense?

Yeah, that's right. Users can bring their own managed identities which belong to any resource group within the tenant. After they assign the required role assignments pod identity will be able to assign the identities to the underlying VMSS/VM.

[devigned]

@nader-ziada or @shysank wdyt?

[nader-ziada]

I think one concern was that we are currently using the managed mode with only nmi because at the time of deployment of the mic we would not be able to know which resource group to pass it since that will depend on the workload cluster. This is because we need to support the use case of having a management cluster outside of azure, for example on kind

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[nader-ziada]

/remove-lifecycle stale
/lifecycle frozen

[devigned]

/assign

[shrinedogg]

Experiencing this exact issue trying to create a cluster using this provider...

1. Attempt to create cluster via CAPZ Cluster API
2. Attempt fails with error in CAPZ-Controller log AzureClusterIdentity is not of type Service Principal using the provided yaml...

---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: AzureManagedControlPlane
metadata:
  name: ${CLUSTER_NAME}
  namespace: default
spec:
  additionalTags:
    jobName: ${JOB_NAME:=""}
    creationTimestamp: ${TIMESTAMP:=""}
    buildProvenance: ${BUILD_PROVENANCE:=""}
  identityRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: AzureClusterIdentity
    name: "${CLUSTER_IDENTITY_NAME}"
    namespace: default
  location: eastus
  resourceGroupName: ${CLUSTER_NAME}
  sshPublicKey: ${AZURE_SSH_PUBLIC_KEY_B64:=""}
  subscriptionID: ${AZURE_SUBSCRIPTION_ID}
  version: ${KUBERNETES_VERSION}
  networkPolicy: azure
  networkPlugin: azure
  sku:
    tier: "Paid"
  loadBalancerProfile:
    managedOutboundIPs: 6
    idleTimeoutInMinutes: 9
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: AzureClusterIdentity
metadata:
  name: ${CLUSTER_IDENTITY_NAME}
  namespace: default
spec:
  type: UserAssignedMSI
  resourceID: ${AZURE_CLUSTER_IDENTITY_RESOURCE_ID}
  clientID: ${AZURE_CLIENT_ID}
  tenantID: ${AZURE_TENANT_ID}  
  allowedNamespaces: {}

We are seeking assistance!

[CecileRobertMichon]

/assign