Implement UserAssignedMSI auth for ASO

[nojnhuh]

Follow-up from #3526

CAPZ currently allows users to create an AzureClusterIdentity with spec.type=UserAssignedMSI driven by AAD Pod Identity and does not require the manual setup steps of Workload Identity. The same credentials in this case should be propagated to CAPZ's ASO.

If AAD Pod Identity support is removed from CAPZ before #3527 closes, then this issue would no longer need to be implemented.

see also:
https://azure.github.io/azure-service-operator/guide/authentication/credential-format/#deprecated-managed-identity-aad-pod-identity

Dependencies

Preview Give feedback

1. Allow multi-tenant environments to authenticate with different Managed Identities through AAD Pod Identity Azure/azure-service-operator#3167
   capz high-priority +2
   

[nojnhuh]

To recap yesterday's discussion with @CecileRobertMichon to help me understand this:

• CAPZ uses AAD Pod ID currently to enable the ServicePrincipal and ServicePrincipalCertificate AzureClusterIdentity types.
• ASO's documented instructions for Service Principal do not involve AAD Pod ID, though configuring a Service Principal with AAD Pod ID does work.
• If we have the option to either configure or not configure ASO to use AAD Pod ID, we'll opt not to use AAD Pod ID, though if some users report issues because they rely on auth going through AAD Pod Identity, we can add that capability as needed.

To follow up with some of the open questions:

Does the NMI pod intercept IMDS requests from ASO when AAD Pod ID is not configured explicitly?

No AFAICT. I don't see any new logs in the NMI pod when creating an ASO resource with a Service Principal like this: https://azure.github.io/azure-service-operator/guide/authentication/credential-format/#service-principal-using-a-client-secret

Does ASO support managed identities either with or without AAD Pod Identity (excluding Workload ID)?

Yes with AAD Pod ID, not without. I can get managed identity working with these instructions: https://azure.github.io/azure-service-operator/guide/authentication/credential-format/#deprecated-managed-identity-aad-pod-identity. Those steps modify the global aso-controller-settings secret though, and if I try to use the managed identity client ID in a secret referenced by a resource's credential-from annotation ASO hits an error that indicates it's trying to use Workload ID, so I'm not sure ASO supports multi-tenant managed identity setups with AAD Pod ID. Still looking into that.

[nojnhuh]

I re-scoped this issue to reflect #3698 no longer implementing UserAssignedMSI AzureClusterIdentities due to the newly-discovered ASO gap. More details in the issue description.

/retitle Implement UserAssignedMSI auth for ASO

[nojnhuh]

/assign