Reuse Azure Service Operator for IaC

[lieberlois]

/kind feature

Describe the solution you'd like
I want to be able to reuse the Azure Service Operator deployment that is part of the CAPI Provider Azure for my Azure infrastructure (e.g. Storage Accounts etc.) to prevent the necessity of deploying another solution like Crossplane or even a second ASO. The current implementation appears to deploy the ASO but scoped to a limited set of custom resources (only those that re.

Anything else you would like to add:

Environment:

• cluster-api-provider-azure version: v1.11.2
• Kubernetes version: 1.25
• OS (e.g. from /etc/os-release): Linux

[CecileRobertMichon]

cc @dtzar @nojnhuh @matthchr

[nojnhuh]

@lieberlois Even though the installation of ASO that comes with CAPZ only comes with a limited number of CRDs currently, it is possible to configure ASO to install and manage more CRDs. Steps to do that are described here: https://capz.sigs.k8s.io/topics/aso#using-aso-for-non-capz-resources. Fair warning that I've only done some light manual testing of that path, so please report any issues if you do try that out.

The inverse of that is described here, where you install ASO however you'd like then install CAPZ later: #3900. I haven't tried that out but I think it's possible.

[dtzar]

We discussed this topic today on the CAPZ community call, more specifically:

1. This is now enabling a feature-set (ASO) completely outside of what CAPZ itself is designed to do - should it be included inside clusterctl and/or CAPI Operator or a separate tool?
2. If we are including it inside the purview of CAPZ somehow, would it make sense to only have it enabled via CAPI Operator helm chart and not clusterctl?

IMO - I'd like to see this happen via CAPI Operator helm chart switches.

[illrill]

We are reusing the CAPZ-bundled ASO to create other Azure resources following https://capz.sigs.k8s.io/topics/aso#using-aso-for-non-capz-resources. This is how we set up idempotent CI/CD to handle initialisation of the management cluster and installation of additional ASO CRDs using the manual CRD installation approach.

# Install CAPI & CAPZ
export EXP_MACHINE_POOL=true
brew install clusterctl
clusterctl version
clusterctl init --infrastructure=azure

# Configure this variable with additional ASO CRD groups we should install (https://azure.github.io/azure-service-operator/reference/)
ASO_CRD_GROUPS=(
    "network.azure.com"
    "storage.azure.com"
)

# Find what version the CAPZ-provided ASO is
ASO_VERSION=$(kubectl -n capz-system get deployment azureserviceoperator-controller-manager -ojson | jq -r '.metadata.labels."app.kubernetes.io/version"')
[[ -z $ASO_VERSION ]] && { echo "Could not determine which version of ASO is installed by CAPZ. ASO_VERSION: ${ASO_VERSION}"; exit 1; }

# Download all ASO CRDs
ASO_CRDS_URL="https://github.com/Azure/azure-service-operator/releases/download/${ASO_VERSION}/azureserviceoperator_customresourcedefinitions_${ASO_VERSION}.yaml"
curl -sSL ${ASO_CRDS_URL} -o aso-crds.yaml

# Replace some fields to make the CAPZ-provided ASO able to use them (https://capz.sigs.k8s.io/topics/aso#using-aso-for-non-capz-resources)
yq -i '.metadata.annotations."cert-manager.io/inject-ca-from" = "capz-system/azureserviceoperator-serving-cert"' aso-crds.yaml
yq -i '.spec.conversion.webhook.clientConfig.service.namespace = "capz-system"' aso-crds.yaml

# Select and apply the CRDs listed in $ASO_CRD_GROUPS
for g in "${ASO_CRD_GROUPS[@]}"; do
    g=$g yq 'select(.spec.group == env(g))' aso-crds.yaml | kubectl apply --server-side --force-conflicts -f -
done

Here are some takeaways.

• The way we identify the CAPZ-bundled ASO version isn't super crisp (but needed given that ASO is strict about version matching). Would be nice if there was a simpler way to get the ASO (and CAPZ) version, perhaps via clusterctl get? Regarding CRD management overall, I prefer installing CRDs outside of Helm so I did not try any other approach.
• In terms of authentication we use workload identity with a user-assigned managed identity.
  • Global scope does not work, at least not seamlessly without having to modify the aso-controller-settings Secret. Didn't try too hard though, figured it's best to not touch stuff managed by clusterctl. It would be nice if there was an officially supported UX for global scope authentication.
  • Resource scope works just fine by adding the serviceoperator.azure.com/credential-from annotation to ASO CRs, referencing the <cluster-name>-aso-secret Secret that's created by CAPZ in conjunction with creating a workload cluster.
  • Namespace scope (without the annotation) works too if there's a Secret named aso-credential in the same namespace. This approach doesn't have any dependency on the existence of a CAPI/CAPZ workload cluster 👍

Example using resource scope authentication

apiVersion: resources.azure.com/v1api20200601
kind: ResourceGroup
metadata:
  name: testrg
  namespace: default
  annotations:
    serviceoperator.azure.com/credential-from: <cluster-name>-aso-secret
spec:
  location: westeurope

Example using namespace scope authentication

apiVersion: v1
stringData:
  AUTH_MODE: workloadidentity
  AZURE_CLIENT_ID: ***
  AZURE_SUBSCRIPTION_ID: ***
  AZURE_TENANT_ID: ***
kind: Secret
metadata:
  name: aso-credential
  namespace: default
---
apiVersion: resources.azure.com/v1api20200601
kind: ResourceGroup
metadata:
  name: testrg
  namespace: default
spec:
  location: westeurope