Integrate AzureMachine with AzureManagedControlPlane (BYO nodes on AKS)

[alexeldeib]

/kind feature

Describe the solution you'd like
Extend CAPZ as necessary to allow CAPZ-managed AzureMachines to join AKS clusters provisioned by AzureManagedControlPlane. A PoC exists in #822, which itself relies on #824 and #825. On top of those changes, we need to add logic in AzureMachineController to allow instantiating the appropriate cluster describer at runtime using the duck typing from #825.

Anything else you would like to add:

Currently there are two issues with the PoC:

• nodes register with master labels. I think this is due to using the admin kubeconfig to join nodes. we need to drop privileges and try to join in that way. We can also try doing a token join via kubeconfig with a low privilege serviceaccount.
• need to validate azure-cni functionality and/or debug any issues there

Warning

This is 100% unsupported by AKS right now, I just think it's cool and I don't see any technical reasons it can't work. This feature would be indefinitely experimental unless AKS supported BYO node

[alexeldeib]

Issues I encountered working on https://github.com/kubernetes-sigs/cluster-api-provider-azure/compare/master...alexeldeib:ace/integrate?expand=1

Critical issues // functional problems

• Currently nodes use an admin kubeconfig to join the cluster, then patch their own role and labels to drop privileges and become agents. This is awful security practice and I feel is the most critical item to fix before anyone could use this.
• Azure CNI requires pre-allocating IP for pods to the VMs, which is not currently handled by CAPZ. This would be a critical requirement for anyone using Azure CNI, unless we can take advantage of another IPAM/IP allocation scheme.
• Joining nodes with CAPBK uses 2 preKubeadmCommands and 2 postKubeadmCommands, which is less than ideal. We should consider encapsulating this more cleanly, either in a bootstrap provider or VHD/script somehow.

Minor issues // implementation details

• group service should not always attempt to reconcile the resource group, and it should differentiate node/control plane resource groups
• AKS vnet name is not predictable and nodes require it for provisioning. The managed cluster API does not return the vnet name, but we can extract it from the node pools API which contains vnetSubnetID as a fully qualified ARM resource
• anywhere that currently uses a cluster scope now needs to accept either an AzureCluster or AzureManagedControlPlane as a ClusterDescriber, increasing code bloat.
• Patching node labels/roles seems to work well on VM, but seems to have issues on VMSS when nodes get deleted and recreated?

Other observations // no action necessarily required

• CAPZ VHD does not include Azure CNI, which would be required for nodes joining AKS cluster
• AKS VHD does not include kubeadm, which would be required for CAPBK-driven bootstrap
• Can't access AKS Shared Image Gallery (??) using a random client, get auth failures

More testing required

• service type LoadBalancer (does it work?)
• further validation on CNI functionality

[alexeldeib]

/assign

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[CecileRobertMichon]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[alexeldeib]

/remove-lifecycle stale
/lifecycle frozen

still interested in this, but it's not super critical for the AzureMangedCluster/AzureManagedControlPlane functionality. I probably will have a small spike on some of the gritty details here, and then would need a lot of incremental refactoring to make the reconcilers/scopes work as expected.

[alexeldeib]

note: we should add .status.externalManagedControlPlane to AMCP so node deletion in machine controller works properly:
https://github.com/kubernetes-sigs/cluster-api/blob/master/controllers/machine_controller.go#L442-L467

[bridgetkromhout]

Related: kubernetes/kubernetes#112313

[CecileRobertMichon]

This is no longer blocked since #3861 is done