Integrate Azure Service Operator

[nojnhuh]

/kind feature

This is an umbrella issue to track the overall progress of the effort described in the proposal to adopt ASO (#3113). It's currently a work-in-progress and will be updated as we identify and break down the work to be done.

Common Functionality

Preview Give feedback

1. Integrate ASO install with clusterctl init, tilt up #3520
   +0
   
2. Implement reconciliation of ASO resources #3521
   +0
   
3. Implement all currently supported flavors of Azure authentication for ASO #3526
   2 of 2
   +0
   
4. Manage resource tags with ASO #3522
   +0
   
5. Set reconcile-policy: skip on ASO resources when Cluster is paused #3525
   1 of 1
   +0
   
6. Expose ASO metrics in Tilt's Prometheus #3523
   +0
   
7. Collect ASO resources/logs in e2e artifacts #3532
   1 of 1
   +0
   
8. Enable ASO #3527
   9 of 9
   +0
   
9. Allow authentication with Azure Workload Identity for ASO #3732
   +0
   
10. Implement UserAssignedMSI auth for ASO #3743
    1 of 1
    +0
    
11. Handle CAPI's block-move annotation #3839
    kind/feature +1
    
12. Configure controllers to watch ASO resources and queue reconcile requests for the appropriate CAPZ resources #4339
    kind/cleanup priority/important-soon +2
13. Set a controller: true OwnerReference on ASO resources referring to its managing CAPZ resource #4340
    kind/feature +1
    
14. Removing a resource reference from a CAPZ spec doesn't delete it in Azure #4338
    kind/bug priority/important-longterm +2
    
15. CAPZ with ASO doesn't work for AzureUSGovernment #4113
    area/managedclusters kind/bug size/XL triage/accepted +4
16. Reuse Azure Service Operator for IaC #4065
    kind/feature priority/important-longterm size/L +3
    

New Tests

Preview Give feedback

1. Ensure both BYO Azure resource directly and BYO ASO resource are tested in e2e #3545
   1 of 1
   lifecycle/rotten +1
2. Test rollback from ASO to SDK-backed service implementation #3546
   2 of 2
   lifecycle/rotten +1
3. Assess scalability of CAPZ using ASO vs. Azure SDK #3547
   1 of 1
   priority/important-longterm size/XL +2
   
4. Test configuring ASO to install more CRDs than only what CAPZ requires
5. Test clusterctl move with ASO #3795
   +0
   

Service Implementations

Preview Give feedback

1. ASO service: groups #3524
   +0
   
2. ASO service: virtualnetworks and subnets #3528
   1 of 1
   +0
   
3. ASO service: managedclusters and agentpools #3529
   1 of 1
   area/managedclusters priority/important-soon size/L +3
   
4. ASO service: privateendpoints #3530
   1 of 1
   +0
   
5. ASO service: resourcehealth #3531
   1 of 2
   lifecycle/rotten +1
6. ASO service: natgateways #4040
   1 of 1
   kind/feature +1
   
7. ASO service: bastionhosts #4085
   +0
   
8. ASO service: disks #4084
   lifecycle/rotten +1
9. ASO service: inboundnatrules #4083
   lifecycle/rotten +1
10. ASO service: loadbalancers #4082
    lifecycle/rotten +1
11. ASO service: networkinterfaces #4081
    lifecycle/rotten +1
12. ASO service: privatedns #4080
    lifecycle/rotten +1
13. ASO service: publicips #4078
    lifecycle/rotten +1
14. ASO service: roleassignments #4077
    lifecycle/rotten +1
15. ASO service: routetables #4076
    lifecycle/rotten +1
16. ASO service: scalesets #4075
    lifecycle/rotten +1
17. ASO service: scalesetvms #4074
    lifecycle/rotten +1
18. ASO service: securitygroups #4073
    lifecycle/rotten +1
19. ASO service: virtualmachines #4072
    lifecycle/rotten +1
    
20. ASO service: vnetpeerings #4071
    lifecycle/rotten +1

Currently listed above are the services required for AzureManagedCluster which are the highest priority.

Relevant External Issues

Preview Give feedback

1. https://github.com/Azure/azure-service-operator/labels/capz

[nojnhuh]

Test configuring ASO to install more CRDs than only what CAPZ requires

I played around with this a bit and got it working AFACT. The steps I roughly took were:

1. Install CAPZ with ASO with clusterctl init
2. Install another ASO CRD
3. clusterctl upgrade
4. Check the ASO Pod is healthy and manually-installed CRD still exists

I added a section to the docs as part of #3723 with more details.

I think that gives me enough confidence to move forward for now.

[nojnhuh]

Here is an implementation/review checklist I put together for converting each service to use ASO: https://gist.github.com/nojnhuh/a1e06bb6d98b0eacb51134480ea4acea

cc @nawazkh @willie-yao

[dtzar]

I'd vote to close this issue out once we have everything completed in common functionality

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[nojnhuh]

I'd vote to close this issue out once we have everything completed in common functionality

The couple of lingering issues here I think are marginal enough that I think we can at least call this issue complete. We'll of course continue to track those open issues but I think it's fair to close this big tracking issue now.

/close

[k8s-ci-robot]

@nojnhuh: Closing this issue.

In response to this:

I'd vote to close this issue out once we have everything completed in common functionality

The couple of lingering issues here I think are marginal enough that I think we can at least call this issue complete. We'll of course continue to track those open issues but I think it's fair to close this big tracking issue now.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.