Skip to content

Commit

Permalink
ASOAPI: basic automated adoption
Browse files Browse the repository at this point in the history
  • Loading branch information
nojnhuh committed May 6, 2024
1 parent 39189af commit 076bde4
Show file tree
Hide file tree
Showing 12 changed files with 514 additions and 26 deletions.
12 changes: 12 additions & 0 deletions config/rbac/role.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,12 @@ rules:
- get
- list
- watch
- apiGroups:
- cluster.x-k8s.io
resources:
- clusters
verbs:
- create
- apiGroups:
- cluster.x-k8s.io
resources:
Expand All @@ -64,6 +70,12 @@ rules:
- list
- patch
- watch
- apiGroups:
- cluster.x-k8s.io
resources:
- machinepools
verbs:
- create
- apiGroups:
- cluster.x-k8s.io
resources:
Expand Down
34 changes: 33 additions & 1 deletion docs/book/src/topics/managedcluster.md
Original file line number Diff line number Diff line change
Expand Up @@ -629,6 +629,38 @@ Some notes about how this works under the hood:

## Adopting Existing AKS Clusters

### Option 1: Using the experimental ASO-based API

<!-- markdown-link-check-disable-next-line -->
The [experimental AzureASOManagedControlPlane and related APIs](/topics/aso.html#experimental-aso-api) support
adoption as a first-class use case. Going forward, this method is likely to be easier, more reliable, include
more features, and better supported for adopting AKS clusters than Option 2 below.

To adopt an AKS cluster into a full Cluster API Cluster, create an ASO ManagedCluster and associated
ManagedClustersAgentPool resources annotated with `sigs.k8s.io/cluster-api-provider-azure-adopt=true`. The
annotation may also be added to existing ASO resources to trigger adoption. CAPZ will automatically scaffold
the Cluster API resources like the Cluster, AzureASOManagedCluster, AzureASOManagedControlPlane, MachinePools,
and AzureASOManagedMachinePools. The [`asoctl import
azure-resource`](https://azure.github.io/azure-service-operator/tools/asoctl/#import-azure-resource) command
can help generate the required YAML.

Caveats:
- The `asoctl import azure-resource` command has at least [one known
bug](https://github.com/Azure/azure-service-operator/issues/3805) requiring the YAML it generates to be
edited before it can be applied to a cluster.
- CAPZ currently only records the ASO resources in the CAPZ resources' `spec.resources` that it needs to
function, which include the ManagedCluster, its ResourceGroup, and associated ManagedClustersAgentPools.
Other resources owned by the ManagedCluster like Kubernetes extensions or Fleet memberships are not
currently imported to the CAPZ specs.
- Configuring the automatically generated Cluster API resources is not currently possible. If you need to
change something like the `metadata.name` of a resource from what CAPZ generates, create the Cluster API
resources manually referencing the pre-existing resources.
- Adopting existing clusters created with the GA AzureManagedControlPlane API to the experimental API with
this method is theoretically possible, but untested. Care should be taken to prevent CAPZ from reconciling
two different representations of the same underlying Azure resources.

### Option 2: Using the current AzureManagedControlPlane API

<aside class="note">

<h1> Warning </h1>
Expand Down Expand Up @@ -665,7 +697,7 @@ Managed Cluster and Agent Pool resources do not need this tag in order to be ado
After applying the CAPI and CAPZ resources for the cluster, other means of managing the cluster should be
disabled to avoid ongoing conflicts with CAPZ's reconciliation process.

### Pitfalls
#### Pitfalls

The following describes some specific pieces of configuration that deserve particularly careful attention,
adapted from https://gist.github.com/mtougeron/1e5d7a30df396cd4728a26b2555e0ef0#file-capz-md.
Expand Down
202 changes: 202 additions & 0 deletions exp/controllers/agentpooladopt_controller.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,202 @@
/*
Copyright 2024 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package controllers

import (
"context"
"fmt"

asocontainerservicev1 "github.com/Azure/azure-service-operator/v2/api/containerservice/v1api20231001"
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/utils/ptr"
infrav1exp "sigs.k8s.io/cluster-api-provider-azure/exp/api/v1alpha1"
"sigs.k8s.io/cluster-api-provider-azure/util/tele"
clusterv1 "sigs.k8s.io/cluster-api/api/v1beta1"
expv1 "sigs.k8s.io/cluster-api/exp/api/v1beta1"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/event"
"sigs.k8s.io/controller-runtime/pkg/predicate"
)

// AgentPoolAdoptReconciler adopts ASO ManagedClustersAgentPool resources into a CAPI Cluster.
type AgentPoolAdoptReconciler struct {
client.Client
}

// SetupWithManager sets up the controller with the Manager.
func (r *AgentPoolAdoptReconciler) SetupWithManager(ctx context.Context, mgr ctrl.Manager) error {
_, err := ctrl.NewControllerManagedBy(mgr).
For(&asocontainerservicev1.ManagedClustersAgentPool{}).
WithEventFilter(predicate.Funcs{
UpdateFunc: func(ev event.UpdateEvent) bool {
return ev.ObjectOld.GetAnnotations()[adoptAnnotation] != ev.ObjectNew.GetAnnotations()[adoptAnnotation]
},
DeleteFunc: func(_ event.DeleteEvent) bool { return false },

Check warning on line 51 in exp/controllers/agentpooladopt_controller.go

View check run for this annotation

Codecov / codecov/patch

exp/controllers/agentpooladopt_controller.go#L44-L51

Added lines #L44 - L51 were not covered by tests
}).
Build(r)
if err != nil {
return err

Check warning on line 55 in exp/controllers/agentpooladopt_controller.go

View check run for this annotation

Codecov / codecov/patch

exp/controllers/agentpooladopt_controller.go#L54-L55

Added lines #L54 - L55 were not covered by tests
}

return nil

Check warning on line 58 in exp/controllers/agentpooladopt_controller.go

View check run for this annotation

Codecov / codecov/patch

exp/controllers/agentpooladopt_controller.go#L58

Added line #L58 was not covered by tests
}

// +kubebuilder:rbac:groups=cluster.x-k8s.io,resources=machinepools,verbs=create

// Reconcile reconciles an AzureASOManagedCluster.
func (r *AgentPoolAdoptReconciler) Reconcile(ctx context.Context, req ctrl.Request) (result ctrl.Result, resultErr error) {
ctx, log, done := tele.StartSpanWithLogger(ctx,
"controllers.AgentPoolAdoptReconciler.Reconcile",
tele.KVP("namespace", req.Namespace),
tele.KVP("name", req.Name),
tele.KVP("kind", "ManagedCluster"),
)
defer done()

Check warning on line 71 in exp/controllers/agentpooladopt_controller.go

View check run for this annotation

Codecov / codecov/patch

exp/controllers/agentpooladopt_controller.go#L64-L71

Added lines #L64 - L71 were not covered by tests

agentPool := &asocontainerservicev1.ManagedClustersAgentPool{}
err := r.Get(ctx, req.NamespacedName, agentPool)
if err != nil {
return ctrl.Result{}, client.IgnoreNotFound(err)

Check warning on line 76 in exp/controllers/agentpooladopt_controller.go

View check run for this annotation

Codecov / codecov/patch

exp/controllers/agentpooladopt_controller.go#L73-L76

Added lines #L73 - L76 were not covered by tests
}

if agentPool.GetAnnotations()[adoptAnnotation] != "true" {
return ctrl.Result{}, nil

Check warning on line 80 in exp/controllers/agentpooladopt_controller.go

View check run for this annotation

Codecov / codecov/patch

exp/controllers/agentpooladopt_controller.go#L79-L80

Added lines #L79 - L80 were not covered by tests
}

for _, owner := range agentPool.GetOwnerReferences() {
if owner.APIVersion == infrav1exp.GroupVersion.Identifier() &&
owner.Kind == infrav1exp.AzureASOManagedMachinePoolKind {
return ctrl.Result{}, nil

Check warning on line 86 in exp/controllers/agentpooladopt_controller.go

View check run for this annotation

Codecov / codecov/patch

exp/controllers/agentpooladopt_controller.go#L83-L86

Added lines #L83 - L86 were not covered by tests
}
}

log.Info("adopting")

Check warning on line 90 in exp/controllers/agentpooladopt_controller.go

View check run for this annotation

Codecov / codecov/patch

exp/controllers/agentpooladopt_controller.go#L90

Added line #L90 was not covered by tests

namespace := agentPool.Namespace

Check warning on line 92 in exp/controllers/agentpooladopt_controller.go

View check run for this annotation

Codecov / codecov/patch

exp/controllers/agentpooladopt_controller.go#L92

Added line #L92 was not covered by tests

// filter down to what will be persisted in the AzureASOManagedMachinePool
agentPool = &asocontainerservicev1.ManagedClustersAgentPool{
TypeMeta: metav1.TypeMeta{
APIVersion: asocontainerservicev1.GroupVersion.Identifier(),
Kind: "ManagedClustersAgentPool",
},
ObjectMeta: metav1.ObjectMeta{
Name: agentPool.Name,
},
Spec: agentPool.Spec,

Check warning on line 103 in exp/controllers/agentpooladopt_controller.go

View check run for this annotation

Codecov / codecov/patch

exp/controllers/agentpooladopt_controller.go#L95-L103

Added lines #L95 - L103 were not covered by tests
}

var replicas *int32
if agentPool.Spec.Count != nil {
replicas = ptr.To(int32(*agentPool.Spec.Count))
agentPool.Spec.Count = nil

Check warning on line 109 in exp/controllers/agentpooladopt_controller.go

View check run for this annotation

Codecov / codecov/patch

exp/controllers/agentpooladopt_controller.go#L106-L109

Added lines #L106 - L109 were not covered by tests
}

managedCluster := &asocontainerservicev1.ManagedCluster{}
if agentPool.Owner() == nil {
return ctrl.Result{}, fmt.Errorf("agent pool %s/%s has no owner", namespace, agentPool.Name)

Check warning on line 114 in exp/controllers/agentpooladopt_controller.go

View check run for this annotation

Codecov / codecov/patch

exp/controllers/agentpooladopt_controller.go#L112-L114

Added lines #L112 - L114 were not covered by tests
}
managedClusterKey := client.ObjectKey{
Namespace: namespace,
Name: agentPool.Owner().Name,

Check warning on line 118 in exp/controllers/agentpooladopt_controller.go

View check run for this annotation

Codecov / codecov/patch

exp/controllers/agentpooladopt_controller.go#L116-L118

Added lines #L116 - L118 were not covered by tests
}
err = r.Get(ctx, managedClusterKey, managedCluster)
if err != nil {
return ctrl.Result{}, fmt.Errorf("failed to get ManagedCluster %s: %w", managedClusterKey, err)

Check warning on line 122 in exp/controllers/agentpooladopt_controller.go

View check run for this annotation

Codecov / codecov/patch

exp/controllers/agentpooladopt_controller.go#L120-L122

Added lines #L120 - L122 were not covered by tests
}
var managedControlPlaneOwner *metav1.OwnerReference
for _, owner := range managedCluster.GetOwnerReferences() {
if owner.APIVersion == infrav1exp.GroupVersion.Identifier() &&
owner.Kind == infrav1exp.AzureASOManagedControlPlaneKind &&
owner.Name == agentPool.Owner().Name {
managedControlPlaneOwner = ptr.To(owner)
break

Check warning on line 130 in exp/controllers/agentpooladopt_controller.go

View check run for this annotation

Codecov / codecov/patch

exp/controllers/agentpooladopt_controller.go#L124-L130

Added lines #L124 - L130 were not covered by tests
}
}
if managedControlPlaneOwner == nil {
return ctrl.Result{}, fmt.Errorf("ManagedCluster %s is not owned by any AzureASOManagedControlPlane", managedClusterKey)

Check warning on line 134 in exp/controllers/agentpooladopt_controller.go

View check run for this annotation

Codecov / codecov/patch

exp/controllers/agentpooladopt_controller.go#L133-L134

Added lines #L133 - L134 were not covered by tests
}
asoManagedControlPlane := &infrav1exp.AzureASOManagedControlPlane{}
managedControlPlaneKey := client.ObjectKey{
Namespace: namespace,
Name: managedControlPlaneOwner.Name,

Check warning on line 139 in exp/controllers/agentpooladopt_controller.go

View check run for this annotation

Codecov / codecov/patch

exp/controllers/agentpooladopt_controller.go#L136-L139

Added lines #L136 - L139 were not covered by tests
}
err = r.Get(ctx, managedControlPlaneKey, asoManagedControlPlane)
if err != nil {
return ctrl.Result{}, fmt.Errorf("failed to get AzureASOManagedControlPlane %s: %w", managedControlPlaneKey, err)

Check warning on line 143 in exp/controllers/agentpooladopt_controller.go

View check run for this annotation

Codecov / codecov/patch

exp/controllers/agentpooladopt_controller.go#L141-L143

Added lines #L141 - L143 were not covered by tests
}
clusterName := asoManagedControlPlane.Labels[clusterv1.ClusterNameLabel]

Check warning on line 145 in exp/controllers/agentpooladopt_controller.go

View check run for this annotation

Codecov / codecov/patch

exp/controllers/agentpooladopt_controller.go#L145

Added line #L145 was not covered by tests

asoManagedMachinePool := &infrav1exp.AzureASOManagedMachinePool{
ObjectMeta: metav1.ObjectMeta{
Namespace: namespace,
Name: agentPool.Name,
},
Spec: infrav1exp.AzureASOManagedMachinePoolSpec{
AzureASOManagedMachinePoolTemplateResourceSpec: infrav1exp.AzureASOManagedMachinePoolTemplateResourceSpec{
Resources: []runtime.RawExtension{
{Object: agentPool},
},
},
},

Check warning on line 158 in exp/controllers/agentpooladopt_controller.go

View check run for this annotation

Codecov / codecov/patch

exp/controllers/agentpooladopt_controller.go#L147-L158

Added lines #L147 - L158 were not covered by tests
}

machinePool := &expv1.MachinePool{
ObjectMeta: metav1.ObjectMeta{
Namespace: namespace,
Name: agentPool.Name,
},
Spec: expv1.MachinePoolSpec{
ClusterName: clusterName,
Replicas: replicas,
Template: clusterv1.MachineTemplateSpec{
Spec: clusterv1.MachineSpec{
Bootstrap: clusterv1.Bootstrap{
DataSecretName: ptr.To(""),
},
ClusterName: clusterName,
InfrastructureRef: corev1.ObjectReference{
APIVersion: infrav1exp.GroupVersion.Identifier(),
Kind: infrav1exp.AzureASOManagedMachinePoolKind,
Name: asoManagedMachinePool.Name,
},
},
},
},

Check warning on line 182 in exp/controllers/agentpooladopt_controller.go

View check run for this annotation

Codecov / codecov/patch

exp/controllers/agentpooladopt_controller.go#L161-L182

Added lines #L161 - L182 were not covered by tests
}

if ptr.Deref(agentPool.Spec.EnableAutoScaling, false) {
machinePool.Annotations = map[string]string{
clusterv1.ReplicasManagedByAnnotation: infrav1exp.ReplicasManagedByAKS,

Check warning on line 187 in exp/controllers/agentpooladopt_controller.go

View check run for this annotation

Codecov / codecov/patch

exp/controllers/agentpooladopt_controller.go#L185-L187

Added lines #L185 - L187 were not covered by tests
}
}

err = r.Create(ctx, machinePool)
if client.IgnoreAlreadyExists(err) != nil {
return ctrl.Result{}, err

Check warning on line 193 in exp/controllers/agentpooladopt_controller.go

View check run for this annotation

Codecov / codecov/patch

exp/controllers/agentpooladopt_controller.go#L191-L193

Added lines #L191 - L193 were not covered by tests
}

err = r.Create(ctx, asoManagedMachinePool)
if client.IgnoreAlreadyExists(err) != nil {
return ctrl.Result{}, err

Check warning on line 198 in exp/controllers/agentpooladopt_controller.go

View check run for this annotation

Codecov / codecov/patch

exp/controllers/agentpooladopt_controller.go#L196-L198

Added lines #L196 - L198 were not covered by tests
}

return ctrl.Result{}, nil

Check warning on line 201 in exp/controllers/agentpooladopt_controller.go

View check run for this annotation

Codecov / codecov/patch

exp/controllers/agentpooladopt_controller.go#L201

Added line #L201 was not covered by tests
}
Loading

0 comments on commit 076bde4

Please sign in to comment.