-
Notifications
You must be signed in to change notification settings - Fork 1.3k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add examples of using kustomize with CAPI. Include information on modifying kustomize transformer configuration. Signed-off-by: Scott Lowe <[email protected]>
- Loading branch information
1 parent
c080446
commit a2251ad
Showing
1 changed file
with
154 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,154 @@ | ||
# Using Kustomize with Workload Cluster Manifests | ||
|
||
Although the `clusterctl config cluster` command exposes a number of different configuration values | ||
for customizing workload cluster YAML manifests, some users may need additional flexibility above | ||
and beyond what `clusterctl config cluster` or the example "flavor" templates that some CAPI providers | ||
supply (as an example, see [these flavor templates](https://github.com/kubernetes-sigs/cluster-api-provider-azure/tree/master/templates/flavors) | ||
for the Cluster API Provider for Azure). In the future, a [templating solution](https://github.com/kubernetes-sigs/cluster-api/issues/3252) | ||
may be integrated into `clusterctl` to help address this need, but in the meantime users can use | ||
`kustomize` as a solution to this need. | ||
|
||
This document provides a few examples of using `kustomize` with Cluster API. All of these examples | ||
assume that the "base" (unmodified) Cluster API configuration (perhaps generated using | ||
`clusterctl config cluster`) is in a directory whose relative path is `../../base`. | ||
|
||
## Example: Using Kustomize to Specify Custom Images | ||
|
||
Users can use `kustomize` to specify custom OS images for Cluster API nodes. Using the Cluster API | ||
Provider for AWS (CAPA) as an example, the following `kustomization.yaml` would leverage a JSON 6902 patch | ||
to modify the AMI for nodes in a workload cluster: | ||
|
||
```yaml | ||
--- | ||
apiVersion: kustomize.config.k8s.io/v1beta1 | ||
kind: Kustomization | ||
resources: | ||
- ../../base | ||
patchesJson6902: | ||
- path: custom-ami.json | ||
target: | ||
group: infrastructure.cluster.x-k8s.io | ||
kind: AWSMachineTemplate | ||
name: ".*" | ||
version: v1alpha3 | ||
``` | ||
The referenced JSON 6902 patch looks like this: | ||
```json | ||
[ | ||
{ "op": "add", "path": "/spec/template/spec/ami", "value": "ami-042db61632f72f145"} | ||
] | ||
``` | ||
|
||
This configuration assumes that the workload cluster _only_ uses MachineDeployments. Since | ||
MachineDeployments and the KubeadmControlPlane both leverage AWSMachineTemplates, this `kustomize` | ||
configuration would catch all nodes in the workload cluster. | ||
|
||
## Example: Adding a MachineHealthCheck for a Workload Cluster | ||
|
||
Users could also use `kustomize` to combine additional resources, like a MachineHealthCheck (MHC), with the | ||
base Cluster API manifest. In an overlay, specify the following in `kustomization.yaml`: | ||
|
||
```yaml | ||
--- | ||
apiVersion: kustomize.config.k8s.io/v1beta1 | ||
kind: Kustomization | ||
resources: | ||
- ../../base | ||
- workload-mhc.yaml | ||
``` | ||
The content of the `workload-mhc.yaml` file would be the definition of a standard MHC: | ||
|
||
```yaml | ||
apiVersion: cluster.x-k8s.io/v1alpha3 | ||
kind: MachineHealthCheck | ||
metadata: | ||
name: md-0-mhc | ||
spec: | ||
clusterName: test | ||
# maxUnhealthy: 40% | ||
nodeStartupTimeout: 10m | ||
selector: | ||
matchLabels: | ||
cluster.x-k8s.io/deployment-name: md-0 | ||
unhealthyConditions: | ||
- type: Ready | ||
status: Unknown | ||
timeout: 300s | ||
- type: Ready | ||
status: "False" | ||
timeout: 300s | ||
``` | ||
|
||
Running `kustomize build .` with this configuration would append the MHC to the base | ||
Cluster API manifest, thus creating the MHC at the same time as the workload cluster. | ||
|
||
## Configuring Kustomize for Cluster API | ||
|
||
Although `kustomize` can be used with Cluster API without any changes, modifying the transformer | ||
configurations for `kustomize` can make it more effective with Cluster API. For example, changes to | ||
the `nameReference` transformer in `kustomize` will enable `kustomize` to know about the references | ||
between Cluster API objects in a manifest. See [here](https://github.com/kubernetes-sigs/kustomize/tree/master/examples/transformerconfigs) | ||
for more information on transformer configurations. | ||
|
||
Add the following content to the `namereference.yaml` transformer configuration: | ||
|
||
```yaml | ||
- kind: Cluster | ||
group: cluster.x-k8s.io | ||
version: v1alpha3 | ||
fieldSpecs: | ||
- path: spec/clusterName | ||
kind: MachineDeployment | ||
- path: spec/template/spec/clusterName | ||
kind: MachineDeployment | ||
- kind: AWSCluster | ||
group: infrastructure.cluster.x-k8s.io | ||
version: v1alpha3 | ||
fieldSpecs: | ||
- path: spec/infrastructureRef/name | ||
kind: Cluster | ||
- kind: KubeadmControlPlane | ||
group: controlplane.cluster.x-k8s.io | ||
version: v1alpha3 | ||
fieldSpecs: | ||
- path: spec/controlPlaneRef/name | ||
kind: Cluster | ||
- kind: AWSMachine | ||
group: infrastructure.cluster.x-k8s.io | ||
version: v1alpha3 | ||
fieldSpecs: | ||
- path: spec/infrastructureRef/name | ||
kind: Machine | ||
- kind: KubeadmConfig | ||
group: bootstrap.cluster.x-k8s.io | ||
version: v1alpha3 | ||
fieldSpecs: | ||
- path: spec/bootstrap/configRef/name | ||
kind: Machine | ||
- kind: AWSMachineTemplate | ||
group: infrastructure.cluster.x-k8s.io | ||
version: v1alpha3 | ||
fieldSpecs: | ||
- path: spec/template/spec/infrastructureRef/name | ||
kind: MachineDeployment | ||
- path: spec/infrastructureTemplate/name | ||
kind: KubeadmControlPlane | ||
- kind: KubeadmConfigTemplate | ||
group: bootstrap.cluster.x-k8s.io | ||
version: v1alpha3 | ||
fieldSpecs: | ||
- path: spec/template/spec/bootstrap/configRef/name | ||
kind: MachineDeployment | ||
``` | ||
|
||
This will enable the use of the `namePrefix` and `nameSuffix` directives in `kustomization.yaml` | ||
and ensure that _all_ references to the names are properly updated. |