Merge branch 'master' into feature-aws-abs

charts/cluster-autoscaler/Chart.yaml

@@ -17,4 +17,4 @@ name: cluster-autoscaler
 sources:
   - https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler
 type: application
-version: 9.14.0
+version: 9.15.0

charts/cluster-autoscaler/README.md

@@ -209,6 +209,18 @@ Install the chart with
 ```
 $ helm install my-release autoscaler/cluster-autoscaler -f myvalues.yaml
 ```
+### Cluster-API
+
+`cloudProvider: clusterapi` must be set, and then one or more of
+- `autoDiscovery.clusterName`
+- or `autoDiscovery.labels`
+See [here](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/clusterapi/README.md#configuring-node-group-auto-discovery) for more details
+
+Additional config parameters avaible, see the `values.yaml` for more details
+`clusterAPIMode`
+`clusterAPIKubeconfigSecret`
+`clusterAPIWorkloadKubeconfigPath`
+`clusterAPICloudConfigPath`
 
 ## Uninstalling the Chart
 
@@ -338,7 +350,8 @@ Though enough for the majority of installations, the default PodSecurityPolicy _
 |-----|------|---------|-------------|
 | additionalLabels | object | `{}` | Labels to add to each object of the chart. |
 | affinity | object | `{}` | Affinity for pod assignment |
-| autoDiscovery.clusterName | string | `nil` | Enable autodiscovery for `cloudProvider=aws`, for groups matching `autoDiscovery.tags`. Enable autodiscovery for `cloudProvider=gce`, but no MIG tagging required. Enable autodiscovery for `cloudProvider=magnum`, for groups matching `autoDiscovery.roles`. |
+| autoDiscovery.clusterName | string | `nil` | Enable autodiscovery for `cloudProvider=aws`, for groups matching `autoDiscovery.tags`. Enable autodiscovery for `cloudProvider=clusterapi`, for groups matching `autoDiscovery.labels`. Enable autodiscovery for `cloudProvider=gce`, but no MIG tagging required. Enable autodiscovery for `cloudProvider=magnum`, for groups matching `autoDiscovery.roles`. |
+| autoDiscovery.labels | list | `[]` | Cluster-API labels to match  https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/clusterapi/README.md#configuring-node-group-auto-discovery |
 | autoDiscovery.roles | list | `["worker"]` | Magnum node group roles to match. |
 | autoDiscovery.tags | list | `["k8s.io/cluster-autoscaler/enabled","k8s.io/cluster-autoscaler/{{ .Values.autoDiscovery.clusterName }}"]` | ASG tags to match, run through `tpl`. |
 | autoscalingGroups | list | `[]` | For AWS, Azure AKS or Magnum. At least one element is required if not using `autoDiscovery`. For example: <pre> - name: asg1<br />   maxSize: 2<br />   minSize: 1 </pre> |
@@ -356,7 +369,11 @@ Though enough for the majority of installations, the default PodSecurityPolicy _
 | azureUseManagedIdentityExtension | bool | `false` | Whether to use Azure's managed identity extension for credentials. If using MSI, ensure subscription ID, resource group, and azure AKS cluster name are set. |
 | azureVMType | string | `"AKS"` | Azure VM type. |
 | cloudConfigPath | string | `"/etc/gce.conf"` | Configuration file for cloud provider. |
-| cloudProvider | string | `"aws"` | The cloud provider where the autoscaler runs. Currently only `gce`, `aws`, `azure` and `magnum` are supported. `aws` supported for AWS. `gce` for GCE. `azure` for Azure AKS. `magnum` for OpenStack Magnum. |
+| cloudProvider | string | `"aws"` | The cloud provider where the autoscaler runs. Currently only `gce`, `aws`, `azure`, `magnum` and `clusterapi` are supported. `aws` supported for AWS. `gce` for GCE. `azure` for Azure AKS. `magnum` for OpenStack Magnum, `clusterapi` for Cluster API. |
+| clusterAPICloudConfigPath | string | `"/etc/kubernetes/mgmt-kubeconfig"` | Path to kubeconfig for connecting to Cluster API Management Cluster, only used if `clusterAPIMode=kubeconfig-kubeconfig or incluster-kubeconfig` |
+| clusterAPIKubeconfigSecret | string | `""` | Secret containing kubeconfig for connecting to Cluster API managed workloadcluster Required if `cloudProvider=clusterapi` and `clusterAPIMode=kubeconfig-kubeconfig,kubeconfig-incluster or incluster-kubeconfig` |
+| clusterAPIMode | string | `"incluster-incluster"` | Cluster API mode, see https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/clusterapi/README.md#connecting-cluster-autoscaler-to-cluster-api-management-and-workload-clusters Syntax: workloadClusterMode-ManagementClusterMode for `kubeconfig-kubeconfig`, `incluster-kubeconfig` and `single-kubeconfig` you always must mount the external kubeconfig using either `extraVolumeSecrets` or `extraMounts` and `extraVolumes` if you dont set `clusterAPIKubeconfigSecret`and thus use an in-cluster config or want to use a non capi generated kubeconfig you must do so for the workload kubeconfig as well |
+| clusterAPIWorkloadKubeconfigPath | string | `"/etc/kubernetes/value"` | Path to kubeconfig for connecting to Cluster API managed workloadcluster, only used if `clusterAPIMode=kubeconfig-kubeconfig or kubeconfig-incluster` |
 | containerSecurityContext | object | `{}` | [Security context for container](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/) |
 | dnsPolicy | string | `"ClusterFirst"` | Defaults to `ClusterFirst`. Valid values are: `ClusterFirstWithHostNet`, `ClusterFirst`, `Default` or `None`. If autoscaler does not depend on cluster DNS, recommended to set this to `Default`. |
 | envFromConfigMap | string | `""` | ConfigMap name to use as envFrom. |
@@ -389,6 +406,7 @@ Though enough for the majority of installations, the default PodSecurityPolicy _
 | prometheusRule.interval | string | `nil` | How often rules in the group are evaluated (falls back to `global.evaluation_interval` if not set). |
 | prometheusRule.namespace | string | `"monitoring"` | Namespace which Prometheus is running in. |
 | prometheusRule.rules | list | `[]` | Rules spec template (see https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#rule). |
+| rbac.clusterScoped | bool | `true` | if set to false will only provision RBAC to alter resources in the current namespace. Most useful for Cluster-API |
 | rbac.create | bool | `true` | If `true`, create and use RBAC resources. |
 | rbac.pspEnabled | bool | `false` | If `true`, creates and uses RBAC resources required in the cluster with [Pod Security Policies](https://kubernetes.io/docs/concepts/policy/pod-security-policy/) enabled. Must be used with `rbac.create` set to `true`. |
 | rbac.serviceAccount.annotations | object | `{}` | Additional Service Account annotations. |

charts/cluster-autoscaler/README.md.gotmpl

@@ -209,6 +209,19 @@ Install the chart with
 ```
 $ helm install my-release autoscaler/cluster-autoscaler -f myvalues.yaml
 ```
+### Cluster-API
+
+`cloudProvider: clusterapi` must be set, and then one or more of
+- `autoDiscovery.clusterName`
+- or `autoDiscovery.labels`
+See [here](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/clusterapi/README.md#configuring-node-group-auto-discovery) for more details
+
+
+Additional config parameters avaible, see the `values.yaml` for more details
+`clusterAPIMode`
+`clusterAPIKubeconfigSecret`
+`clusterAPIWorkloadKubeconfigPath`
+`clusterAPICloudConfigPath`
 
 ## Uninstalling the Chart
 

charts/cluster-autoscaler/templates/_helpers.tpl

@@ -94,4 +94,24 @@ Return true if the priority expander is enabled
 {{- if has "priority" $expanders -}}
 {{- true -}}
 {{- end -}}
-{{- end -}}
+{{- end -}}
+
+{{/*
+Return the autodiscoveryparameters for clusterapi.
+*/}}
+{{- define "cluster-autoscaler.capiAutodiscoveryConfig" -}}
+{{- if .Values.autoDiscovery.clusterName -}}
+{{- print "clusterName=" -}}{{ .Values.autoDiscovery.clusterName }}
+{{- end -}}
+{{- if and .Values.autoDiscovery.clusterName .Values.autoDiscovery.labels -}}
+{{- print "," -}}
+{{- end -}}
+{{- if .Values.autoDiscovery.labels -}}
+{{- range $i, $el := .Values.autoDiscovery.labels -}}
+{{- if $i -}}{{- print "," -}}{{- end -}}
+{{- range $key, $val := $el -}}
+{{- $key -}}{{- print "=" -}}{{- $val -}}
+{{- end -}}
+{{- end -}}
+{{- end -}}
+{{- end -}}

charts/cluster-autoscaler/templates/clusterrole.yaml

@@ -1,4 +1,4 @@
-{{- if .Values.rbac.create -}}
+{{- if and .Values.rbac.create .Values.rbac.clusterScoped -}}
 apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRole
 metadata:
@@ -146,5 +146,18 @@ rules:
     verbs:
     - use
 {{- end -}}
-
+{{- if and ( and ( eq .Values.cloudProvider "clusterapi" ) ( .Values.rbac.clusterScoped ) ( or ( eq .Values.clusterAPIMode "incluster-incluster" ) ( eq .Values.clusterAPIMode "incluster-kubeconfig" ) ))}}
+  - apiGroups:
+    - cluster.x-k8s.io
+    resources:
+    - machinedeployments
+    - machinedeployments/scale
+    - machines
+    - machinesets
+    verbs:
+    - get
+    - list
+    - update
+    - watch
+{{- end }}
 {{- end -}}

charts/cluster-autoscaler/templates/clusterrolebinding.yaml

@@ -1,4 +1,4 @@
-{{- if .Values.rbac.create -}}
+{{- if and .Values.rbac.create .Values.rbac.clusterScoped -}}
 apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRoleBinding
 metadata:

charts/cluster-autoscaler/templates/deployment.yaml

@@ -1,4 +1,4 @@
-{{- if or .Values.autoDiscovery.clusterName .Values.autoscalingGroups }}
+{{- if or ( or .Values.autoDiscovery.clusterName .Values.autoDiscovery.labels ) .Values.autoscalingGroups }}
 {{/* one of the above is required */}}
 apiVersion: {{ template "deployment.apiVersion" . }}
 kind: Deployment
@@ -69,6 +69,21 @@ spec:
             {{- else }}
             - --cluster-name={{ .Values.magnumClusterName }}
             {{- end }}
+          {{- else if eq .Values.cloudProvider "clusterapi" }}
+            {{- if or .Values.autoDiscovery.clusterName .Values.autoDiscovery.labels }}
+            - --node-group-auto-discovery=clusterapi:{{ template "cluster-autoscaler.capiAutodiscoveryConfig" . }}
+            {{- end }}
+            {{- if eq .Values.clusterAPIMode "incluster-kubeconfig"}}
+            - --cloud-config={{ .Values.clusterAPICloudConfigPath }}
+            {{- else if eq .Values.clusterAPIMode "kubeconfig-incluster"}}
+            - --kubeconfig={{ .Values.clusterAPIWorkloadKubeconfigPath }}
+            - --clusterapi-cloud-config-authoritative
+            {{- else if eq .Values.clusterAPIMode "kubeconfig-kubeconfig"}}
+            - --kubeconfig={{ .Values.clusterAPIWorkloadKubeconfigPath }}
+            - --cloud-config={{ .Values.clusterAPICloudConfigPath }}
+            {{- else if eq .Values.clusterAPIMode "single-kubeconfig"}}
+            - --kubeconfig={{ .Values.clusterAPIWorkloadKubeconfigPath }}
+            {{- end }}
           {{- end }}
           {{- if eq .Values.cloudProvider "magnum" }}
             - --cloud-config={{ .Values.cloudConfigPath }}
@@ -203,6 +218,10 @@ spec:
               mountPath: {{ required "Must specify mountPath!" $value.mountPath }}
               readOnly: true
           {{- end }}
+          {{- if .Values.clusterAPIKubeconfigSecret }}
+            - name: cluster-api-kubeconfig
+              mountPath: {{ .Values.clusterAPIWorkloadKubeconfigPath | trimSuffix "/value" }}
+          {{- end }}
           {{- if .Values.extraVolumeMounts }}
             {{ toYaml .Values.extraVolumeMounts | nindent 12 }}
           {{- end }}
@@ -250,6 +269,11 @@ spec:
       {{- if .Values.extraVolumes }}
         {{- toYaml .Values.extraVolumes | nindent 10 }}
       {{- end }}
+      {{- if .Values.clusterAPIKubeconfigSecret }}
+        - name: cluster-api-kubeconfig
+          secret:
+            secretName: {{ .Values.clusterAPIKubeconfigSecret }}
+      {{- end }}
       {{- end }}
       {{- if .Values.image.pullSecrets }}
       imagePullSecrets:

charts/cluster-autoscaler/templates/role.yaml

@@ -43,4 +43,35 @@ rules:
       - get
       - update
 {{- end }}
+{{- if and ( and ( eq .Values.cloudProvider "clusterapi" ) ( not .Values.rbac.clusterScoped ) ( or ( eq .Values.clusterAPIMode "incluster-incluster" ) ( eq .Values.clusterAPIMode "incluster-kubeconfig" ) ))}}
+  - apiGroups:
+    - cluster.x-k8s.io
+    resources:
+    - machinedeployments
+    - machinedeployments/scale
+    - machines
+    - machinesets
+    verbs:
+    - get
+    - list
+    - update
+    - watch
+{{- end }}
+{{- if ( not .Values.rbac.clusterScoped ) }}
+  - apiGroups:
+    - coordination.k8s.io
+    resources:
+    - leases
+    verbs:
+    - create
+  - apiGroups:
+    - coordination.k8s.io
+    resourceNames:
+    - cluster-autoscaler
+    resources:
+    - leases
+    verbs:
+    - get
+    - update
+{{- end }}
 {{- end -}}

charts/cluster-autoscaler/values.yaml

@@ -3,24 +3,29 @@
 affinity: {}
 
 autoDiscovery:
-  # cloudProviders `aws`, `gce` and `magnum` are supported by auto-discovery at this time
+  # cloudProviders `aws`, `gce`, `magnum` and `clusterapi` are supported by auto-discovery at this time
   # AWS: Set tags as described in https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md#auto-discovery-setup
 
   # autoDiscovery.clusterName -- Enable autodiscovery for `cloudProvider=aws`, for groups matching `autoDiscovery.tags`.
+  # Enable autodiscovery for `cloudProvider=clusterapi`, for groups matching `autoDiscovery.labels`.
   # Enable autodiscovery for `cloudProvider=gce`, but no MIG tagging required.
   # Enable autodiscovery for `cloudProvider=magnum`, for groups matching `autoDiscovery.roles`.
   clusterName:  # cluster.local
 
   # autoDiscovery.tags -- ASG tags to match, run through `tpl`.
   tags:
-  - k8s.io/cluster-autoscaler/enabled
-  - k8s.io/cluster-autoscaler/{{ .Values.autoDiscovery.clusterName }}
+    - k8s.io/cluster-autoscaler/enabled
+    - k8s.io/cluster-autoscaler/{{ .Values.autoDiscovery.clusterName }}
   # - kubernetes.io/cluster/{{ .Values.autoDiscovery.clusterName }}
 
   # autoDiscovery.roles -- Magnum node group roles to match.
   roles:
-  - worker
+    - worker
 
+  # autoDiscovery.labels -- Cluster-API labels to match  https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/clusterapi/README.md#configuring-node-group-auto-discovery
+  labels: []
+    # - color: green
+    # - shape: circle
 # autoscalingGroups -- For AWS, Azure AKS or Magnum. At least one element is required if not using `autoDiscovery`. For example:
 # <pre>
 # - name: asg1<br />
@@ -99,13 +104,29 @@ magnumClusterName: ""
 # magnumCABundlePath -- Path to the host's CA bundle, from `ca-file` in the cloud-config file.
 magnumCABundlePath: "/etc/kubernetes/ca-bundle.crt"
 
+# clusterAPIMode --  Cluster API mode, see https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/clusterapi/README.md#connecting-cluster-autoscaler-to-cluster-api-management-and-workload-clusters
+# Syntax: workloadClusterMode-ManagementClusterMode
+# for `kubeconfig-kubeconfig`, `incluster-kubeconfig` and `single-kubeconfig` you always must mount the external kubeconfig using either `extraVolumeSecrets` or `extraMounts` and `extraVolumes`
+# if you dont set `clusterAPIKubeconfigSecret`and thus use an in-cluster config or want to use a non capi generated kubeconfig you must do so for the workload kubeconfig as well
+clusterAPIMode: incluster-incluster  # incluster-incluster, incluster-kubeconfig, kubeconfig-incluster, kubeconfig-kubeconfig, single-kubeconfig
+
+# clusterAPIKubeconfigSecret -- Secret containing kubeconfig for connecting to Cluster API managed workloadcluster
+# Required if `cloudProvider=clusterapi` and `clusterAPIMode=kubeconfig-kubeconfig,kubeconfig-incluster or incluster-kubeconfig`
+clusterAPIKubeconfigSecret: ""
+
+# clusterAPIWorkloadKubeconfigPath -- Path to kubeconfig for connecting to Cluster API managed workloadcluster, only used if `clusterAPIMode=kubeconfig-kubeconfig or kubeconfig-incluster`
+clusterAPIWorkloadKubeconfigPath: /etc/kubernetes/value
+
+# clusterAPICloudConfigPath -- Path to kubeconfig for connecting to Cluster API Management Cluster, only used if `clusterAPIMode=kubeconfig-kubeconfig or incluster-kubeconfig`
+clusterAPICloudConfigPath: /etc/kubernetes/mgmt-kubeconfig
+
 # cloudConfigPath -- Configuration file for cloud provider.
 cloudConfigPath: /etc/gce.conf
 
 # cloudProvider -- The cloud provider where the autoscaler runs.
-# Currently only `gce`, `aws`, `azure` and `magnum` are supported.
+# Currently only `gce`, `aws`, `azure`, `magnum` and `clusterapi` are supported.
 # `aws` supported for AWS. `gce` for GCE. `azure` for Azure AKS.
-# `magnum` for OpenStack Magnum.
+# `magnum` for OpenStack Magnum, `clusterapi` for Cluster API.
 cloudProvider: aws
 
 # containerSecurityContext -- [Security context for container](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/)
@@ -247,6 +268,8 @@ rbac:
   # rbac.pspEnabled -- If `true`, creates and uses RBAC resources required in the cluster with [Pod Security Policies](https://kubernetes.io/docs/concepts/policy/pod-security-policy/) enabled.
   # Must be used with `rbac.create` set to `true`.
   pspEnabled: false
+  # rbac.clusterScoped -- if set to false will only provision RBAC to alter resources in the current namespace. Most useful for Cluster-API
+  clusterScoped: true
   serviceAccount:
     # rbac.serviceAccount.annotations -- Additional Service Account annotations.
     annotations: {}

cluster-autoscaler/FAQ.md

@@ -683,8 +683,7 @@ would match the cluster size. This expander is described in more details
 
 * `priority` - selects the node group that has the highest priority assigned by the user. It's configuration is described in more details [here](expander/priority/readme.md)
 
-
-Multiple expanders may be passed, i.e.
+From 1.23.0 onwards, multiple expanders may be passed, i.e.
 `.cluster-autoscaler --expander=priority,least-waste`
 
 This will cause the `least-waste` expander to be used as a fallback in the event that the priority expander selects multiple node groups. In general, a list of expanders can be used, where the output of one is passed to the next and the final decision by randomly selecting one. An expander must not appear in the list more than once.

cluster-autoscaler/cloudprovider/aws/CA_with_AWS_IAM_OIDC.md

@@ -53,9 +53,11 @@ D) Set up [Cluster Autoscaler Auto-Discovery] using the [tutorial] .
 
 Note: The keys for the tags that you entered don't have values. Cluster Autoscaler ignores any value set for the keys.
 
-- Create an IAM Policy for cluster autoscaler and to enable AutoDiscovery. 
+__NOTE:__ Please see [the README](README.md#IAM-Policy) for more information on best practices with this IAM role.
 
-```sh
+- Create an IAM Policy for cluster autoscaler and to enable AutoDiscovery as well as discovery of instance types.
+
+```json
 {
     "Version": "2012-10-17",
     "Statement": [
@@ -66,17 +68,23 @@ Note: The keys for the tags that you entered don't have values. Cluster Autoscal
                 "autoscaling:DescribeAutoScalingInstances",
                 "autoscaling:DescribeLaunchConfigurations",
                 "autoscaling:DescribeTags",
+                "ec2:DescribeInstanceTypes",
+                "ec2:DescribeLaunchTemplateVersions"
+            ],
+            "Resource": ["*"]
+        },
+        {
+            "Effect": "Allow",
+            "Action": [
                 "autoscaling:SetDesiredCapacity",
                 "autoscaling:TerminateInstanceInAutoScalingGroup"
             ],
-            "Resource": "*"
+            "Resource": ["*"]
         }
     ]
 }
 ```
 
-NOTE: ``` autoscaling:DescribeTags ``` is very important if you are making use of the AutoDiscovery feature of the Cluster AutoScaler. 
-
 - Attach the above created policy to the *instance role* that's attached to your Amazon EKS worker nodes.
 - Download a deployment example file provided by the Cluster Autoscaler project on GitHub, run the following command: