Skip to content

Commit

Permalink
Merge pull request #3719 from gjtempleton/cluster-autoscaler-chart-Ch…
Browse files Browse the repository at this point in the history
…art-Deprecate-Final-Version

CA-Chart - Mark Cluster-Autoscaler-Chart as Deprecated
k8s-ci-robot authored Feb 12, 2021
2 parents 1c4946d + cf4e5db commit 962bbd0
Showing 19 changed files with 1,617 additions and 0 deletions.
23 changes: 23 additions & 0 deletions charts/cluster-autoscaler-chart/.helmignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Patterns to ignore when building packages.
# This supports shell glob matching, relative path matching, and
# negation (prefixed with !). Only one pattern per line.
.DS_Store
# Common VCS dirs
.git/
.gitignore
.bzr/
.bzrignore
.hg/
.hgignore
.svn/
# Common backup files
*.swp
*.bak
*.tmp
*.orig
*~
# Various IDEs
.project
.idea/
*.tmproj
.vscode/
21 changes: 21 additions & 0 deletions charts/cluster-autoscaler-chart/Chart.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
apiVersion: v2
appVersion: 1.18.1
description: Scales Kubernetes worker nodes within autoscaling groups.
engine: gotpl
home: https://github.com/kubernetes/autoscaler
icon: https://github.com/kubernetes/kubernetes/blob/master/logo/logo.png
maintainers:
- email: e.bailey@sportradar.com
name: yurrriq
- email: mgoodness@gmail.com
name: mgoodness
- email: guyjtempleton@googlemail.com
name: gjtempleton
- email: scott.crooks@gmail.com
name: sc250024
name: cluster-autoscaler-chart
sources:
- https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler
type: application
version: 2.0.0
deprecated: true
354 changes: 354 additions & 0 deletions charts/cluster-autoscaler-chart/README.md

Large diffs are not rendered by default.

292 changes: 292 additions & 0 deletions charts/cluster-autoscaler-chart/README.md.gotmpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,292 @@
{{ template "chart.header" . }}

{{ template "chart.description" . }}

# NOTE:

This chart is now deprecated and has been renamed from `cluster-autoscaler-chart` to `cluster-autoscaler` in the same repository, no further releases will be made under this name.

## TL;DR:

```console
$ helm repo add autoscaler https://kubernetes.github.io/autoscaler

# Method 1 - Using Autodiscovery
$ helm install my-release autoscaler/cluster-autoscaler-chart \
--set 'autoDiscovery.clusterName'=<CLUSTER NAME>

# Method 2 - Specifying groups manually
$ helm install my-release autoscaler/cluster-autoscaler-chart \
--set "autoscalingGroups[0].name=your-asg-name" \
--set "autoscalingGroups[0].maxSize=10" \
--set "autoscalingGroups[0].minSize=1"
```

## Introduction

This chart bootstraps a cluster-autoscaler deployment on a [Kubernetes](http://kubernetes.io) cluster using the [Helm](https://helm.sh) package manager.

## Prerequisites

- Helm 3+
- Kubernetes 1.8+
- [Older versions](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler#releases) may work by overriding the `image`. Cluster autoscaler internally simulates the scheduler and bugs between mismatched versions may be subtle.
- Azure AKS specific Prerequisites:
- Kubernetes 1.10+ with RBAC-enabled.

## Previous Helm Chart

The previous `cluster-autoscaler` Helm chart hosted at [helm/charts](https://github.com/helm/charts) has been moved to this repository in accordance with the [Deprecation timeline](https://github.com/helm/charts#deprecation-timeline). Note that a few things have changed between this version and the old version:

- This repository **only** supports Helm chart installations using Helm 3+ since the `apiVersion` on the charts has been marked as `v2`.
- Previous versions of the Helm chart have not been migrated, and the version was reset to `1.0.0` at the onset. If you are looking for old versions of the chart, it's best to run `helm pull stable/cluster-autoscaler --version <your-version>` until you are ready to move to this repository's version.

## Installing the Chart

**By default, no deployment is created and nothing will autoscale**.

You must provide some minimal configuration, either to specify instance groups or enable auto-discovery. It is not recommended to do both.

Either:

- Set `autoDiscovery.clusterName` and tag your autoscaling groups appropriately (`--cloud-provider=aws` only) **or**
- Set at least one ASG as an element in the `autoscalingGroups` array with its three values: `name`, `minSize` and `maxSize`.

To install the chart with the release name `my-release`:

### AWS - Using auto-discovery of tagged instance groups

Auto-discovery finds ASGs tags as below and automatically manages them based on the min and max size specified in the ASG. `cloudProvider=aws` only.

- Tag the ASGs with keys to match `.Values.autoDiscovery.tags`, by default: `k8s.io/cluster-autoscaler/enabled` and `k8s.io/cluster-autoscaler/<YOUR CLUSTER NAME>`
- Verify the [IAM Permissions](#aws---iam)
- Set `autoDiscovery.clusterName=<YOUR CLUSTER NAME>`
- Set `awsRegion=<YOUR AWS REGION>`
- Set `awsAccessKeyID=<YOUR AWS KEY ID>` and `awsSecretAccessKey=<YOUR AWS SECRET KEY>` if you want to [use AWS credentials directly instead of an instance role](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md#using-aws-credentials)

```console
$ helm install my-release autoscaler/cluster-autoscaler-chart --set autoDiscovery.clusterName=<CLUSTER NAME>
```

#### Specifying groups manually

Without autodiscovery, specify an array of elements each containing ASG name, min size, max size. The sizes specified here will be applied to the ASG, assuming IAM permissions are correctly configured.

- Verify the [IAM Permissions](#aws---iam)
- Either provide a yaml file setting `autoscalingGroups` (see values.yaml) or use `--set` e.g.:

```console
$ helm install my-release autoscaler/cluster-autoscaler-chart \
--set "autoscalingGroups[0].name=your-asg-name" \
--set "autoscalingGroups[0].maxSize=10" \
--set "autoscalingGroups[0].minSize=1"
```

#### Auto-discovery

For auto-discovery of instances to work, they must be tagged with the keys in `.Values.autoDiscovery.tags`, which by default are
`k8s.io/cluster-autoscaler/enabled` and `k8s.io/cluster-autoscaler/<ClusterName>`

The value of the tag does not matter, only the key.

An example kops spec excerpt:

```yaml
apiVersion: kops/v1alpha2
kind: Cluster
metadata:
name: my.cluster.internal
spec:
additionalPolicies:
node: |
[
{"Effect":"Allow","Action":["autoscaling:DescribeAutoScalingGroups","autoscaling:DescribeAutoScalingInstances","autoscaling:DescribeLaunchConfigurations","autoscaling:DescribeTags","autoscaling:SetDesiredCapacity","autoscaling:TerminateInstanceInAutoScalingGroup"],"Resource":"*"}
]
...
---
apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
labels:
kops.k8s.io/cluster: my.cluster.internal
name: my-instances
spec:
cloudLabels:
k8s.io/cluster-autoscaler/enabled: ""
k8s.io/cluster-autoscaler/my.cluster.internal: ""
image: kops.io/k8s-1.8-debian-jessie-amd64-hvm-ebs-2018-01-14
machineType: r4.large
maxSize: 4
minSize: 0
```

In this example you would need to `--set autoDiscovery.clusterName=my.cluster.internal` when installing.

It is not recommended to try to mix this with setting `autoscalingGroups`

See [autoscaler AWS documentation](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md#auto-discovery-setup) for a more discussion of the setup.

### GCE

The following parameters are required:

- `autoDiscovery.clusterName=any-name`
- `cloud-provider=gce`
- `autoscalingGroupsnamePrefix[0].name=your-ig-prefix,autoscalingGroupsnamePrefix[0].maxSize=10,autoscalingGroupsnamePrefix[0].minSize=1`

To use Managed Instance Group (MIG) auto-discovery, provide a YAML file setting `autoscalingGroupsnamePrefix` (see values.yaml) or use `--set` when installing the Chart - e.g.

```console
$ helm install my-release autoscaler/cluster-autoscaler-chart \
--set "autoscalingGroupsnamePrefix[0].name=your-ig-prefix,autoscalingGroupsnamePrefix[0].maxSize=10,autoscalingGroupsnamePrefi[0].minSize=1" \
--set autoDiscovery.clusterName=<CLUSTER NAME> \
--set cloudProvider=gce
```

Note that `your-ig-prefix` should be a _prefix_ matching one or more MIGs, and _not_ the full name of the MIG. For example, to match multiple instance groups - `k8s-node-group-a-standard`, `k8s-node-group-b-gpu`, you would use a prefix of `k8s-node-group-`.

In the event you want to explicitly specify MIGs instead of using auto-discovery, set members of the `autoscalingGroups` array directly - e.g.

```
# where 'n' is the index, starting at 0
-- set autoscalingGroups[n].name=https://content.googleapis.com/compute/v1/projects/$PROJECTID/zones/$ZONENAME/instanceGroupManagers/$FULL-MIG-NAME,autoscalingGroups[n].maxSize=$MAXSIZE,autoscalingGroups[n].minSize=$MINSIZE
```

### Azure AKS

The following parameters are required:

- `cloudProvider=azure`
- `autoscalingGroups[0].name=your-agent-pool,autoscalingGroups[0].maxSize=10,autoscalingGroups[0].minSize=1`
- `azureClientID: "your-service-principal-app-id"`
- `azureClientSecret: "your-service-principal-client-secret"`
- `azureSubscriptionID: "your-azure-subscription-id"`
- `azureTenantID: "your-azure-tenant-id"`
- `azureClusterName: "your-aks-cluster-name"`
- `azureResourceGroup: "your-aks-cluster-resource-group-name"`
- `azureVMType: "AKS"`
- `azureNodeResourceGroup: "your-aks-cluster-node-resource-group"`

## Uninstalling the Chart

To uninstall `my-release`:

```console
$ helm uninstall my-release
```

The command removes all the Kubernetes components associated with the chart and deletes the release.

> **Tip**: List all releases using `helm list` or start clean with `helm uninstall my-release`

## Additional Configuration

### AWS - IAM

The worker running the cluster autoscaler will need access to certain resources and actions:

```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"autoscaling:DescribeAutoScalingGroups",
"autoscaling:DescribeAutoScalingInstances",
"autoscaling:DescribeLaunchConfigurations",
"autoscaling:DescribeTags",
"autoscaling:SetDesiredCapacity",
"autoscaling:TerminateInstanceInAutoScalingGroup"
],
"Resource": "*"
}
]
}
```

- `DescribeTags` is required for autodiscovery.
- `DescribeLaunchConfigurations` is required to scale up an ASG from 0.

If you would like to limit the scope of the Cluster Autoscaler to ***only*** modify ASGs for a particular cluster, use the following policy instead:

```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"autoscaling:DescribeAutoScalingGroups",
"autoscaling:DescribeAutoScalingInstances",
"autoscaling:DescribeLaunchConfigurations",
"autoscaling:DescribeTags",
"ec2:DescribeLaunchTemplateVersions"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"autoscaling:SetDesiredCapacity",
"autoscaling:TerminateInstanceInAutoScalingGroup",
"autoscaling:UpdateAutoScalingGroup"
],
"Resource": [
"arn:aws:autoscaling:<aws-region>:<account-id>:autoScalingGroup:<some-random-id>:autoScalingGroupName/node-group-1",
"arn:aws:autoscaling:<aws-region>:<account-id>:autoScalingGroup:<some-random-id>:autoScalingGroupName/node-group-2",
"arn:aws:autoscaling:<aws-region>:<account-id>:autoScalingGroup:<some-random-id>:autoScalingGroupName/node-group-3"
],
"Condition": {
"StringEquals": {
"autoscaling:ResourceTag/k8s.io/cluster-autoscaler/enabled": "true",
"autoscaling:ResourceTag/kubernetes.io/cluster/<cluster-name>": "owned"
}
}
}
]
}
```

Make sure to replace the variables `<aws-region>`, `<cluster-name>`, `<account-id>`, and the ARNs of the ASGs where applicable.

### AWS - IAM Roles for Service Accounts (IRSA)

For Kubernetes clusters that use Amazon EKS, the service account can be configured with an IAM role using [IAM Roles for Service Accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) to avoid needing to grant access to the worker nodes for AWS resources.

In order to accomplish this, you will first need to create a new IAM role with the above mentions policies. Take care in [configuring the trust relationship](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html#iam-role-configuration) to restrict access just to the service account used by cluster autoscaler.

Once you have the IAM role configured, you would then need to `--set rbac.serviceAccountAnnotations."eks\.amazonaws\.com/role-arn"=arn:aws:iam::123456789012:role/MyRoleName` when installing.

## Troubleshooting

The chart will succeed even if the container arguments are incorrect. A few minutes after starting
`kubectl logs -l "app=aws-cluster-autoscaler" --tail=50` should loop through something like

```
polling_autoscaler.go:111] Poll finished
static_autoscaler.go:97] Starting main loop
utils.go:435] No pod using affinity / antiaffinity found in cluster, disabling affinity predicate for this loop
static_autoscaler.go:230] Filtering out schedulables
```

If not, find a pod that the deployment created and `describe` it, paying close attention to the arguments under `Command`. e.g.:

```
Containers:
cluster-autoscaler:
Command:
./cluster-autoscaler
--cloud-provider=aws
# if specifying ASGs manually
--nodes=1:10:your-scaling-group-name
# if using autodiscovery
--node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/<ClusterName>
--v=4
```

### PodSecurityPolicy

Though enough for the majority of installations, the default PodSecurityPolicy _could_ be too restrictive depending on the specifics of your release. Please make sure to check that the template fits with any customizations made or disable it by setting `rbac.pspEnabled` to `false`.

{{ template "chart.valuesSection" . }}
18 changes: 18 additions & 0 deletions charts/cluster-autoscaler-chart/templates/NOTES.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
{{- if or .Values.autoDiscovery.clusterName .Values.autoscalingGroups -}}

To verify that cluster-autoscaler has started, run:

kubectl --namespace={{ .Release.Namespace }} get pods -l "app.kubernetes.io/name={{ template "cluster-autoscaler.name" . }},app.kubernetes.io/instance={{ .Release.Name }}"

{{- else -}}

##############################################################################
#### ERROR: You must specify values for either ####
#### autoDiscovery.clusterName or autoscalingGroups[] ####
##############################################################################

The deployment and pod will not be created and the installation is not functional
See README:
open https://github.com/kubernetes/charts/tree/master/stable/cluster-autoscaler

{{- end -}}
84 changes: 84 additions & 0 deletions charts/cluster-autoscaler-chart/templates/_helpers.tpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
{{/* vim: set filetype=mustache: */}}
{{/*
Expand the name of the chart.
*/}}
{{- define "cluster-autoscaler.name" -}}
{{- default (printf "%s-%s" .Values.cloudProvider .Chart.Name) .Values.nameOverride | trunc 63 | trimSuffix "-" -}}
{{- end -}}

{{/*
Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
*/}}
{{- define "cluster-autoscaler.fullname" -}}
{{- if .Values.fullnameOverride -}}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- $name := default (printf "%s-%s" .Values.cloudProvider .Chart.Name) .Values.nameOverride -}}
{{- if ne $name .Release.Name -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- printf "%s" $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{- end -}}
{{- end -}}

{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "cluster-autoscaler.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}}
{{- end -}}

{{/*
Return instance and name labels.
*/}}
{{- define "cluster-autoscaler.instance-name" -}}
app.kubernetes.io/instance: {{ .Release.Name | quote }}
app.kubernetes.io/name: {{ include "cluster-autoscaler.name" . | quote }}
{{- end -}}


{{/*
Return labels, including instance and name.
*/}}
{{- define "cluster-autoscaler.labels" -}}
{{ include "cluster-autoscaler.instance-name" . }}
app.kubernetes.io/managed-by: {{ .Release.Service | quote }}
helm.sh/chart: {{ include "cluster-autoscaler.chart" . | quote }}
{{- end -}}

{{/*
Return the appropriate apiVersion for deployment.
*/}}
{{- define "deployment.apiVersion" -}}
{{- $kubeTargetVersion := default .Capabilities.KubeVersion.GitVersion .Values.kubeTargetVersionOverride }}
{{- if semverCompare "<1.9-0" $kubeTargetVersion -}}
{{- print "apps/v1beta2" -}}
{{- else -}}
{{- print "apps/v1" -}}
{{- end -}}
{{- end -}}

{{/*
Return the appropriate apiVersion for podsecuritypolicy.
*/}}
{{- define "podsecuritypolicy.apiVersion" -}}
{{- $kubeTargetVersion := default .Capabilities.KubeVersion.GitVersion .Values.kubeTargetVersionOverride }}
{{- if semverCompare "<1.10-0" $kubeTargetVersion -}}
{{- print "extensions/v1beta1" -}}
{{- else -}}
{{- print "policy/v1beta1" -}}
{{- end -}}
{{- end -}}

{{/*
Return the service account name used by the pod.
*/}}
{{- define "cluster-autoscaler.serviceAccountName" -}}
{{- if .Values.rbac.serviceAccount.create -}}
{{ default (include "cluster-autoscaler.fullname" .) .Values.rbac.serviceAccount.name }}
{{- else -}}
{{ default "default" .Values.rbac.serviceAccount.name }}
{{- end -}}
{{- end -}}
147 changes: 147 additions & 0 deletions charts/cluster-autoscaler-chart/templates/clusterrole.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
{{- if .Values.rbac.create -}}
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
{{ include "cluster-autoscaler.labels" . | indent 4 }}
name: {{ template "cluster-autoscaler.fullname" . }}
rules:
- apiGroups:
- ""
resources:
- events
- endpoints
verbs:
- create
- patch
- apiGroups:
- ""
resources:
- pods/eviction
verbs:
- create
- apiGroups:
- ""
resources:
- pods/status
verbs:
- update
- apiGroups:
- ""
resources:
- endpoints
resourceNames:
- cluster-autoscaler
verbs:
- get
- update
- apiGroups:
- ""
resources:
- nodes
verbs:
- watch
- list
- get
- update
- apiGroups:
- ""
resources:
- pods
- services
- replicationcontrollers
- persistentvolumeclaims
- persistentvolumes
verbs:
- watch
- list
- get
- apiGroups:
- batch
resources:
- jobs
- cronjobs
verbs:
- watch
- list
- get
- apiGroups:
- batch
- extensions
resources:
- jobs
verbs:
- get
- list
- patch
- watch
- apiGroups:
- extensions
resources:
- replicasets
- daemonsets
verbs:
- watch
- list
- get
- apiGroups:
- policy
resources:
- poddisruptionbudgets
verbs:
- watch
- list
- apiGroups:
- apps
resources:
- daemonsets
- replicasets
- statefulsets
verbs:
- watch
- list
- get
- apiGroups:
- storage.k8s.io
resources:
- storageclasses
- csinodes
verbs:
- watch
- list
- get
- apiGroups:
- ""
resources:
- configmaps
verbs:
- list
- watch
- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- create
- apiGroups:
- coordination.k8s.io
resourceNames:
- cluster-autoscaler
resources:
- leases
verbs:
- get
- update
{{- if .Values.rbac.pspEnabled }}
- apiGroups:
- extensions
- policy
resources:
- podsecuritypolicies
resourceNames:
- {{ template "cluster-autoscaler.fullname" . }}
verbs:
- use
{{- end -}}

{{- end -}}
16 changes: 16 additions & 0 deletions charts/cluster-autoscaler-chart/templates/clusterrolebinding.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
{{- if .Values.rbac.create -}}
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
{{ include "cluster-autoscaler.labels" . | indent 4 }}
name: {{ template "cluster-autoscaler.fullname" . }}
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: {{ template "cluster-autoscaler.fullname" . }}
subjects:
- kind: ServiceAccount
name: {{ template "cluster-autoscaler.serviceAccountName" . }}
namespace: {{ .Release.Namespace }}
{{- end -}}
203 changes: 203 additions & 0 deletions charts/cluster-autoscaler-chart/templates/deployment.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,203 @@
{{- if or .Values.autoDiscovery.clusterName .Values.autoscalingGroups }}
{{/* one of the above is required */}}
apiVersion: {{ template "deployment.apiVersion" . }}
kind: Deployment
metadata:
labels:
{{ include "cluster-autoscaler.labels" . | indent 4 }}
name: {{ template "cluster-autoscaler.fullname" . }}
spec:
replicas: {{ .Values.replicaCount }}
selector:
matchLabels:
{{ include "cluster-autoscaler.instance-name" . | indent 6 }}
{{- if .Values.podLabels }}
{{ toYaml .Values.podLabels | indent 6 }}
{{- end }}
{{- if .Values.updateStrategy }}
strategy:
{{ toYaml .Values.updateStrategy | nindent 4 | trim }}
{{- end }}
template:
metadata:
{{- if .Values.podAnnotations }}
annotations:
{{ toYaml .Values.podAnnotations | indent 8 }}
{{- end }}
labels:
{{ include "cluster-autoscaler.instance-name" . | indent 8 }}
{{- if .Values.podLabels }}
{{ toYaml .Values.podLabels | indent 8 }}
{{- end }}
spec:
{{- if .Values.priorityClassName }}
priorityClassName: "{{ .Values.priorityClassName }}"
{{- end }}
{{- if .Values.dnsPolicy }}
dnsPolicy: "{{ .Values.dnsPolicy }}"
{{- end }}
containers:
- name: {{ template "cluster-autoscaler.name" . }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
imagePullPolicy: "{{ .Values.image.pullPolicy }}"
command:
- ./cluster-autoscaler
- --cloud-provider={{ .Values.cloudProvider }}
- --namespace={{ .Release.Namespace }}
{{- if .Values.autoscalingGroups }}
{{- range .Values.autoscalingGroups }}
- --nodes={{ .minSize }}:{{ .maxSize }}:{{ .name }}
{{- end }}
{{- end }}
{{- if eq .Values.cloudProvider "aws" }}
{{- if .Values.autoDiscovery.clusterName }}
- --node-group-auto-discovery=asg:tag={{ tpl (join "," .Values.autoDiscovery.tags) . }}
{{- end }}
{{- else if eq .Values.cloudProvider "gce" }}
{{- if .Values.autoscalingGroupsnamePrefix }}
{{- range .Values.autoscalingGroupsnamePrefix }}
- --node-group-auto-discovery=mig:namePrefix={{ .name }},min={{ .minSize }},max={{ .maxSize }}
{{- end }}
{{- end }}
{{- end }}
{{- if eq .Values.cloudProvider "gce" }}
- --cloud-config={{ .Values.cloudConfigPath }}
{{- end }}
{{- range $key, $value := .Values.extraArgs }}
- --{{ $key }}={{ $value }}
{{- end }}

env:
{{- if and (eq .Values.cloudProvider "aws") (ne .Values.awsRegion "") }}
- name: AWS_REGION
value: "{{ .Values.awsRegion }}"
{{- if .Values.awsAccessKeyID }}
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
key: AwsAccessKeyId
name: {{ template "cluster-autoscaler.fullname" . }}
{{- end }}
{{- if .Values.awsSecretAccessKey }}
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
key: AwsSecretAccessKey
name: {{ template "cluster-autoscaler.fullname" . }}
{{- end }}
{{- else if eq .Values.cloudProvider "azure" }}
- name: ARM_SUBSCRIPTION_ID
valueFrom:
secretKeyRef:
key: SubscriptionID
name: {{ template "cluster-autoscaler.fullname" . }}
- name: ARM_RESOURCE_GROUP
valueFrom:
secretKeyRef:
key: ResourceGroup
name: {{ template "cluster-autoscaler.fullname" . }}
- name: ARM_VM_TYPE
valueFrom:
secretKeyRef:
key: VMType
name: {{ template "cluster-autoscaler.fullname" . }}
{{- if .Values.azureUseManagedIdentityExtension }}
- name: ARM_USE_MANAGED_IDENTITY_EXTENSION
value: "true"
{{- else }}
- name: ARM_TENANT_ID
valueFrom:
secretKeyRef:
key: TenantID
name: {{ template "cluster-autoscaler.fullname" . }}
- name: ARM_CLIENT_ID
valueFrom:
secretKeyRef:
key: ClientID
name: {{ template "cluster-autoscaler.fullname" . }}
- name: ARM_CLIENT_SECRET
valueFrom:
secretKeyRef:
key: ClientSecret
name: {{ template "cluster-autoscaler.fullname" . }}
- name: AZURE_CLUSTER_NAME
valueFrom:
secretKeyRef:
key: ClusterName
name: {{ template "cluster-autoscaler.fullname" . }}
- name: AZURE_NODE_RESOURCE_GROUP
valueFrom:
secretKeyRef:
key: NodeResourceGroup
name: {{ template "cluster-autoscaler.fullname" . }}
{{- end }}
{{- end }}
{{- range $key, $value := .Values.extraEnv }}
- name: {{ $key }}
value: "{{ $value }}"
{{- end }}
{{- range $key, $value := .Values.envFromConfigMap }}
- name: {{ $key }}
valueFrom:
configMapKeyRef:
name: {{ default (include "cluster-autoscaler.fullname" $) $value.name }}
key: {{ required "Must specify key!" $value.key }}
{{- end }}
{{- range $key, $value := .Values.extraEnvSecrets }}
- name: {{ $key }}
valueFrom:
secretKeyRef:
name: {{ default (include "cluster-autoscaler.fullname" $) $value.name }}
key: {{ required "Must specify key!" $value.key }}
{{- end }}
{{- if .Values.envFromSecret }}
envFrom:
- secretRef:
name: {{ .Values.envFromSecret }}
{{- end }}
livenessProbe:
httpGet:
path: /health-check
port: 8085
ports:
- containerPort: 8085
resources:
{{ toYaml .Values.resources | indent 12 }}
{{- if .Values.containerSecurityContext }}
securityContext:
{{ toYaml .Values.containerSecurityContext | nindent 12 | trim }}
{{- end }}
{{- if eq .Values.cloudProvider "gce" }}
volumeMounts:
- name: cloudconfig
mountPath: {{ .Values.cloudConfigPath }}
readOnly: true
{{- end }}
{{- if .Values.affinity }}
affinity:
{{ toYaml .Values.affinity | indent 8 }}
{{- end }}
{{- if .Values.nodeSelector }}
nodeSelector:
{{ toYaml .Values.nodeSelector | indent 8 }}
{{- end }}
serviceAccountName: {{ template "cluster-autoscaler.serviceAccountName" . }}
tolerations:
{{ toYaml .Values.tolerations | indent 8 }}
{{- if .Values.securityContext }}
securityContext:
{{ toYaml .Values.securityContext | nindent 8 | trim }}
{{- end }}
{{- if eq .Values.cloudProvider "gce" }}
volumes:
- name: cloudconfig
hostPath:
path: {{ .Values.cloudConfigPath }}
{{- end }}
{{- if .Values.image.pullSecrets }}
imagePullSecrets:
{{- range .Values.image.pullSecrets }}
- name: {{ . }}
{{- end }}
{{- end }}
{{- end }}
15 changes: 15 additions & 0 deletions charts/cluster-autoscaler-chart/templates/pdb.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
{{- if .Values.podDisruptionBudget -}}
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
labels:
{{ include "cluster-autoscaler.labels" . | indent 4 }}
name: {{ template "cluster-autoscaler.fullname" . }}
spec:
selector:
matchLabels:
{{ include "cluster-autoscaler.instance-name" . | indent 6 }}
{{- if .Values.podDisruptionBudget }}
{{ toYaml .Values.podDisruptionBudget | nindent 2 }}
{{- end }}
{{- end -}}
46 changes: 46 additions & 0 deletions charts/cluster-autoscaler-chart/templates/podsecuritypolicy.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
{{- if .Values.rbac.pspEnabled }}
apiVersion: {{ template "podsecuritypolicy.apiVersion" . }}
kind: PodSecurityPolicy
metadata:
name: {{ template "cluster-autoscaler.fullname" . }}
labels:
{{ include "cluster-autoscaler.labels" . | indent 4 }}
spec:
# Prevents running in privileged mode
privileged: false
# Required to prevent escalations to root.
allowPrivilegeEscalation: false
requiredDropCapabilities:
- ALL
volumes:
- 'configMap'
- 'secret'
- 'hostPath'
- 'emptyDir'
- 'projected'
- 'downwardAPI'
{{- if eq .Values.cloudProvider "gce" }}
allowedHostPaths:
- pathPrefix: {{ .Values.cloudConfigPath }}
{{- end }}
hostNetwork: false
hostIPC: false
hostPID: false
runAsUser:
rule: RunAsAny
seLinux:
rule: RunAsAny
supplementalGroups:
rule: 'MustRunAs'
ranges:
# Forbid adding the root group.
- min: 1
max: 65535
fsGroup:
rule: 'MustRunAs'
ranges:
# Forbid adding the root group.
- min: 1
max: 65535
readOnlyRootFilesystem: false
{{- end }}
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
{{- if hasKey .Values.extraArgs "expander" }}
{{- if and (.Values.expanderPriorities) (eq .Values.extraArgs.expander "priority") -}}
apiVersion: v1
kind: ConfigMap
metadata:
name: cluster-autoscaler-priority-expander
labels:
{{ include "cluster-autoscaler.labels" . | indent 4 }}
data:
priorities: |-
{{ .Values.expanderPriorities | indent 4 }}
{{- end -}}
{{- end -}}
25 changes: 25 additions & 0 deletions charts/cluster-autoscaler-chart/templates/role.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
{{- if .Values.rbac.create -}}
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
labels:
{{ include "cluster-autoscaler.labels" . | indent 4 }}
name: {{ template "cluster-autoscaler.fullname" . }}
rules:
- apiGroups:
- ""
resources:
- configmaps
verbs:
- create
- apiGroups:
- ""
resources:
- configmaps
resourceNames:
- cluster-autoscaler-status
verbs:
- delete
- get
- update
{{- end -}}
16 changes: 16 additions & 0 deletions charts/cluster-autoscaler-chart/templates/rolebinding.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
{{- if .Values.rbac.create -}}
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
{{ include "cluster-autoscaler.labels" . | indent 4 }}
name: {{ template "cluster-autoscaler.fullname" . }}
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: {{ template "cluster-autoscaler.fullname" . }}
subjects:
- kind: ServiceAccount
name: {{ template "cluster-autoscaler.serviceAccountName" . }}
namespace: {{ .Release.Namespace }}
{{- end -}}
20 changes: 20 additions & 0 deletions charts/cluster-autoscaler-chart/templates/secret.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
{{- if or (eq .Values.cloudProvider "azure") (and (eq .Values.cloudProvider "aws") (not (has "" (list .Values.awsAccessKeyID .Values.awsSecretAccessKey)))) }}
apiVersion: v1
kind: Secret
metadata:
name: {{ template "cluster-autoscaler.fullname" . }}
data:
{{- if eq .Values.cloudProvider "azure" }}
ClientID: "{{ .Values.azureClientID | b64enc }}"
ClientSecret: "{{ .Values.azureClientSecret | b64enc }}"
ResourceGroup: "{{ .Values.azureResourceGroup | b64enc }}"
SubscriptionID: "{{ .Values.azureSubscriptionID | b64enc }}"
TenantID: "{{ .Values.azureTenantID | b64enc }}"
VMType: "{{ .Values.azureVMType | b64enc }}"
ClusterName: "{{ .Values.azureClusterName | b64enc }}"
NodeResourceGroup: "{{ .Values.azureNodeResourceGroup | b64enc }}"
{{- else if eq .Values.cloudProvider "aws" }}
AwsAccessKeyId: "{{ .Values.awsAccessKeyID | b64enc }}"
AwsSecretAccessKey: "{{ .Values.awsSecretAccessKey | b64enc }}"
{{- end }}
{{- end }}
36 changes: 36 additions & 0 deletions charts/cluster-autoscaler-chart/templates/service.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
apiVersion: v1
kind: Service
metadata:
{{- if .Values.service.annotations }}
annotations:
{{ toYaml .Values.service.annotations | indent 4 }}
{{- end }}
labels:
{{ include "cluster-autoscaler.labels" . | indent 4 }}
{{- if .Values.service.labels }}
{{ toYaml .Values.service.labels | indent 4 }}
{{- end }}
name: {{ template "cluster-autoscaler.fullname" . }}
spec:
{{- if .Values.service.clusterIP }}
clusterIP: "{{ .Values.service.clusterIP }}"
{{- end }}
{{- if .Values.service.externalIPs }}
externalIPs:
{{ toYaml .Values.service.externalIPs | indent 4 }}
{{- end }}
{{- if .Values.service.loadBalancerIP }}
loadBalancerIP: "{{ .Values.service.loadBalancerIP }}"
{{- end }}
{{- if .Values.service.loadBalancerSourceRanges }}
loadBalancerSourceRanges:
{{ toYaml .Values.service.loadBalancerSourceRanges | indent 4 }}
{{- end }}
ports:
- port: {{ .Values.service.servicePort }}
protocol: TCP
targetPort: 8085
name: {{ .Values.service.portName }}
selector:
{{ include "cluster-autoscaler.instance-name" . | indent 4 }}
type: "{{ .Values.service.type }}"
11 changes: 11 additions & 0 deletions charts/cluster-autoscaler-chart/templates/serviceaccount.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
{{- if and .Values.rbac.create .Values.rbac.serviceAccount.create }}
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
{{ include "cluster-autoscaler.labels" . | indent 4 }}
name: {{ template "cluster-autoscaler.serviceAccountName" . }}
{{- if .Values.rbac.serviceAccount.annotations }}
annotations: {{ toYaml .Values.rbac.serviceAccount.annotations | nindent 4 }}
{{- end }}
{{- end }}
24 changes: 24 additions & 0 deletions charts/cluster-autoscaler-chart/templates/servicemonitor.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
{{ if .Values.serviceMonitor.enabled }}
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: {{ include "cluster-autoscaler.fullname" . }}
{{- if .Values.serviceMonitor.namespace }}
namespace: {{ .Values.serviceMonitor.namespace }}
{{- end }}
labels:
{{- range $key, $value := .Values.serviceMonitor.selector }}
{{ $key }}: {{ $value | quote }}
{{- end }}
spec:
selector:
matchLabels:
{{ include "cluster-autoscaler.instance-name" . | indent 6 }}
endpoints:
- port: {{ .Values.service.portName }}
interval: {{ .Values.serviceMonitor.interval }}
path: {{ .Values.serviceMonitor.path }}
namespaceSelector:
matchNames:
- {{.Release.Namespace}}
{{ end }}
253 changes: 253 additions & 0 deletions charts/cluster-autoscaler-chart/values.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,253 @@
## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
# affinity -- Affinity for pod assignment
affinity: {}

autoDiscovery:
# Only cloudProvider `aws` and `gce` are supported by auto-discovery at this time
# AWS: Set tags as described in https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md#auto-discovery-setup
# autoDiscovery.clusterName -- Enable autodiscovery for name in ASG tag (only `cloudProvider=aws`). Must be set for `cloudProvider=gce`, but no MIG tagging required.
clusterName: # cluster.local
# autoDiscovery.tags -- ASG tags to match, run through `tpl`.
tags:
- k8s.io/cluster-autoscaler/enabled
- k8s.io/cluster-autoscaler/{{ .Values.autoDiscovery.clusterName }}
# - kubernetes.io/cluster/{{ .Values.autoDiscovery.clusterName }}

# autoscalingGroups -- For AWS. At least one element is required if not using `autoDiscovery`. For example:
# <pre>
# - name: asg1<br />
# maxSize: 2<br />
# minSize: 1
# </pre>
autoscalingGroups: []
# - name: asg1
# maxSize: 2
# minSize: 1
# - name: asg2
# maxSize: 2
# minSize: 1

# autoscalingGroupsnamePrefix -- For GCE. At least one element is required if not using `autoDiscovery`. For example:
# <pre>
# - name: ig01<br />
# maxSize: 10<br />
# minSize: 0
# </pre>
autoscalingGroupsnamePrefix: []
# - name: ig01
# maxSize: 10
# minSize: 0
# - name: ig02
# maxSize: 10
# minSize: 0

# awsAccessKeyID -- AWS access key ID ([if AWS user keys used](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md#using-aws-credentials))
awsAccessKeyID: ""

# awsRegion -- AWS region (required if `cloudProvider=aws`)
awsRegion: us-east-1

# awsSecretAccessKey -- AWS access secret key ([if AWS user keys used](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md#using-aws-credentials))
awsSecretAccessKey: ""

# azureClientID -- Service Principal ClientID with contributor permission to Cluster and Node ResourceGroup.
# Required if `cloudProvider=azure`
azureClientID: ""

# azureClientSecret -- Service Principal ClientSecret with contributor permission to Cluster and Node ResourceGroup.
# Required if `cloudProvider=azure`
azureClientSecret: ""

# azureResourceGroup -- Azure resource group that the cluster is located.
# Required if `cloudProvider=azure`
azureResourceGroup: ""

# azureSubscriptionID -- Azure subscription where the resources are located.
# Required if `cloudProvider=azure`
azureSubscriptionID: ""

# azureTenantID -- Azure tenant where the resources are located.
# Required if `cloudProvider=azure`
azureTenantID: ""

# azureVMType -- Azure VM type.
azureVMType: "AKS"

# azureClusterName -- Azure AKS cluster name.
# Required if `cloudProvider=azure`
azureClusterName: ""

# azureNodeResourceGroup -- Azure resource group where the cluster's nodes are located, typically set as `MC_<cluster-resource-group-name>_<cluster-name>_<location>`.
# Required if `cloudProvider=azure`
azureNodeResourceGroup: ""

# azureUseManagedIdentityExtension -- Whether to use Azure's managed identity extension for credentials. If using MSI, ensure subscription ID and resource group are set.
azureUseManagedIdentityExtension: false

# cloudConfigPath -- Configuration file for cloud provider.
cloudConfigPath: /etc/gce.conf

# cloudProvider -- The cloud provider where the autoscaler runs.
# Currently only `gce`, `aws`, and `azure` are supported.
# `aws` supported for AWS. `gce` for GCE. `azure` for Azure AKS.
cloudProvider: aws

# containerSecurityContext -- [Security context for container](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/)
containerSecurityContext: {}
# capabilities:
# drop:
# - ALL

# dnsPolicy -- Defaults to `ClusterFirst`. Valid values are:
# `ClusterFirstWithHostNet`, `ClusterFirst`, `Default` or `None`.
# If autoscaler does not depend on cluster DNS, recommended to set this to `Default`.
dnsPolicy: ClusterFirst

## Priorities Expander
# expanderPriorities -- The expanderPriorities is used if `extraArgs.expander` is set to `priority` and expanderPriorities is also set with the priorities.
# If `extraArgs.expander` is set to `priority`, then expanderPriorities is used to define cluster-autoscaler-priority-expander priorities.
# See: https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/expander/priority/readme.md
expanderPriorities: {}

# extraArgs -- Additional container arguments.
extraArgs:
logtostderr: true
stderrthreshold: info
v: 4
# write-status-configmap: true
# leader-elect: true
# skip-nodes-with-local-storage: false
# expander: least-waste
# scale-down-enabled: true
# balance-similar-node-groups: true
# min-replica-count: 2
# scale-down-utilization-threshold: 0.5
# scale-down-non-empty-candidates-count: 5
# max-node-provision-time: 15m0s
# scan-interval: 10s
# scale-down-delay-after-add: 10m
# scale-down-delay-after-delete: 0s
# scale-down-delay-after-failure: 3m
# scale-down-unneeded-time: 10m
# skip-nodes-with-system-pods: true

# extraEnv -- Additional container environment variables.
extraEnv: {}

# fullnameOverride -- String to fully override `cluster-autoscaler.fullname` template.
fullnameOverride: ""

image:
# image.repository -- Image repository
repository: us.gcr.io/k8s-artifacts-prod/autoscaling/cluster-autoscaler
# image.tag -- Image tag
tag: v1.18.1
# image.pullPolicy -- Image pull policy
pullPolicy: IfNotPresent
## Optionally specify an array of imagePullSecrets.
## Secrets must be manually created in the namespace.
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
##
# image.pullSecrets -- Image pull secrets
pullSecrets: []
# - myRegistrKeySecretName

# kubeTargetVersionOverride -- Allow overridding the `.Capabilities.KubeVersion.GitVersion` check. Useful for `helm template` commands.
kubeTargetVersionOverride: ""

# nameOverride -- String to partially override `cluster-autoscaler.fullname` template (will maintain the release name)
nameOverride: ""

# nodeSelector -- Node labels for pod assignment. Ref: https://kubernetes.io/docs/user-guide/node-selection/.
nodeSelector: {}

# podAnnotations -- Annotations to add to each pod.
podAnnotations: {}

# podDisruptionBudget -- Pod disruption budget.
podDisruptionBudget:
maxUnavailable: 1
# minAvailable: 2

# podLabels -- Labels to add to each pod.
podLabels: {}

# priorityClassName -- priorityClassName
priorityClassName: ""

rbac:
# rbac.create -- If `true`, create and use RBAC resources.
create: true
# rbac.pspEnabled -- If `true`, creates and uses RBAC resources required in the cluster with [Pod Security Policies](https://kubernetes.io/docs/concepts/policy/pod-security-policy/) enabled.
# Must be used with `rbac.create` set to `true`.
pspEnabled: false
serviceAccount:
# rbac.serviceAccount.annotations -- Additional Service Account annotations.
annotations: {}
# rbac.serviceAccount.create -- If `true` and `rbac.create` is also true, a Service Account will be created.
create: true
# rbac.serviceAccount.name -- The name of the ServiceAccount to use. If not set and create is `true`, a name is generated using the fullname template.
name: ""

# replicaCount -- Desired number of pods
replicaCount: 1

# resources -- Pod resource requests and limits.
resources: {}
# limits:
# cpu: 100m
# memory: 300Mi
# requests:
# cpu: 100m
# memory: 300Mi

# securityContext -- [Security context for pod](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/)
securityContext: {}
# runAsNonRoot: true
# runAsUser: 1001
# runAsGroup: 1001

service:
# service.annotations -- Annotations to add to service
annotations: {}
# service.labels -- Labels to add to service
labels: {}
# service.externalIPs -- List of IP addresses at which the service is available. Ref: https://kubernetes.io/docs/user-guide/services/#external-ips.
externalIPs: []

# service.loadBalancerIP -- IP address to assign to load balancer (if supported).
loadBalancerIP: ""
# service.loadBalancerSourceRanges -- List of IP CIDRs allowed access to load balancer (if supported).
loadBalancerSourceRanges: []
# service.servicePort -- Service port to expose.
servicePort: 8085
# service.portName -- Name for service port.
portName: http
# service.type -- Type of service to create.
type: ClusterIP

## Are you using Prometheus Operator?
serviceMonitor:
# serviceMonitor.enabled -- If true, creates a Prometheus Operator ServiceMonitor.
enabled: false
# serviceMonitor.interval -- Interval that Prometheus scrapes Cluster Autoscaler metrics.
interval: 10s
# serviceMonitor.namespace -- Namespace which Prometheus is running in.
namespace: monitoring
## [Prometheus Selector Label](https://github.com/helm/charts/tree/master/stable/prometheus-operator#prometheus-operator-1)
## [Kube Prometheus Selector Label](https://github.com/helm/charts/tree/master/stable/prometheus-operator#exporters)
# serviceMonitor.selector -- Default to kube-prometheus install (CoreOS recommended), but should be set according to Prometheus install.
selector:
release: prometheus-operator
# serviceMonitor.path -- The path to scrape for metrics; autoscaler exposes `/metrics` (this is standard)
path: /metrics

# tolerations -- List of node taints to tolerate (requires Kubernetes >= 1.6).
tolerations: []

# updateStrategy -- [Deployment update strategy](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#strategy)
updateStrategy: {}
# rollingUpdate:
# maxSurge: 1
# maxUnavailable: 0
# type: RollingUpdate

0 comments on commit 962bbd0

Please sign in to comment.