Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[aws] minor documentation fixes #2482

Merged
merged 2 commits into from
Oct 25, 2019
Merged

[aws] minor documentation fixes #2482

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
873ce0f
[aws] correct placement of static instance docs
jaypipes Oct 24, 2019
804a948
[aws] clean up docs around IAM policy
jaypipes Oct 24, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 20 additions & 36 deletions cluster-autoscaler/cloudprovider/aws/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Cluster Autoscaler on AWS
The cluster autoscaler on AWS scales worker nodes within any specified autoscaling group. It will run as a `Deployment` in your cluster. This README will go over some of the necessary steps required to get the cluster autoscaler up and running.
The cluster autoscaler on AWS scales worker nodes within any specified autoscaling group. It will run as a `Deployment` in your cluster. This README covers the steps required to configure and run the cluster autoscaler.

## Kubernetes Version
Cluster autoscaler must run on v1.3.0 or greater.
Expand Down Expand Up @@ -27,35 +27,23 @@ A minimum IAM policy would look like:
]
}
```
If you'd like to scale node groups from 0, an
`autoscaling:DescribeLaunchConfigurations` or
`ec2:DescribeLaunchTemplateVersions` permission is required depending on if you
made your ASG with Launch Configuration or Launch Template.

If you'd like to auto-discover node groups by specifying the `--node-group-auto-discovery` flag, a `DescribeTags` permission is also required:
If you'd like the cluster autoscaler to [automatically
discover](#auto-discovery-setup) EC2 AutoScalingGroups, the
`autoscaling:DescribeTags` permission is also required.

```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"autoscaling:DescribeAutoScalingGroups",
"autoscaling:DescribeAutoScalingInstances",
"autoscaling:DescribeLaunchConfigurations",
"autoscaling:DescribeTags",
"autoscaling:SetDesiredCapacity",
"autoscaling:TerminateInstanceInAutoScalingGroup"
],
"Resource": "*"
}
]
}
```

AWS supports ARNs for autoscaling groups. More information [here](https://docs.aws.amazon.com/autoscaling/latest/userguide/control-access-using-iam.html#policy-auto-scaling-resources).
**NOTE**: You can restrict the target resources for the autoscaling actions by
specifying autoscaling group ARNS. More information can be found
[here](https://docs.aws.amazon.com/autoscaling/latest/userguide/control-access-using-iam.html#policy-auto-scaling-resources).

### Using AWS Credentials
For on premise users like to scale out to AWS, above approach that attaching policy to nodegroup role won't work. Instead, you can create an aws secret manually and add following environment variables to cluster-autoscaler deployment manifest. Cluster autoscaler will use credential to authenticate and authorize itself. Please make sure your role has above permissions.
For on premise users wishing to scale out to AWS, the above approach of attaching policy to a nodegroup role won't work. Instead, you can create an aws secret manually and add following environment variables to cluster-autoscaler deployment manifest. Cluster autoscaler will use credential to authenticate and authorize itself. Please make sure your role has above permissions.

```
```yaml
apiVersion: v1
kind: Secret
metadata:
Expand All @@ -64,11 +52,10 @@ type: Opaque
data:
aws_access_key_id: BASE64_OF_YOUR_AWS_ACCESS_KEY_ID
aws_secret_access_key: BASE64_OF_YOUR_AWS_SECRET_ACCESS_KEY

```
Please check [guidance](https://kubernetes.io/docs/concepts/configuration/secret/#creating-a-secret-manually) for creating a secret manually.

```
```yaml
env:
- name: AWS_ACCESS_KEY_ID
valueFrom:
Expand All @@ -82,7 +69,6 @@ env:
key: aws_secret_access_key
- name: AWS_REGION
value: YOUR_AWS_REGION

```

## Deployment Specification
Expand All @@ -108,7 +94,6 @@ Please replace `{{ node_asg_min }}`, `{{ node_asg_max }}` and `{{ name }}` with
kubectl apply -f examples/cluster-autoscaler-run-on-master.yaml
```


### Auto-Discovery Setup

To run a cluster-autoscaler which auto-discovers ASGs with nodes use the `--node-group-auto-discovery` flag. For example, `--node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/<YOUR CLUSTER NAME>` will find the ASGs where those tag keys
Expand Down Expand Up @@ -204,13 +189,6 @@ spec:
- r5ad.2xlarge
```

## Use Static Instance List
The set of the latest supported EC2 instance types will be fetched by the CA at run time. You can find all the available instance types in the CA logs.
If your network access is restricted such that fetching this set is infeasible, you can specify the command-line flag `--aws-use-static-instance-list=true` to switch the CA back to its original use of a statically defined set.

To refresh static list, please run `go run ec2_instance_types/gen.go` under `cluster-autoscaler/cloudprovider/aws/` and update `staticListLastUpdateTime` in `aws_util.go`


### Example usage:

* Create a [Launch Template](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-autoscaling-autoscalinggroup-launchtemplate.html) (LT) with an instance type, for example, r5.2xlarge. Consider this the 'base' instance type. Do not define any spot purchase options here.
Expand All @@ -222,6 +200,12 @@ To refresh static list, please run `go run ec2_instance_types/gen.go` under `clu

See CloudFormation example [here](MixedInstancePolicy.md).

## Use Static Instance List
The set of the latest supported EC2 instance types will be fetched by the CA at run time. You can find all the available instance types in the CA logs.
If your network access is restricted such that fetching this set is infeasible, you can specify the command-line flag `--aws-use-static-instance-list=true` to switch the CA back to its original use of a statically defined set.

To refresh static list, please run `go run ec2_instance_types/gen.go` under `cluster-autoscaler/cloudprovider/aws/` and update `staticListLastUpdateTime` in `aws_util.go`

## Common Notes and Gotchas:
- The `/etc/ssl/certs/ca-bundle.crt` should exist by default on ec2 instance in your EKS cluster. If you use other cluster privision tools like [kops](https://github.com/kubernetes/kops) with different operating systems other than Amazon Linux 2, please use `/etc/ssl/certs/ca-certificates.crt` or correct path on your host instead for the volume hostPath in your cluster autoscaler manifest.
- Cluster autoscaler does not support Auto Scaling Groups which span multiple Availability Zones; instead you should use an Auto Scaling Group for each Availability Zone and enable the [--balance-similar-node-groups](../../FAQ.md#im-running-cluster-with-nodes-in-multiple-zones-for-ha-purposes-is-that-supported-by-cluster-autoscaler) feature. If you do use a single Auto Scaling Group that spans multiple Availability Zones you will find that AWS unexpectedly terminates nodes without them being drained because of the [rebalancing feature](https://docs.aws.amazon.com/autoscaling/ec2/userguide/auto-scaling-benefits.html#arch-AutoScalingMultiAZ).
Expand Down