From 82162e555b059094c8ddd67e93bbd1f69279b351 Mon Sep 17 00:00:00 2001 From: Sam Weston Date: Mon, 4 Mar 2019 11:31:16 +0000 Subject: [PATCH 1/5] Improve AWS gotchas section and make cloud providers READMEs more discoverable #1744 Signed-off-by: Sam Weston --- cluster-autoscaler/README.md | 8 +++++++- cluster-autoscaler/cloudprovider/aws/README.md | 4 ++-- 2 files changed, 9 insertions(+), 3 deletions(-) diff --git a/cluster-autoscaler/README.md b/cluster-autoscaler/README.md index 7b492ebdc65..901754489e2 100644 --- a/cluster-autoscaler/README.md +++ b/cluster-autoscaler/README.md @@ -9,7 +9,13 @@ Cluster Autoscaler is a tool that automatically adjusts the size of the Kubernet # FAQ/Documentation -Is available [HERE](./FAQ.md). +An FAQ is available [HERE](./FAQ.md). + +You should also take a look at the notes and "gotchas" for your specific cloud provider: +* [AliCloud](./cloudprovider/alicloud/README.md) +* [Azure](./cloudprovider/azure/README.md) +* [AWS](./cloudprovider/aws/README.md) +* [BaiduCloud](./cloudprovider/baiducloud/README.md) # Releases diff --git a/cluster-autoscaler/cloudprovider/aws/README.md b/cluster-autoscaler/cloudprovider/aws/README.md index 1b296bee810..3251b278810 100644 --- a/cluster-autoscaler/cloudprovider/aws/README.md +++ b/cluster-autoscaler/cloudprovider/aws/README.md @@ -142,8 +142,8 @@ If you'd like to scale node groups from 0, an `autoscaling:DescribeLaunchConfigu ``` ## Common Notes and Gotchas: -- The `/etc/ssl/certs/ca-certificates.crt` should exist by default on your ec2 instance. If you use Amazon Linux 2 (EKS woker node AMI by default), use `/etc/ssl/certs/ca-bundle.crt` instead. -- Cluster autoscaler is not zone aware (for now), so if you wish to span multiple availability zones in your autoscaling groups beware that cluster autoscaler will not evenly distribute them. For more information, see https://github.com/kubernetes/contrib/pull/1552#discussion_r75532949. +- The `/etc/ssl/certs/ca-certificates.crt` should exist by default on your ec2 instance. If you use Amazon Linux 2 (EKS worker node AMI by default), use `/etc/kubernetes/pki/ca.crt` instead for the volume hostPath of your cluster-autoscaler manifest. +- Cluster autoscaler is not zone aware (for now), so if you wish to span multiple availability zones in your autoscaling groups beware that cluster autoscaler will not evenly distribute them. This is not a problem for all use cases, as AWS will try to evenly distribute the nodes in ASG on scale up, but you should ["Suspend" the "AZRebalance" process](https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-suspend-resume-processes.html) if you want to avoid AWS unexpectedly terminating your nodes to try and keep your ASG balanced. More information on how this works under the hood in the cluster autoscaler is [available here](https://github.com/kubernetes/contrib/pull/1552#discussion_r75532949). - By default, cluster autoscaler will not terminate nodes running pods in the kube-system namespace. You can override this default behaviour by passing in the `--skip-nodes-with-system-pods=false` flag. - By default, cluster autoscaler will wait 10 minutes between scale down operations, you can adjust this using the `--scale-down-delay-after-add`, `--scale-down-delay-after-delete`, and `--scale-down-delay-after-failure` flag. E.g. `--scale-down-delay-after-add=5m` to decrease the scale down delay to 5 minutes after a node has been added. - If you're running multiple ASGs, the `--expander` flag supports three options: `random`, `most-pods` and `least-waste`. `random` will expand a random ASG on scale up. `most-pods` will scale up the ASG that will schedule the most amount of pods. `least-waste` will expand the ASG that will waste the least amount of CPU/MEM resources. In the event of a tie, cluster autoscaler will fall back to `random`. From e37deb09903c2464c15e47d1854e25c5ae65cdec Mon Sep 17 00:00:00 2001 From: Sam Weston Date: Mon, 4 Mar 2019 11:37:20 +0000 Subject: [PATCH 2/5] Reformat multi-az gotcha and add link to main FAQ Signed-off-by: Sam Weston --- cluster-autoscaler/cloudprovider/aws/README.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/cluster-autoscaler/cloudprovider/aws/README.md b/cluster-autoscaler/cloudprovider/aws/README.md index 3251b278810..881133f003b 100644 --- a/cluster-autoscaler/cloudprovider/aws/README.md +++ b/cluster-autoscaler/cloudprovider/aws/README.md @@ -143,7 +143,10 @@ If you'd like to scale node groups from 0, an `autoscaling:DescribeLaunchConfigu ## Common Notes and Gotchas: - The `/etc/ssl/certs/ca-certificates.crt` should exist by default on your ec2 instance. If you use Amazon Linux 2 (EKS worker node AMI by default), use `/etc/kubernetes/pki/ca.crt` instead for the volume hostPath of your cluster-autoscaler manifest. -- Cluster autoscaler is not zone aware (for now), so if you wish to span multiple availability zones in your autoscaling groups beware that cluster autoscaler will not evenly distribute them. This is not a problem for all use cases, as AWS will try to evenly distribute the nodes in ASG on scale up, but you should ["Suspend" the "AZRebalance" process](https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-suspend-resume-processes.html) if you want to avoid AWS unexpectedly terminating your nodes to try and keep your ASG balanced. More information on how this works under the hood in the cluster autoscaler is [available here](https://github.com/kubernetes/contrib/pull/1552#discussion_r75532949). +- Cluster autoscaler is not zone aware (for now), so if you wish to span multiple availability zones in your autoscaling groups beware that cluster autoscaler will not evenly distribute them. + * This is not a problem for all use cases, as AWS will try to evenly distribute the nodes in your ASG on scale up, but you should ["Suspend" the "AZRebalance" process](https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-suspend-resume-processes.html) if you want to avoid AWS unexpectedly terminating your nodes to try and keep your ASG balanced. + * More information on how this works under the hood in the cluster autoscaler is [available here](https://github.com/kubernetes/contrib/pull/1552#discussion_r75532949). + * You may also want to look at having multiple autoscaling groups (one for each availability zone) and using [--balance-similar-node-groups](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#im-running-cluster-with-nodes-in-multiple-zones-for-ha-purposes-is-that-supported-by-cluster-autoscaler). - By default, cluster autoscaler will not terminate nodes running pods in the kube-system namespace. You can override this default behaviour by passing in the `--skip-nodes-with-system-pods=false` flag. - By default, cluster autoscaler will wait 10 minutes between scale down operations, you can adjust this using the `--scale-down-delay-after-add`, `--scale-down-delay-after-delete`, and `--scale-down-delay-after-failure` flag. E.g. `--scale-down-delay-after-add=5m` to decrease the scale down delay to 5 minutes after a node has been added. - If you're running multiple ASGs, the `--expander` flag supports three options: `random`, `most-pods` and `least-waste`. `random` will expand a random ASG on scale up. `most-pods` will scale up the ASG that will schedule the most amount of pods. `least-waste` will expand the ASG that will waste the least amount of CPU/MEM resources. In the event of a tie, cluster autoscaler will fall back to `random`. From 1a2010f85b32e493b478e536d598f587d1709077 Mon Sep 17 00:00:00 2001 From: Sam Weston Date: Mon, 4 Mar 2019 11:38:45 +0000 Subject: [PATCH 3/5] Make link to FAQ relative Signed-off-by: Sam Weston --- cluster-autoscaler/cloudprovider/aws/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cluster-autoscaler/cloudprovider/aws/README.md b/cluster-autoscaler/cloudprovider/aws/README.md index 881133f003b..245c5750d93 100644 --- a/cluster-autoscaler/cloudprovider/aws/README.md +++ b/cluster-autoscaler/cloudprovider/aws/README.md @@ -146,7 +146,7 @@ If you'd like to scale node groups from 0, an `autoscaling:DescribeLaunchConfigu - Cluster autoscaler is not zone aware (for now), so if you wish to span multiple availability zones in your autoscaling groups beware that cluster autoscaler will not evenly distribute them. * This is not a problem for all use cases, as AWS will try to evenly distribute the nodes in your ASG on scale up, but you should ["Suspend" the "AZRebalance" process](https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-suspend-resume-processes.html) if you want to avoid AWS unexpectedly terminating your nodes to try and keep your ASG balanced. * More information on how this works under the hood in the cluster autoscaler is [available here](https://github.com/kubernetes/contrib/pull/1552#discussion_r75532949). - * You may also want to look at having multiple autoscaling groups (one for each availability zone) and using [--balance-similar-node-groups](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#im-running-cluster-with-nodes-in-multiple-zones-for-ha-purposes-is-that-supported-by-cluster-autoscaler). + * You may also want to look at having multiple autoscaling groups (one for each availability zone) and using [--balance-similar-node-groups](../../FAQ.md#im-running-cluster-with-nodes-in-multiple-zones-for-ha-purposes-is-that-supported-by-cluster-autoscaler). - By default, cluster autoscaler will not terminate nodes running pods in the kube-system namespace. You can override this default behaviour by passing in the `--skip-nodes-with-system-pods=false` flag. - By default, cluster autoscaler will wait 10 minutes between scale down operations, you can adjust this using the `--scale-down-delay-after-add`, `--scale-down-delay-after-delete`, and `--scale-down-delay-after-failure` flag. E.g. `--scale-down-delay-after-add=5m` to decrease the scale down delay to 5 minutes after a node has been added. - If you're running multiple ASGs, the `--expander` flag supports three options: `random`, `most-pods` and `least-waste`. `random` will expand a random ASG on scale up. `most-pods` will scale up the ASG that will schedule the most amount of pods. `least-waste` will expand the ASG that will waste the least amount of CPU/MEM resources. In the event of a tie, cluster autoscaler will fall back to `random`. From f61c8d0ba43a0241ac57dd8046e1661293979248 Mon Sep 17 00:00:00 2001 From: Sam Weston Date: Tue, 5 Mar 2019 09:56:04 +0000 Subject: [PATCH 4/5] Update multi-az gotcha to represent recommended practice Signed-off-by: Sam Weston --- cluster-autoscaler/cloudprovider/aws/README.md | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/cluster-autoscaler/cloudprovider/aws/README.md b/cluster-autoscaler/cloudprovider/aws/README.md index 245c5750d93..97eef77e035 100644 --- a/cluster-autoscaler/cloudprovider/aws/README.md +++ b/cluster-autoscaler/cloudprovider/aws/README.md @@ -143,10 +143,7 @@ If you'd like to scale node groups from 0, an `autoscaling:DescribeLaunchConfigu ## Common Notes and Gotchas: - The `/etc/ssl/certs/ca-certificates.crt` should exist by default on your ec2 instance. If you use Amazon Linux 2 (EKS worker node AMI by default), use `/etc/kubernetes/pki/ca.crt` instead for the volume hostPath of your cluster-autoscaler manifest. -- Cluster autoscaler is not zone aware (for now), so if you wish to span multiple availability zones in your autoscaling groups beware that cluster autoscaler will not evenly distribute them. - * This is not a problem for all use cases, as AWS will try to evenly distribute the nodes in your ASG on scale up, but you should ["Suspend" the "AZRebalance" process](https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-suspend-resume-processes.html) if you want to avoid AWS unexpectedly terminating your nodes to try and keep your ASG balanced. - * More information on how this works under the hood in the cluster autoscaler is [available here](https://github.com/kubernetes/contrib/pull/1552#discussion_r75532949). - * You may also want to look at having multiple autoscaling groups (one for each availability zone) and using [--balance-similar-node-groups](../../FAQ.md#im-running-cluster-with-nodes-in-multiple-zones-for-ha-purposes-is-that-supported-by-cluster-autoscaler). +- Cluster autoscaler does not support Auto Scaling Groups which span multiple Availability Zones; instead you should use an Auto Scaling Group for each Availability Zone and enable the [--balance-similar-node-groups](../../FAQ.md#im-running-cluster-with-nodes-in-multiple-zones-for-ha-purposes-is-that-supported-by-cluster-autoscaler) feature. If you do use a single Auto Scaling Group that spans multiple Availability Zones you will find that AWS unexpectedly terminates nodes without them being drained because of the [rebalancing feature](https://docs.aws.amazon.com/autoscaling/ec2/userguide/auto-scaling-benefits.html#arch-AutoScalingMultiAZ). - By default, cluster autoscaler will not terminate nodes running pods in the kube-system namespace. You can override this default behaviour by passing in the `--skip-nodes-with-system-pods=false` flag. - By default, cluster autoscaler will wait 10 minutes between scale down operations, you can adjust this using the `--scale-down-delay-after-add`, `--scale-down-delay-after-delete`, and `--scale-down-delay-after-failure` flag. E.g. `--scale-down-delay-after-add=5m` to decrease the scale down delay to 5 minutes after a node has been added. - If you're running multiple ASGs, the `--expander` flag supports three options: `random`, `most-pods` and `least-waste`. `random` will expand a random ASG on scale up. `most-pods` will scale up the ASG that will schedule the most amount of pods. `least-waste` will expand the ASG that will waste the least amount of CPU/MEM resources. In the event of a tie, cluster autoscaler will fall back to `random`. From 5d2d3f2407528907234f0efdd49ddb8f54a38bd9 Mon Sep 17 00:00:00 2001 From: Sam Weston Date: Tue, 5 Mar 2019 13:50:34 +0000 Subject: [PATCH 5/5] Fix bad english Signed-off-by: Sam Weston --- cluster-autoscaler/cloudprovider/aws/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cluster-autoscaler/cloudprovider/aws/README.md b/cluster-autoscaler/cloudprovider/aws/README.md index 97eef77e035..6950796f554 100644 --- a/cluster-autoscaler/cloudprovider/aws/README.md +++ b/cluster-autoscaler/cloudprovider/aws/README.md @@ -142,7 +142,7 @@ If you'd like to scale node groups from 0, an `autoscaling:DescribeLaunchConfigu ``` ## Common Notes and Gotchas: -- The `/etc/ssl/certs/ca-certificates.crt` should exist by default on your ec2 instance. If you use Amazon Linux 2 (EKS worker node AMI by default), use `/etc/kubernetes/pki/ca.crt` instead for the volume hostPath of your cluster-autoscaler manifest. +- The `/etc/ssl/certs/ca-certificates.crt` should exist by default on your ec2 instance. If you use Amazon Linux 2 (EKS worker node AMI by default), use `/etc/kubernetes/pki/ca.crt` instead for the volume hostPath in your cluster autoscaler manifest. - Cluster autoscaler does not support Auto Scaling Groups which span multiple Availability Zones; instead you should use an Auto Scaling Group for each Availability Zone and enable the [--balance-similar-node-groups](../../FAQ.md#im-running-cluster-with-nodes-in-multiple-zones-for-ha-purposes-is-that-supported-by-cluster-autoscaler) feature. If you do use a single Auto Scaling Group that spans multiple Availability Zones you will find that AWS unexpectedly terminates nodes without them being drained because of the [rebalancing feature](https://docs.aws.amazon.com/autoscaling/ec2/userguide/auto-scaling-benefits.html#arch-AutoScalingMultiAZ). - By default, cluster autoscaler will not terminate nodes running pods in the kube-system namespace. You can override this default behaviour by passing in the `--skip-nodes-with-system-pods=false` flag. - By default, cluster autoscaler will wait 10 minutes between scale down operations, you can adjust this using the `--scale-down-delay-after-add`, `--scale-down-delay-after-delete`, and `--scale-down-delay-after-failure` flag. E.g. `--scale-down-delay-after-add=5m` to decrease the scale down delay to 5 minutes after a node has been added.