Skip to content

Commit

Permalink
Merge pull request #3550 from benmoss/capi-backports-1.19
Browse files Browse the repository at this point in the history
[CA-1.19] CAPI backports for autoscaling workload clusters
  • Loading branch information
k8s-ci-robot authored Oct 1, 2020
2 parents 550e6ef + 520b117 commit 1529a20
Show file tree
Hide file tree
Showing 33 changed files with 4,015 additions and 2,375 deletions.
80 changes: 70 additions & 10 deletions cluster-autoscaler/cloudprovider/clusterapi/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,14 +10,6 @@ cluster.
The cluster-api provider requires Kubernetes v1.16 or greater to run the
v1alpha3 version of the API.

## Cluster API Prerequisites

Please be aware that currently the cluster autoscaler only supports CAPI
clusters that have joined their management and workload clusters into a single
cluster. For more information about this please see the
[Cluster API Concepts documentations](https://cluster-api.sigs.k8s.io/user/concepts.html)
and the [`clusterctl move` command documentation](https://cluster-api.sigs.k8s.io/user/concepts.html).

## Starting the Autoscaler

To enable the Cluster API provider, you must first specify it in the command
Expand All @@ -31,6 +23,74 @@ Please note, this example only shows the cloud provider options, you will
most likely need other command line flags. For more information you can invoke
`cluster-autoscaler --help` to see a full list of options.

## Configuring node group auto discovery

If you do not configure node group auto discovery, cluster autoscaler will attempt
to match nodes against any scalable resources found in any namespace and belonging
to any Cluster.

Limiting cluster autoscaler to only match against resources in the blue namespace

```
--node-group-auto-discovery=clusterapi:namespace=blue
```

Limiting cluster autoscaler to only match against resources belonging to Cluster test1

```
--node-group-auto-discovery=clusterapi:clusterName=test1
```

Limiting cluster autoscaler to only match against resources matching the provided labels

```
--node-group-auto-discovery=clusterapi:color=green,shape=square
```

These can be mixed and matched in any combination, for example to only match resources
in the staging namespace, belonging to the purple cluster, with the label owner=jim:

```
--node-group-auto-discovery=clusterapi:namespace=staging,clusterName=purple,owner=jim
```

## Connecting cluster-autoscaler to Cluster API management and workload Clusters

You will also need to provide the path to the kubeconfig(s) for the management
and workload cluster you wish cluster-autoscaler to run against. To specify the
kubeconfig path for the workload cluster to monitor, use the `--kubeconfig`
option and supply the path to the kubeconfig. If the `--kubeconfig` option is
not specified, cluster-autoscaler will attempt to use an in-cluster configuration.
To specify the kubeconfig path for the management cluster to monitor, use the
`--cloud-config` option and supply the path to the kubeconfig. If the
`--cloud-config` option is not specified it will fall back to using the kubeconfig
that was provided with the `--kubeconfig` option.

Use in-cluster config for both management and workload cluster:
```
cluster-autoscaler --cloud-provider=clusterapi
```

Use in-cluster config for workload cluster, specify kubeconfig for management cluster:
```
cluster-autoscaler --cloud-provider=clusterapi --cloud-config=/mnt/kubeconfig
```

Use in-cluster config for management cluster, specify kubeconfig for workload cluster:
```
cluster-autoscaler --cloud-provider=clusterapi --kubeconfig=/mnt/kubeconfig --clusterapi-cloud-config-authoritative
```

Use separate kubeconfigs for both management and workload cluster:
```
cluster-autoscaler --cloud-provider=clusterapi --kubeconfig=/mnt/workload.kubeconfig --cloud-config=/mnt/management.kubeconfig
```

Use a single provided kubeconfig for both management and workload cluster:
```
cluster-autoscaler --cloud-provider=clusterapi --kubeconfig=/mnt/workload.kubeconfig
```

## Enabling Autoscaling

To enable the automatic scaling of components in your cluster-api managed
Expand All @@ -41,12 +101,12 @@ resources depending on the type of cluster-api mechanism that you are using.

There are two annotations that control how a cluster resource should be scaled:

* `cluster.k8s.io/cluster-api-autoscaler-node-group-min-size` - This specifies
* `cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size` - This specifies
the minimum number of nodes for the associated resource group. The autoscaler
will not scale the group below this number. Please note that currently the
cluster-api provider will not scale down to zero nodes.

* `cluster.k8s.io/cluster-api-autoscaler-node-group-max-size` - This specifies
* `cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size` - This specifies
the maximum number of nodes for the associated resource group. The autoscaler
will not scale the group above this number.

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
/*
Copyright 2020 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package clusterapi

import (
"fmt"
"strings"

"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
"k8s.io/apimachinery/pkg/labels"
"k8s.io/apimachinery/pkg/selection"

"k8s.io/autoscaler/cluster-autoscaler/utils/errors"
)

type clusterAPIAutoDiscoveryConfig struct {
clusterName string
namespace string
labelSelector labels.Selector
}

func parseAutoDiscoverySpec(spec string) (*clusterAPIAutoDiscoveryConfig, error) {
cfg := &clusterAPIAutoDiscoveryConfig{
labelSelector: labels.NewSelector(),
}

tokens := strings.Split(spec, ":")
if len(tokens) != 2 {
return cfg, errors.NewAutoscalerError(errors.ConfigurationError, fmt.Sprintf("spec \"%s\" should be discoverer:key=value,key=value", spec))
}
discoverer := tokens[0]
if discoverer != autoDiscovererTypeClusterAPI {
return cfg, errors.NewAutoscalerError(errors.ConfigurationError, fmt.Sprintf("unsupported discoverer specified: %s", discoverer))
}

for _, arg := range strings.Split(tokens[1], ",") {
if len(arg) == 0 {
continue
}
kv := strings.Split(arg, "=")
if len(kv) != 2 {
return cfg, errors.NewAutoscalerError(errors.ConfigurationError, fmt.Sprintf("invalid key=value pair %s", kv))
}
k, v := kv[0], kv[1]

switch k {
case autoDiscovererClusterNameKey:
cfg.clusterName = v
case autoDiscovererNamespaceKey:
cfg.namespace = v
default:
req, err := labels.NewRequirement(k, selection.Equals, []string{v})
if err != nil {
return cfg, errors.NewAutoscalerError(errors.ConfigurationError, fmt.Sprintf("failed to create label selector; %v", err))
}
cfg.labelSelector = cfg.labelSelector.Add(*req)
}
}
return cfg, nil
}

func parseAutoDiscovery(specs []string) ([]*clusterAPIAutoDiscoveryConfig, error) {
result := make([]*clusterAPIAutoDiscoveryConfig, 0, len(specs))
for _, spec := range specs {
autoDiscoverySpec, err := parseAutoDiscoverySpec(spec)
if err != nil {
return result, err
}
result = append(result, autoDiscoverySpec)
}
return result, nil
}

func allowedByAutoDiscoverySpec(spec *clusterAPIAutoDiscoveryConfig, r *unstructured.Unstructured) bool {
switch {
case spec.namespace != "" && spec.namespace != r.GetNamespace():
return false
case spec.clusterName != "" && spec.clusterName != clusterNameFromResource(r):
return false
case !spec.labelSelector.Matches(labels.Set(r.GetLabels())):
return false
default:
return true
}
}
Loading

0 comments on commit 1529a20

Please sign in to comment.