diff --git a/.github/dependabot.yml b/.github/dependabot.yml index f4d62b23de36..d3cb083c110e 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -28,3 +28,9 @@ updates: open-pull-requests-limit: 3 labels: - "vertical-pod-autoscaler" +- package-ecosystem: gomod + directory: "/addon-resizer" + schedule: + interval: daily + target-branch: "addon-resizer-release-1.8" + open-pull-requests-limit: 3 diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml index 1d86e5b5ae7f..8b97185f8c9e 100644 --- a/.github/workflows/ci.yaml +++ b/.github/workflows/ci.yaml @@ -17,7 +17,7 @@ jobs: - name: Set up Go uses: actions/setup-go@v2 with: - go-version: '1.21.6' + go-version: '1.22.1' - uses: actions/checkout@v2 with: diff --git a/addon-resizer/OWNERS b/addon-resizer/OWNERS index 69418fadfaaa..0ab0406b2913 100644 --- a/addon-resizer/OWNERS +++ b/addon-resizer/OWNERS @@ -8,4 +8,4 @@ emeritus_approvers: - bskiba # 2022-09-30 - wojtek-t # 2022-09-30 labels: -- addon-resizer +- area/addon-resizer diff --git a/balancer/OWNERS b/balancer/OWNERS index e09e0ee75e8c..cafbafa19048 100644 --- a/balancer/OWNERS +++ b/balancer/OWNERS @@ -5,4 +5,4 @@ reviewers: - mwielgus - kgolab labels: -- balancer +- area/balancer diff --git a/builder/Dockerfile b/builder/Dockerfile index ad1381de3777..243f6250ab81 100644 --- a/builder/Dockerfile +++ b/builder/Dockerfile @@ -12,7 +12,6 @@ # See the License for the specific language governing permissions and # limitations under the License. - FROM golang:1.22.2 LABEL maintainer="Marcin Wielgus " @@ -22,6 +21,6 @@ ENV GO111MODULE auto RUN apt-get update && apt-get --yes install libseccomp-dev RUN go version -RUN go get github.com/tools/godep +RUN go install github.com/tools/godep@latest RUN godep version CMD ["/bin/bash"] diff --git a/charts/OWNERS b/charts/OWNERS index 9e777b5c8593..8a0cd4a0ce2e 100644 --- a/charts/OWNERS +++ b/charts/OWNERS @@ -4,4 +4,4 @@ reviewers: - gjtempleton labels: -- helm-charts +- area/helm-charts diff --git a/charts/cluster-autoscaler/Chart.yaml b/charts/cluster-autoscaler/Chart.yaml index 504eea76d3a8..e58e4b197ba3 100644 --- a/charts/cluster-autoscaler/Chart.yaml +++ b/charts/cluster-autoscaler/Chart.yaml @@ -1,5 +1,5 @@ apiVersion: v2 -appVersion: 1.28.2 +appVersion: 1.29.0 description: Scales Kubernetes worker nodes within autoscaling groups. engine: gotpl home: https://github.com/kubernetes/autoscaler @@ -11,4 +11,4 @@ name: cluster-autoscaler sources: - https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler type: application -version: 9.34.1 +version: 9.36.0 diff --git a/charts/cluster-autoscaler/README.md b/charts/cluster-autoscaler/README.md index 2d13c3b0a44f..a0cc5587a6c7 100644 --- a/charts/cluster-autoscaler/README.md +++ b/charts/cluster-autoscaler/README.md @@ -73,6 +73,8 @@ To create a valid configuration, follow instructions for your cloud provider: - [Azure](#azure) - [OpenStack Magnum](#openstack-magnum) - [Cluster API](#cluster-api) +- [Exoscale](#exoscale) +- [Hetzner Cloud](#hetzner-cloud) ### Templating the autoDiscovery.clusterName @@ -222,6 +224,7 @@ $ helm install my-release autoscaler/cluster-autoscaler -f myvalues.yaml `cloudProvider: clusterapi` must be set, and then one or more of - `autoDiscovery.clusterName` +- or `autoDiscovery.namespace` - or `autoDiscovery.labels` See [here](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/clusterapi/README.md#configuring-node-group-auto-discovery) for more details. @@ -259,6 +262,18 @@ $ helm install my-release autoscaler/cluster-autoscaler \ Read [cluster-autoscaler/cloudprovider/exoscale/README.md](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/exoscale/README.md) for further information on the setup without helm. +### Hetzner Cloud + +The following parameters are required: + +- `cloudProvider=hetzner` +- `extraEnv.HCLOUD_TOKEN=...` +- `autoscalingGroups=...` + +Each autoscaling group requires an additional `instanceType` and `region` key to be set. + +Read [cluster-autoscaler/cloudprovider/hetzner/README.md](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/hetzner/README.md) for further information on the setup without helm. + ## Uninstalling the Chart To uninstall `my-release`: @@ -373,17 +388,17 @@ vpa: | affinity | object | `{}` | Affinity for pod assignment | | autoDiscovery.clusterName | string | `nil` | Enable autodiscovery for `cloudProvider=aws`, for groups matching `autoDiscovery.tags`. autoDiscovery.clusterName -- Enable autodiscovery for `cloudProvider=azure`, using tags defined in https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/azure/README.md#auto-discovery-setup. Enable autodiscovery for `cloudProvider=clusterapi`, for groups matching `autoDiscovery.labels`. Enable autodiscovery for `cloudProvider=gce`, but no MIG tagging required. Enable autodiscovery for `cloudProvider=magnum`, for groups matching `autoDiscovery.roles`. | | autoDiscovery.labels | list | `[]` | Cluster-API labels to match https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/clusterapi/README.md#configuring-node-group-auto-discovery | +| autoDiscovery.namespace | string | `nil` | Enable autodiscovery via cluster namespace for for `cloudProvider=clusterapi` | | autoDiscovery.roles | list | `["worker"]` | Magnum node group roles to match. | | autoDiscovery.tags | list | `["k8s.io/cluster-autoscaler/enabled","k8s.io/cluster-autoscaler/{{ .Values.autoDiscovery.clusterName }}"]` | ASG tags to match, run through `tpl`. | -| autoscalingGroups | list | `[]` | For AWS, Azure AKS or Magnum. At least one element is required if not using `autoDiscovery`. For example: 
 - name: asg1
   maxSize: 2
   minSize: 1
| +| autoscalingGroups | list | `[]` | For AWS, Azure AKS or Magnum. At least one element is required if not using `autoDiscovery`. For example: 
 - name: asg1
   maxSize: 2
   minSize: 1
For Hetzner Cloud, the `instanceType` and `region` keys are also required. 
 - name: mypool
   maxSize: 2
   minSize: 1
   instanceType: CPX21
   region: FSN1
| | autoscalingGroupsnamePrefix | list | `[]` | For GCE. At least one element is required if not using `autoDiscovery`. For example: 
 - name: ig01
   maxSize: 10
   minSize: 0
| | awsAccessKeyID | string | `""` | AWS access key ID ([if AWS user keys used](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md#using-aws-credentials)) | | awsRegion | string | `"us-east-1"` | AWS region (required if `cloudProvider=aws`) | | awsSecretAccessKey | string | `""` | AWS access secret key ([if AWS user keys used](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md#using-aws-credentials)) | | azureClientID | string | `""` | Service Principal ClientID with contributor permission to Cluster and Node ResourceGroup. Required if `cloudProvider=azure` | | azureClientSecret | string | `""` | Service Principal ClientSecret with contributor permission to Cluster and Node ResourceGroup. Required if `cloudProvider=azure` | -| azureClusterName | string | `""` | Azure AKS cluster name. Required if `cloudProvider=azure` | -| azureNodeResourceGroup | string | `""` | Azure resource group where the cluster's nodes are located, typically set as `MC___`. Required if `cloudProvider=azure` | +| azureEnableForceDelete | bool | `false` | Whether to force delete VMs or VMSS instances when scaling down. | | azureResourceGroup | string | `""` | Azure resource group that the cluster is located. Required if `cloudProvider=azure` | | azureSubscriptionID | string | `""` | Azure subscription where the resources are located. Required if `cloudProvider=azure` | | azureTenantID | string | `""` | Azure tenant where the resources are located. Required if `cloudProvider=azure` | @@ -415,7 +430,7 @@ vpa: | image.pullPolicy | string | `"IfNotPresent"` | Image pull policy | | image.pullSecrets | list | `[]` | Image pull secrets | | image.repository | string | `"registry.k8s.io/autoscaling/cluster-autoscaler"` | Image repository | -| image.tag | string | `"v1.28.2"` | Image tag | +| image.tag | string | `"v1.29.0"` | Image tag | | kubeTargetVersionOverride | string | `""` | Allow overriding the `.Capabilities.KubeVersion.GitVersion` check. Useful for `helm template` commands. | | kwokConfigMapName | string | `"kwok-provider-config"` | configmap for configuring kwok provider | | magnumCABundlePath | string | `"/etc/kubernetes/ca-bundle.crt"` | Path to the host's CA bundle, from `ca-file` in the cloud-config file. | diff --git a/charts/cluster-autoscaler/README.md.gotmpl b/charts/cluster-autoscaler/README.md.gotmpl index c91d0d572447..a680369ddf40 100644 --- a/charts/cluster-autoscaler/README.md.gotmpl +++ b/charts/cluster-autoscaler/README.md.gotmpl @@ -73,6 +73,8 @@ To create a valid configuration, follow instructions for your cloud provider: - [Azure](#azure) - [OpenStack Magnum](#openstack-magnum) - [Cluster API](#cluster-api) +- [Exoscale](#exoscale) +- [Hetzner Cloud](#hetzner-cloud) ### Templating the autoDiscovery.clusterName @@ -222,6 +224,7 @@ $ helm install my-release autoscaler/cluster-autoscaler -f myvalues.yaml `cloudProvider: clusterapi` must be set, and then one or more of - `autoDiscovery.clusterName` +- or `autoDiscovery.namespace` - or `autoDiscovery.labels` See [here](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/clusterapi/README.md#configuring-node-group-auto-discovery) for more details. @@ -259,6 +262,18 @@ $ helm install my-release autoscaler/cluster-autoscaler \ Read [cluster-autoscaler/cloudprovider/exoscale/README.md](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/exoscale/README.md) for further information on the setup without helm. +### Hetzner Cloud + +The following parameters are required: + +- `cloudProvider=hetzner` +- `extraEnv.HCLOUD_TOKEN=...` +- `autoscalingGroups=...` + +Each autoscaling group requires an additional `instanceType` and `region` key to be set. + +Read [cluster-autoscaler/cloudprovider/hetzner/README.md](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/hetzner/README.md) for further information on the setup without helm. + ## Uninstalling the Chart To uninstall `my-release`: diff --git a/charts/cluster-autoscaler/templates/NOTES.txt b/charts/cluster-autoscaler/templates/NOTES.txt index 94e211e96b10..1a87a3d10b50 100644 --- a/charts/cluster-autoscaler/templates/NOTES.txt +++ b/charts/cluster-autoscaler/templates/NOTES.txt @@ -1,4 +1,4 @@ -{{- if or .Values.autoDiscovery.clusterName .Values.autoscalingGroups -}} +{{- if or ( or .Values.autoDiscovery.clusterName .Values.autoDiscovery.namespace .Values.autoDiscovery.labels ) .Values.autoscalingGroups }} To verify that cluster-autoscaler has started, run: @@ -8,7 +8,7 @@ To verify that cluster-autoscaler has started, run: ############################################################################## #### ERROR: You must specify values for either #### -#### autoDiscovery.clusterName or autoscalingGroups[] #### +#### autoDiscovery or autoscalingGroups[] #### ############################################################################## The deployment and pod will not be created and the installation is not functional diff --git a/charts/cluster-autoscaler/templates/_helpers.tpl b/charts/cluster-autoscaler/templates/_helpers.tpl index b0aee4fc080d..c7e80f4d8e4c 100644 --- a/charts/cluster-autoscaler/templates/_helpers.tpl +++ b/charts/cluster-autoscaler/templates/_helpers.tpl @@ -40,14 +40,11 @@ app.kubernetes.io/name: {{ include "cluster-autoscaler.name" . | quote }} {{/* -Return labels, including instance, name and version. +Return labels, including instance and name. */}} {{- define "cluster-autoscaler.labels" -}} {{ include "cluster-autoscaler.instance-name" . }} app.kubernetes.io/managed-by: {{ .Release.Service | quote }} -{{- if .Chart.AppVersion }} -app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} -{{- end }} helm.sh/chart: {{ include "cluster-autoscaler.chart" . | quote }} {{- if .Values.additionalLabels }} {{ toYaml .Values.additionalLabels }} @@ -112,16 +109,23 @@ Return true if the priority expander is enabled {{- end -}} {{/* -Return the autodiscoveryparameters for clusterapi. +autoDiscovery.clusterName for clusterapi. */}} -{{- define "cluster-autoscaler.capiAutodiscoveryConfig" -}} -{{- if .Values.autoDiscovery.clusterName -}} +{{- define "cluster-autoscaler.capiAutodiscovery.clusterName" -}} {{- print "clusterName=" -}}{{ tpl (.Values.autoDiscovery.clusterName) . }} {{- end -}} -{{- if and .Values.autoDiscovery.clusterName .Values.autoDiscovery.labels -}} -{{- print "," -}} + +{{/* +autoDiscovery.namespace for clusterapi. +*/}} +{{- define "cluster-autoscaler.capiAutodiscovery.namespace" -}} +{{- print "namespace=" }}{{ .Values.autoDiscovery.namespace -}} {{- end -}} -{{- if .Values.autoDiscovery.labels -}} + +{{/* +autoDiscovery.labels for clusterapi. +*/}} +{{- define "cluster-autoscaler.capiAutodiscovery.labels" -}} {{- range $i, $el := .Values.autoDiscovery.labels -}} {{- if $i -}}{{- print "," -}}{{- end -}} {{- range $key, $val := $el -}} @@ -129,4 +133,28 @@ Return the autodiscoveryparameters for clusterapi. {{- end -}} {{- end -}} {{- end -}} + +{{/* +Return the autodiscoveryparameters for clusterapi. +*/}} +{{- define "cluster-autoscaler.capiAutodiscoveryConfig" -}} +{{- if .Values.autoDiscovery.clusterName -}} +{{ include "cluster-autoscaler.capiAutodiscovery.clusterName" . }} + {{- if .Values.autoDiscovery.namespace }} + {{- print "," -}} + {{ include "cluster-autoscaler.capiAutodiscovery.namespace" . }} + {{- end -}} + {{- if .Values.autoDiscovery.labels }} + {{- print "," -}} + {{ include "cluster-autoscaler.capiAutodiscovery.labels" . }} + {{- end -}} +{{- else if .Values.autoDiscovery.namespace -}} +{{ include "cluster-autoscaler.capiAutodiscovery.namespace" . }} + {{- if .Values.autoDiscovery.labels }} + {{- print "," -}} + {{ include "cluster-autoscaler.capiAutodiscovery.labels" . }} + {{- end -}} +{{- else if .Values.autoDiscovery.labels -}} + {{ include "cluster-autoscaler.capiAutodiscovery.labels" . }} +{{- end -}} {{- end -}} diff --git a/charts/cluster-autoscaler/templates/deployment.yaml b/charts/cluster-autoscaler/templates/deployment.yaml index ccbe4353edc8..1b06186bf212 100644 --- a/charts/cluster-autoscaler/templates/deployment.yaml +++ b/charts/cluster-autoscaler/templates/deployment.yaml @@ -1,4 +1,4 @@ -{{- if or ( or .Values.autoDiscovery.clusterName .Values.autoDiscovery.labels ) .Values.autoscalingGroups }} +{{- if or ( or .Values.autoDiscovery.clusterName .Values.autoDiscovery.namespace .Values.autoDiscovery.labels ) .Values.autoscalingGroups }} {{/* one of the above is required */}} apiVersion: {{ template "deployment.apiVersion" . }} kind: Deployment @@ -60,8 +60,12 @@ spec: {{- end }} {{- if .Values.autoscalingGroups }} {{- range .Values.autoscalingGroups }} + {{- if eq $.Values.cloudProvider "hetzner" }} + - --nodes={{ .minSize }}:{{ .maxSize }}:{{ .instanceType }}:{{ .region }}:{{ .name }} + {{- else }} - --nodes={{ .minSize }}:{{ .maxSize }}:{{ .name }} {{- end }} + {{- end }} {{- end }} {{- if eq .Values.cloudProvider "rancher" }} {{- if .Values.cloudConfigPath }} @@ -95,7 +99,7 @@ spec: - --cluster-name={{ tpl (.Values.magnumClusterName) . }} {{- end }} {{- else if eq .Values.cloudProvider "clusterapi" }} - {{- if or .Values.autoDiscovery.clusterName .Values.autoDiscovery.labels }} + {{- if or .Values.autoDiscovery.clusterName .Values.autoDiscovery.labels .Values.autoDiscovery.namepace }} - --node-group-auto-discovery=clusterapi:{{ template "cluster-autoscaler.capiAutodiscoveryConfig" . }} {{- end }} {{- if eq .Values.clusterAPIMode "incluster-kubeconfig"}} @@ -166,6 +170,8 @@ spec: secretKeyRef: key: VMType name: {{ default (include "cluster-autoscaler.fullname" .) .Values.secretKeyRefNameOverride }} + - name: AZURE_ENABLE_FORCE_DELETE + value: "{{ .Values.azureEnableForceDelete }}" {{- if .Values.azureUseWorkloadIdentityExtension }} - name: ARM_USE_WORKLOAD_IDENTITY_EXTENSION value: "true" diff --git a/charts/cluster-autoscaler/values.yaml b/charts/cluster-autoscaler/values.yaml index edc17f9addd8..88581381eeda 100644 --- a/charts/cluster-autoscaler/values.yaml +++ b/charts/cluster-autoscaler/values.yaml @@ -16,6 +16,9 @@ autoDiscovery: # Enable autodiscovery for `cloudProvider=magnum`, for groups matching `autoDiscovery.roles`. clusterName: # cluster.local + # autoDiscovery.namespace -- Enable autodiscovery via cluster namespace for for `cloudProvider=clusterapi` + namespace: # default + # autoDiscovery.tags -- ASG tags to match, run through `tpl`. tags: - k8s.io/cluster-autoscaler/enabled @@ -36,6 +39,14 @@ autoDiscovery: # maxSize: 2
# minSize: 1 # +# For Hetzner Cloud, the `instanceType` and `region` keys are also required. +# 
+# - name: mypool

+#   maxSize: 2

+#   minSize: 1

+#   instanceType: CPX21

+#   region: FSN1
+#
autoscalingGroups: [] # - name: asg1 # maxSize: 2 @@ -75,14 +86,6 @@ azureClientID: "" # Required if `cloudProvider=azure` azureClientSecret: "" -# azureClusterName -- Azure AKS cluster name. -# Required if `cloudProvider=azure` -azureClusterName: "" - -# azureNodeResourceGroup -- Azure resource group where the cluster's nodes are located, typically set as `MC___`. -# Required if `cloudProvider=azure` -azureNodeResourceGroup: "" - # azureResourceGroup -- Azure resource group that the cluster is located. # Required if `cloudProvider=azure` azureResourceGroup: "" @@ -104,6 +107,9 @@ azureUseWorkloadIdentityExtension: false # azureVMType -- Azure VM type. azureVMType: "vmss" +# azureEnableForceDelete -- Whether to force delete VMs or VMSS instances when scaling down. +azureEnableForceDelete: false + # cloudConfigPath -- Configuration file for cloud provider. cloudConfigPath: "" @@ -230,7 +236,7 @@ image: # image.repository -- Image repository repository: registry.k8s.io/autoscaling/cluster-autoscaler # image.tag -- Image tag - tag: v1.28.2 + tag: v1.29.0 # image.pullPolicy -- Image pull policy pullPolicy: IfNotPresent ## Optionally specify an array of imagePullSecrets. @@ -375,7 +381,6 @@ serviceMonitor: # serviceMonitor.metricRelabelings -- MetricRelabelConfigs to apply to samples before ingestion. metricRelabelings: {} - # tolerations -- List of node taints to tolerate (requires Kubernetes >= 1.6). tolerations: [] diff --git a/cluster-autoscaler/.gitignore b/cluster-autoscaler/.gitignore index cdfefe3f0b13..69c0a6474e57 100644 --- a/cluster-autoscaler/.gitignore +++ b/cluster-autoscaler/.gitignore @@ -1,10 +1,8 @@ -cluster-autoscaler -cluster-autoscaler-amd64 -cluster-autoscaler-arm64 -cluster-autoscaler-s390x -cluster_autoscaler -main -.cover +/cluster-autoscaler +/cluster-autoscaler-amd64 +/cluster-autoscaler-arm64 +/cluster-autoscaler-s390x +/.cover # Vim-related files [._]*.s[a-w][a-z] @@ -13,4 +11,4 @@ main Session.vim .netrwhist .vscode -/integration/logs +./integration/logs diff --git a/cluster-autoscaler/FAQ.md b/cluster-autoscaler/FAQ.md index cecf928e9a6c..025bae238573 100644 --- a/cluster-autoscaler/FAQ.md +++ b/cluster-autoscaler/FAQ.md @@ -103,8 +103,13 @@ Cluster Autoscaler decreases the size of the cluster when some nodes are consist "cluster-autoscaler.kubernetes.io/safe-to-evict-local-volumes": "volume-1,volume-2,.." ``` and all of the pod's local volumes are listed in the annotation value. -* Pods that cannot be moved elsewhere due to various constraints (lack of resources, non-matching node selectors or affinity, -matching anti-affinity, etc) +* Pods that cannot be moved elsewhere due to scheduling constraints. CA simulates kube-scheduler behavior, and if there's no other node where a given pod can schedule, the pod's node won't be scaled down. + * This can be particularly visible if a given workloads' pods are configured to only fit one pod per node on some subset of nodes. Such pods will always block CA from scaling down their nodes, because all + other valid nodes are either taken by another pod, or empty (and CA prefers scaling down empty nodes). + * Examples of scenarios where scheduling constraints prevent CA from deleting a node: + * No other node has enough resources to satisfy a pod's request + * No other node has available ports to satisfy a pod's `hostPort` configuration. + * No other node with enough resources has the labels required by a pod's node selector * Pods that have the following annotation set: ``` "cluster-autoscaler.kubernetes.io/safe-to-evict": "false" @@ -597,7 +602,7 @@ any nodes left unregistered after this time. Every 10 seconds (configurable by `--scan-interval` flag), if no scale-up is needed, Cluster Autoscaler checks which nodes are unneeded. A node is considered for removal when **all** below conditions hold: -* The sum of cpu and memory requests of all pods running on this node ([DaemonSet pods](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) and [Mirror pods](https://kubernetes.io/docs/tasks/configure-pod-container/static-pod/) are included by default but this is configurable with `--ignore-daemonsets-utilization` and `--ignore-mirror-pods-utilization` flags) is smaller +* The sum of cpu requests and sum of memory requests of all pods running on this node ([DaemonSet pods](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) and [Mirror pods](https://kubernetes.io/docs/tasks/configure-pod-container/static-pod/) are included by default but this is configurable with `--ignore-daemonsets-utilization` and `--ignore-mirror-pods-utilization` flags) are smaller than 50% of the node's allocatable. (Before 1.1.0, node capacity was used instead of allocatable.) Utilization threshold can be configured using `--scale-down-utilization-threshold` flag. @@ -777,7 +782,7 @@ The following startup parameters are supported for cluster autoscaler: | `scale-down-delay-after-failure` | How long after scale down failure that scale down evaluation resumes | 3 minutes | `scale-down-unneeded-time` | How long a node should be unneeded before it is eligible for scale down | 10 minutes | `scale-down-unready-time` | How long an unready node should be unneeded before it is eligible for scale down | 20 minutes -| `scale-down-utilization-threshold` | Node utilization level, defined as sum of requested resources divided by capacity, below which a node can be considered for scale down | 0.5 +| `scale-down-utilization-threshold` | The maximum value between the sum of cpu requests and sum of memory requests of all pods running on the node divided by node's corresponding allocatable resource, below which a node can be considered for scale down. This value is a floating point number that can range between zero and one. | 0.5 | `scale-down-non-empty-candidates-count` | Maximum number of non empty nodes considered in one iteration as candidates for scale down with drain
Lower value means better CA responsiveness but possible slower scale down latency
Higher value can affect CA performance with big clusters (hundreds of nodes)
Set to non positive value to turn this heuristic off - CA will not limit the number of nodes it considers." | 30 | `scale-down-candidates-pool-ratio` | A ratio of nodes that are considered as additional non empty candidates for
scale down when some candidates from previous iteration are no longer valid
Lower value means better CA responsiveness but possible slower scale down latency
Higher value can affect CA performance with big clusters (hundreds of nodes)
Set to 1.0 to turn this heuristics off - CA will take all nodes as additional candidates. | 0.1 | `scale-down-candidates-pool-min-count` | Minimum number of nodes that are considered as additional non empty candidates
for scale down when some candidates from previous iteration are no longer valid.
When calculating the pool size for additional candidates we take
`max(#nodes * scale-down-candidates-pool-ratio, scale-down-candidates-pool-min-count)` | 50 diff --git a/cluster-autoscaler/Makefile b/cluster-autoscaler/Makefile index a061ce065b65..34e4d729cd63 100644 --- a/cluster-autoscaler/Makefile +++ b/cluster-autoscaler/Makefile @@ -36,7 +36,10 @@ IMAGE=$(REGISTRY)/cluster-autoscaler$(PROVIDER) export DOCKER_CLI_EXPERIMENTAL := enabled -build: build-arch-$(GOARCH) +build: + @echo "⚠️ WARNING: The vendor directory will be removed soon. \ + Please make sure your dependencies are managed via Go modules." + @$(MAKE) build-arch-$(GOARCH) build-arch-%: clean-arch-% $(ENVVAR) GOOS=$(GOOS) GOARCH=$* go build -o cluster-autoscaler-$* ${LDFLAGS_FLAG} ${TAGS_FLAG} @@ -119,7 +122,7 @@ $(LOCALBIN): CONTROLLER_GEN ?= $(LOCALBIN)/controller-gen ## Tool Versions -CONTROLLER_TOOLS_VERSION ?= v0.13.0 +CONTROLLER_TOOLS_VERSION ?= v0.14.0 .PHONY: controller-gen controller-gen: $(CONTROLLER_GEN) ## Download controller-gen locally if necessary. @@ -128,7 +131,8 @@ $(CONTROLLER_GEN): $(LOCALBIN) .PHONY: manifest manifest: controller-gen ## Generate WebhookConfiguration, ClusterRole and CustomResourceDefinition objects. - $(CONTROLLER_GEN) rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd + $(CONTROLLER_GEN) rbac:roleName=manager-role crd webhook paths="./apis/..." output:crd:artifacts:config=apis/config/crd + .PHONY: start start: @GO111MODULE=on go run main.go \ diff --git a/cluster-autoscaler/OWNERS b/cluster-autoscaler/OWNERS index d96aa261a525..ec15a71b84a3 100644 --- a/cluster-autoscaler/OWNERS +++ b/cluster-autoscaler/OWNERS @@ -6,8 +6,9 @@ approvers: reviewers: - BigDarkClown - feiskyer +- vadasambar - x13n emeritus_approvers: - aleksandra-malinowska # 2022-09-30 labels: -- cluster-autoscaler +- area/cluster-autoscaler diff --git a/cluster-autoscaler/README.md b/cluster-autoscaler/README.md index 4c0b78b38dbb..5225b8ee7a82 100644 --- a/cluster-autoscaler/README.md +++ b/cluster-autoscaler/README.md @@ -25,7 +25,7 @@ You should also take a look at the notes and "gotchas" for your specific cloud p * [ClusterAPI](./cloudprovider/clusterapi/README.md) * [DigitalOcean](./cloudprovider/digitalocean/README.md) * [Exoscale](./cloudprovider/exoscale/README.md) -* [Equinix Metal](./cloudprovider/packet/README.md#notes) +* [Equinix Metal](cloudprovider/equinixmetal/README.md#notes) * [External gRPC](./cloudprovider/externalgrpc/README.md) * [Hetzner](./cloudprovider/hetzner/README.md) * [HuaweiCloud](./cloudprovider/huaweicloud/README.md) @@ -47,53 +47,54 @@ We recommend using Cluster Autoscaler with the Kubernetes control plane (previou Starting from Kubernetes 1.12, versioning scheme was changed to match Kubernetes minor releases exactly. -| Kubernetes Version | CA Version | -|--------------------|--------| -| 1.29.X | 1.29.X | -| 1.28.X | 1.28.X | -| 1.27.X | 1.27.X | -| 1.26.X | 1.26.X | -| 1.25.X | 1.25.X | -| 1.24.X | 1.24.X | -| 1.23.X | 1.23.X | -| 1.22.X | 1.22.X | -| 1.21.X | 1.21.X | -| 1.20.X | 1.20.X | -| 1.19.X | 1.19.X | -| 1.18.X | 1.18.X | -| 1.17.X | 1.17.X | -| 1.16.X | 1.16.X | -| 1.15.X | 1.15.X | -| 1.14.X | 1.14.X | -| 1.13.X | 1.13.X | -| 1.12.X | 1.12.X | -| 1.11.X | 1.3.X | -| 1.10.X | 1.2.X | -| 1.9.X | 1.1.X | -| 1.8.X | 1.0.X | -| 1.7.X | 0.6.X | -| 1.6.X | 0.5.X, 0.6.X* | -| 1.5.X | 0.4.X | -| 1.4.X | 0.3.X | +| Kubernetes Version | CA Version | Chart Version | +|--------------------|--------------------------|---------------| +| 1.29.X | 1.29.X |9.35.0+| +| 1.28.X | 1.28.X |9.34.0+| +| 1.27.X | 1.27.X |9.29.0+| +| 1.26.X | 1.26.X |9.28.0+| +| 1.25.X | 1.25.X | | +| 1.24.X | 1.24.X |9.25.0+| +| 1.23.X | 1.23.X |9.14.0+| +| 1.22.X | 1.22.X | | +| 1.21.X | 1.21.X |9.10.0+| +| 1.20.X | 1.20.X |9.5.0+| +| 1.19.X | 1.19.X | | +| 1.18.X | 1.18.X |9.0.0+| +| 1.17.X | 1.17.X | | +| 1.16.X | 1.16.X | | +| 1.15.X | 1.15.X | | +| 1.14.X | 1.14.X | | +| 1.13.X | 1.13.X | | +| 1.12.X | 1.12.X | | +| 1.11.X | 1.3.X | | +| 1.10.X | 1.2.X | | +| 1.9.X | 1.1.X | | +| 1.8.X | 1.0.X | | +| 1.7.X | 0.6.X | | +| 1.6.X | 0.5.X, 0.6.X* | | +| 1.5.X | 0.4.X | | +| 1.4.X | 0.3.X | | *Cluster Autoscaler 0.5.X is the official version shipped with k8s 1.6. We've done some basic tests using k8s 1.6 / CA 0.6 and we're not aware of any problems with this setup. However, Cluster Autoscaler internally simulates Kubernetes' scheduler and using different versions of scheduler code can lead to subtle issues. -# Patch releases +## Schedule -Cluster Autoscaler releases patches for versions corresponding to currently +Cluster Autoscaler releases new minor versions shortly after OSS Kubernetes release +and patches for versions corresponding to currently supported [Kubernetes versions](https://kubernetes.io/releases/) on a roughly 2 month cadence. Currently planned schedule is below. Please note that target dates listed below are approximate and we expect up to a week difference between target ETA and the actual releases. -| Date | Maintainer Preparing Release | Backup Maintainer | -|------------|------------------------------|-------------------| -| 2023-03-15 | MaciekPytel | gjtempleton | -| 2023-05-17 | gjtempleton | BigDarkClown | -| 2023-07-19 | BigDarkClown | towca | -| 2023-09-13 | towca | x13n | -| 2023-11-15 | x13n | MaciekPytel | -| 2024-01-17 | MaciekPytel | gjtempleton | +| Date | Maintainer Preparing Release | Backup Maintainer | Type | +|------------|------------------------------|-------------------|-------| +| 2024-01-17 | MaciekPytel | gjtempleton | patch | +| 2024-03-13 | gjtempleton | BigDarkClown | patch | +| 2024-04-17 | BigDarkClown | towca | 1.30 | +| 2024-05-22 | towca | x13n | patch | +| 2024-07-24 | x13n | MaciekPytel | patch | +| 2024-08-21 | MaciekPytel | gjtempleton | 1.31 | Additional patch releases may happen outside of the schedule in case of critical bugs or vulnerabilities. @@ -104,6 +105,7 @@ Starting with Gardener/Autoscaler v1.20, versioning scheme has changed to match | Kubernetes Version | CA Version | Gardener CA Version | |--------------------|------------|---------------------| +| 1.30.X | 1.30.X | 1.30.X | | 1.29.X | 1.29.X | 1.29.X | | 1.28.X | 1.28.X | 1.28.X | | 1.27.X | 1.27.X | 1.27.X | @@ -233,7 +235,7 @@ Supported cloud providers: * ClusterAPI https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/clusterapi/README.md * DigitalOcean https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/digitalocean/README.md * Exoscale https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/exoscale/README.md -* Equinix Metal https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/packet/README.md +* Equinix Metal https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/equinixmetal/README.md * External gRPC https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/externalgrpc/README.md * Hetzner https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/hetzner/README.md * HuaweiCloud https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/huaweicloud/README.md diff --git a/cluster-autoscaler/SYNC-CHANGES/SYNC_CHANGES-1.30.md b/cluster-autoscaler/SYNC-CHANGES/SYNC_CHANGES-1.30.md new file mode 100644 index 000000000000..f53125a23b48 --- /dev/null +++ b/cluster-autoscaler/SYNC-CHANGES/SYNC_CHANGES-1.30.md @@ -0,0 +1,33 @@ + + +- [v1.30.1](#v1290) + - [Synced with which upstream CA](#synced-with-which-upstream-ca) + - [Changes made](#changes-made) + - [During merging](#during-merging) + - [During vendoring k8s](#during-vendoring-k8s) + - [Others](#others) + + +# v1.30.0 + + +## Synced with which upstream CA + +[v1.30.1](https://github.com/kubernetes/autoscaler/releases/tag/cluster-autoscaler-1.30.1) + +## Changes made +- See general release notes of 1.30.1: https://github.com/kubernetes/autoscaler/releases/tag/cluster-autoscaler-1.30.1 +- New flag added in autoscaling options: `flag.Bool("enable-provisioning-requests", false, "Whether the clusterautoscaler will be handling the ProvisioningRequest CRs.")`. +- New flag added in autoscaling options: `flag.Bool("frequent-loops-enabled", false, "Whether clusterautoscaler triggers new iterations more frequently when it's needed")`. +- New interface method `AtomicIncreaseSize` implemented in mcm cloud provider to satisfy interface : `cloudprovider.NodeGroup`. + +### During merging +- import package for `machine-controller-manager-provider-aws` was updated after vendoring latest version. +- import package for `machine-controller-manager-provider-azure` was updated after vendoring latest version. + +### During vendoring k8s +- mcm-provider-aws v0.19.2 -> v0.20.0 +- mcm-provider-azure v0.12.1 -> v0.13.0 + +### Others +- [Release matrix](../README.md#releases-gardenerautoscaler) of Gardener Autoscaler updated. \ No newline at end of file diff --git a/cluster-autoscaler/apis/config/crd/autoscaling.x-k8s.io_provisioningrequests.yaml b/cluster-autoscaler/apis/config/crd/autoscaling.x-k8s.io_provisioningrequests.yaml new file mode 100644 index 000000000000..bdbb53ce338d --- /dev/null +++ b/cluster-autoscaler/apis/config/crd/autoscaling.x-k8s.io_provisioningrequests.yaml @@ -0,0 +1,236 @@ +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.14.0 + name: provisioningrequests.autoscaling.x-k8s.io +spec: + group: autoscaling.x-k8s.io + names: + kind: ProvisioningRequest + listKind: ProvisioningRequestList + plural: provisioningrequests + shortNames: + - provreq + - provreqs + singular: provisioningrequest + scope: Namespaced + versions: + - additionalPrinterColumns: + - jsonPath: .metadata.creationTimestamp + name: Age + type: date + name: v1beta1 + schema: + openAPIV3Schema: + description: |- + ProvisioningRequest is a way to express additional capacity + that we would like to provision in the cluster. Cluster Autoscaler + can use this information in its calculations and signal if the capacity + is available in the cluster or actively add capacity if needed. + properties: + apiVersion: + description: |- + APIVersion defines the versioned schema of this representation of an object. + Servers should convert recognized schemas to the latest internal value, and + may reject unrecognized values. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + type: string + kind: + description: |- + Kind is a string value representing the REST resource this object represents. + Servers may infer this from the endpoint the client submits requests to. + Cannot be updated. + In CamelCase. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + type: string + metadata: + type: object + spec: + description: |- + Spec contains specification of the ProvisioningRequest object. + More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#spec-and-status. + The spec is immutable, to make changes to the request users are expected to delete an existing + and create a new object with the corrected fields. + properties: + parameters: + additionalProperties: + description: Parameter is limited to 255 characters. + maxLength: 255 + type: string + description: |- + Parameters contains all other parameters classes may require. + 'atomic-scale-up.kubernetes.io' supports 'ValidUntilSeconds' parameter, which should contain + a string denoting duration for which we should retry (measured since creation fo the CR). + maxProperties: 100 + type: object + x-kubernetes-validations: + - message: Value is immutable + rule: self == oldSelf + podSets: + description: |- + PodSets lists groups of pods for which we would like to provision + resources. + items: + description: PodSet represents one group of pods for Provisioning + Request to provision capacity. + properties: + count: + description: |- + Count contains the number of pods that will be created with a given + template. + format: int32 + minimum: 1 + type: integer + podTemplateRef: + description: |- + PodTemplateRef is a reference to a PodTemplate object that is representing pods + that will consume this reservation (must be within the same namespace). + Users need to make sure that the fields relevant to scheduler (e.g. node selector tolerations) + are consistent between this template and actual pods consuming the Provisioning Request. + properties: + name: + description: |- + Name of the referenced object. + More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names#names + maxLength: 253 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + type: object + required: + - count + - podTemplateRef + type: object + maxItems: 32 + minItems: 1 + type: array + x-kubernetes-validations: + - message: Value is immutable + rule: self == oldSelf + provisioningClassName: + description: |- + ProvisioningClassName describes the different modes of provisioning the resources. + Currently there is no support for 'ProvisioningClass' objects. + Supported values: + * check-capacity.kubernetes.io - check if current cluster state can fullfil this request, + do not reserve the capacity. Users should provide a reference to a valid PodTemplate object. + CA will check if there is enough capacity in cluster to fulfill the request and put + the answer in 'CapacityAvailable' condition. + * atomic-scale-up.kubernetes.io - provision the resources in an atomic manner. + Users should provide a reference to a valid PodTemplate object. + CA will try to create the VMs in an atomic manner, clean any partially provisioned VMs + and re-try the operation in a exponential back-off manner. Users can configure the timeout + duration after which the request will fail by 'ValidUntilSeconds' key in 'Parameters'. + CA will set 'Failed=true' or 'Provisioned=true' condition according to the outcome. + * ... - potential other classes that are specific to the cloud providers. + 'kubernetes.io' suffix is reserved for the modes defined in Kubernetes projects. + maxLength: 253 + pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$ + type: string + x-kubernetes-validations: + - message: Value is immutable + rule: self == oldSelf + required: + - podSets + - provisioningClassName + type: object + status: + description: Status of the ProvisioningRequest. CA constantly reconciles + this field. + properties: + conditions: + description: |- + Conditions represent the observations of a Provisioning Request's + current state. Those will contain information whether the capacity + was found/created or if there were any issues. The condition types + may differ between different provisioning classes. + items: + description: "Condition contains details for one aspect of the current + state of this API Resource.\n---\nThis struct is intended for + direct use as an array at the field path .status.conditions. For + example,\n\n\n\ttype FooStatus struct{\n\t // Represents the + observations of a foo's current state.\n\t // Known .status.conditions.type + are: \"Available\", \"Progressing\", and \"Degraded\"\n\t // + +patchMergeKey=type\n\t // +patchStrategy=merge\n\t // +listType=map\n\t + \ // +listMapKey=type\n\t Conditions []metav1.Condition `json:\"conditions,omitempty\" + patchStrategy:\"merge\" patchMergeKey:\"type\" protobuf:\"bytes,1,rep,name=conditions\"`\n\n\n\t + \ // other fields\n\t}" + properties: + lastTransitionTime: + description: |- + lastTransitionTime is the last time the condition transitioned from one status to another. + This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. + format: date-time + type: string + message: + description: |- + message is a human readable message indicating details about the transition. + This may be an empty string. + maxLength: 32768 + type: string + observedGeneration: + description: |- + observedGeneration represents the .metadata.generation that the condition was set based upon. + For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date + with respect to the current state of the instance. + format: int64 + minimum: 0 + type: integer + reason: + description: |- + reason contains a programmatic identifier indicating the reason for the condition's last transition. + Producers of specific condition types may define expected values and meanings for this field, + and whether the values are considered a guaranteed API. + The value should be a CamelCase string. + This field may not be empty. + maxLength: 1024 + minLength: 1 + pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ + type: string + status: + description: status of the condition, one of True, False, Unknown. + enum: + - "True" + - "False" + - Unknown + type: string + type: + description: |- + type of condition in CamelCase or in foo.example.com/CamelCase. + --- + Many .condition.type values are consistent across resources like Available, but because arbitrary conditions can be + useful (see .node.status.conditions), the ability to deconflict is important. + The regex it matches is (dns1123SubdomainFmt/)?(qualifiedNameFmt) + maxLength: 316 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ + type: string + required: + - lastTransitionTime + - message + - reason + - status + - type + type: object + type: array + x-kubernetes-list-map-keys: + - type + x-kubernetes-list-type: map + provisioningClassDetails: + additionalProperties: + description: Detail is limited to 32768 characters. + maxLength: 32768 + type: string + description: |- + ProvisioningClassDetails contains all other values custom provisioning classes may + want to pass to end users. + maxProperties: 64 + type: object + type: object + required: + - spec + type: object + served: true + storage: true + subresources: + status: {} diff --git a/cluster-autoscaler/apis/go.mod b/cluster-autoscaler/apis/go.mod new file mode 100644 index 000000000000..59d43f1a2a15 --- /dev/null +++ b/cluster-autoscaler/apis/go.mod @@ -0,0 +1,58 @@ +module k8s.io/autoscaler/cluster-autoscaler/apis + +go 1.22.0 + +require ( + github.com/onsi/ginkgo/v2 v2.16.0 + github.com/onsi/gomega v1.31.1 + k8s.io/apimachinery v0.30.1 + k8s.io/client-go v0.30.1 + k8s.io/code-generator v0.30.1 + sigs.k8s.io/structured-merge-diff/v4 v4.4.1 +) + +require ( + github.com/davecgh/go-spew v1.1.1 // indirect + github.com/emicklei/go-restful/v3 v3.11.0 // indirect + github.com/evanphx/json-patch v4.12.0+incompatible // indirect + github.com/go-logr/logr v1.4.1 // indirect + github.com/go-openapi/jsonpointer v0.19.6 // indirect + github.com/go-openapi/jsonreference v0.20.2 // indirect + github.com/go-openapi/swag v0.22.3 // indirect + github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572 // indirect + github.com/gogo/protobuf v1.3.2 // indirect + github.com/golang/protobuf v1.5.4 // indirect + github.com/google/gnostic-models v0.6.8 // indirect + github.com/google/go-cmp v0.6.0 // indirect + github.com/google/gofuzz v1.2.0 // indirect + github.com/google/pprof v0.0.0-20210720184732-4bb14d4b1be1 // indirect + github.com/google/uuid v1.3.0 // indirect + github.com/josharian/intern v1.0.0 // indirect + github.com/json-iterator/go v1.1.12 // indirect + github.com/mailru/easyjson v0.7.7 // indirect + github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect + github.com/modern-go/reflect2 v1.0.2 // indirect + github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect + github.com/pkg/errors v0.9.1 // indirect + github.com/spf13/pflag v1.0.5 // indirect + golang.org/x/mod v0.15.0 // indirect + golang.org/x/net v0.23.0 // indirect + golang.org/x/oauth2 v0.10.0 // indirect + golang.org/x/sys v0.18.0 // indirect + golang.org/x/term v0.18.0 // indirect + golang.org/x/text v0.14.0 // indirect + golang.org/x/time v0.3.0 // indirect + golang.org/x/tools v0.18.0 // indirect + google.golang.org/appengine v1.6.7 // indirect + google.golang.org/protobuf v1.33.0 // indirect + gopkg.in/inf.v0 v0.9.1 // indirect + gopkg.in/yaml.v2 v2.4.0 // indirect + gopkg.in/yaml.v3 v3.0.1 // indirect + k8s.io/api v0.30.1 // indirect + k8s.io/gengo/v2 v2.0.0-20240228010128-51d4e06bde70 // indirect + k8s.io/klog/v2 v2.120.1 // indirect + k8s.io/kube-openapi v0.0.0-20240228011516-70dd3763d340 // indirect + k8s.io/utils v0.0.0-20230726121419-3b25d923346b // indirect + sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd // indirect + sigs.k8s.io/yaml v1.3.0 // indirect +) diff --git a/cluster-autoscaler/apis/go.sum b/cluster-autoscaler/apis/go.sum new file mode 100644 index 000000000000..a02472d84274 --- /dev/null +++ b/cluster-autoscaler/apis/go.sum @@ -0,0 +1,168 @@ +github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI= +github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI= +github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU= +github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E= +github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= +github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/emicklei/go-restful/v3 v3.11.0 h1:rAQeMHw1c7zTmncogyy8VvRZwtkmkZ4FxERmMY4rD+g= +github.com/emicklei/go-restful/v3 v3.11.0/go.mod h1:6n3XBCmQQb25CM2LCACGz8ukIrRry+4bhvbpWn3mrbc= +github.com/evanphx/json-patch v4.12.0+incompatible h1:4onqiflcdA9EOZ4RxV643DvftH5pOlLGNtQ5lPWQu84= +github.com/evanphx/json-patch v4.12.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk= +github.com/go-logr/logr v1.4.1 h1:pKouT5E8xu9zeFC39JXRDukb6JFQPXM5p5I91188VAQ= +github.com/go-logr/logr v1.4.1/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= +github.com/go-openapi/jsonpointer v0.19.6 h1:eCs3fxoIi3Wh6vtgmLTOjdhSpiqphQ+DaPn38N2ZdrE= +github.com/go-openapi/jsonpointer v0.19.6/go.mod h1:osyAmYz/mB/C3I+WsTTSgw1ONzaLJoLCyoi6/zppojs= +github.com/go-openapi/jsonreference v0.20.2 h1:3sVjiK66+uXK/6oQ8xgcRKcFgQ5KXa2KvnJRumpMGbE= +github.com/go-openapi/jsonreference v0.20.2/go.mod h1:Bl1zwGIM8/wsvqjsOQLJ/SH+En5Ap4rVB5KVcIDZG2k= +github.com/go-openapi/swag v0.22.3 h1:yMBqmnQ0gyZvEb/+KzuWZOXgllrXT4SADYbvDaXHv/g= +github.com/go-openapi/swag v0.22.3/go.mod h1:UzaqsxGiab7freDnrUUra0MwWfN/q7tE4j+VcZ0yl14= +github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572 h1:tfuBGBXKqDEevZMzYi5KSi8KkcZtzBcTgAUUtapy0OI= +github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572/go.mod h1:9Pwr4B2jHnOSGXyyzV8ROjYa2ojvAY6HCGYYfMoC3Ls= +github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q= +github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q= +github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= +github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek= +github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps= +github.com/google/gnostic-models v0.6.8 h1:yo/ABAfM5IMRsS1VnXjTBvUb61tFIHozhlYvRgGre9I= +github.com/google/gnostic-models v0.6.8/go.mod h1:5n7qKqH0f5wFt+aWF8CW6pZLLNOfYuF5OpfBSENuI8U= +github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= +github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI= +github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= +github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= +github.com/google/gofuzz v1.2.0 h1:xRy4A+RhZaiKjJ1bPfwQ8sedCA+YS2YcCHW6ec7JMi0= +github.com/google/gofuzz v1.2.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= +github.com/google/pprof v0.0.0-20210720184732-4bb14d4b1be1 h1:K6RDEckDVWvDI9JAJYCmNdQXq6neHJOYx3V6jnqNEec= +github.com/google/pprof v0.0.0-20210720184732-4bb14d4b1be1/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE= +github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I= +github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/ianlancetaylor/demangle v0.0.0-20200824232613-28f6c0f3b639/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc= +github.com/josharian/intern v1.0.0 h1:vlS4z54oSdjm0bgjRigI+G1HpF+tI+9rE5LLzOg8HmY= +github.com/josharian/intern v1.0.0/go.mod h1:5DoeVV0s6jJacbCEi61lwdGj/aVlrQvzHFFd8Hwg//Y= +github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM= +github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo= +github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8= +github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= +github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI= +github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= +github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk= +github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= +github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= +github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= +github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= +github.com/mailru/easyjson v0.7.7 h1:UGYAvKxe3sBsEDzO8ZeWOSlIQfWFlxbzLZe7hwFURr0= +github.com/mailru/easyjson v0.7.7/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc= +github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= +github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg= +github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= +github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M= +github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk= +github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA= +github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ= +github.com/onsi/ginkgo/v2 v2.16.0 h1:7q1w9frJDzninhXxjZd+Y/x54XNjG/UlRLIYPZafsPM= +github.com/onsi/ginkgo/v2 v2.16.0/go.mod h1:llBI3WDLL9Z6taip6f33H76YcWtJv+7R3HigUjbIBOs= +github.com/onsi/gomega v1.31.1 h1:KYppCUK+bUgAZwHOu7EXVBKyQA6ILvOESHkn/tgoqvo= +github.com/onsi/gomega v1.31.1/go.mod h1:y40C95dwAD1Nz36SsEnxvfFe8FFfNxzI5eJ0EYGyAy0= +github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= +github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= +github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= +github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/rogpeppe/go-internal v1.10.0 h1:TMyTOH3F/DB16zRVcYyreMH6GnZZrwQVAoYjRBZyWFQ= +github.com/rogpeppe/go-internal v1.10.0/go.mod h1:UQnix2H7Ngw/k4C5ijL5+65zddjncjaFoBhdsK/akog= +github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA= +github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= +github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= +github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw= +github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo= +github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= +github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= +github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= +github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU= +github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= +github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk= +github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo= +github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= +github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= +golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= +golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= +golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= +golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= +golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= +golang.org/x/mod v0.15.0 h1:SernR4v+D55NyBH2QiEQrlBAnj1ECL6AGrA5+dPaMY8= +golang.org/x/mod v0.15.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c= +golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= +golang.org/x/net v0.0.0-20190603091049-60506f45cf65/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks= +golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= +golang.org/x/net v0.23.0 h1:7EYJ93RZ9vYSZAIb2x3lnuvqO5zneoD6IvWjuhfxjTs= +golang.org/x/net v0.23.0/go.mod h1:JKghWKKOSdJwpW2GEx0Ja7fmaKnMsbu+MWVZTokSYmg= +golang.org/x/oauth2 v0.10.0 h1:zHCpF2Khkwy4mMB4bv0U37YtJdTGW8jI0glAApi0Kh8= +golang.org/x/oauth2 v0.10.0/go.mod h1:kTpgurOux7LqtuxjuyZa4Gj2gdezIt/jQtGnNFfypQI= +golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.6.0 h1:5BMeUDZ7vkXGfEr1x9B4bRcTH4lpkTkpdh0T/J+qjbQ= +golang.org/x/sync v0.6.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk= +golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20191204072324-ce4227a45e2e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.18.0 h1:DBdB3niSjOA/O0blCZBqDefyWNYveAYMNF1Wum0DYQ4= +golang.org/x/sys v0.18.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= +golang.org/x/term v0.18.0 h1:FcHjZXDMxI8mM3nwhX9HlKop4C0YQvCVCdwYl2wOtE8= +golang.org/x/term v0.18.0/go.mod h1:ILwASektA3OnRv7amZ1xhE/KTR+u50pbXfZ03+6Nx58= +golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= +golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk= +golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= +golang.org/x/text v0.14.0 h1:ScX5w1eTa3QqT8oi6+ziP7dTV1S2+ALU0bI+0zXKWiQ= +golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU= +golang.org/x/time v0.3.0 h1:rg5rLMjNzMS1RkNLzCG38eapWhnYLFYXDXj2gOlr8j4= +golang.org/x/time v0.3.0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= +golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= +golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= +golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= +golang.org/x/tools v0.18.0 h1:k8NLag8AGHnn+PHbl7g43CtqZAwG60vZkLqgyZgIHgQ= +golang.org/x/tools v0.18.0/go.mod h1:GL7B4CwcLLeo59yx/9UWWuNOW1n3VZ4f5axWfML7Lcg= +golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +google.golang.org/appengine v1.6.7 h1:FZR1q0exgwxzPzp/aF+VccGrSfxfPpkBqjIIEq3ru6c= +google.golang.org/appengine v1.6.7/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc= +google.golang.org/protobuf v1.33.0 h1:uNO2rsAINq/JlFpSdYEKIZ0uKD/R9cpdv0T+yoGwGmI= +google.golang.org/protobuf v1.33.0/go.mod h1:c6P6GXX6sHbq/GpV6MGZEdwhWPcYBgnhAHhKbcUYpos= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk= +gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q= +gopkg.in/inf.v0 v0.9.1 h1:73M5CoZyi3ZLMOyDlQh031Cx6N9NDJ2Vvfl76EDAgDc= +gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw= +gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= +gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY= +gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ= +gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= +gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= +gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= +k8s.io/api v0.30.1 h1:kCm/6mADMdbAxmIh0LBjS54nQBE+U4KmbCfIkF5CpJY= +k8s.io/api v0.30.1/go.mod h1:ddbN2C0+0DIiPntan/bye3SW3PdwLa11/0yqwvuRrJM= +k8s.io/apimachinery v0.30.1 h1:ZQStsEfo4n65yAdlGTfP/uSHMQSoYzU/oeEbkmF7P2U= +k8s.io/apimachinery v0.30.1/go.mod h1:iexa2somDaxdnj7bha06bhb43Zpa6eWH8N8dbqVjTUc= +k8s.io/client-go v0.30.1 h1:uC/Ir6A3R46wdkgCV3vbLyNOYyCJ8oZnjtJGKfytl/Q= +k8s.io/client-go v0.30.1/go.mod h1:wrAqLNs2trwiCH/wxxmT/x3hKVH9PuV0GGW0oDoHVqc= +k8s.io/code-generator v0.30.1 h1:ZsG++q5Vt0ScmKCeLhynUuWgcwFGg1Hl1AGfatqPJBI= +k8s.io/code-generator v0.30.1/go.mod h1:hFgxRsvOUg79mbpbVKfjJvRhVz1qLoe40yZDJ/hwRH4= +k8s.io/gengo/v2 v2.0.0-20240228010128-51d4e06bde70 h1:NGrVE502P0s0/1hudf8zjgwki1X/TByhmAoILTarmzo= +k8s.io/gengo/v2 v2.0.0-20240228010128-51d4e06bde70/go.mod h1:VH3AT8AaQOqiGjMF9p0/IM1Dj+82ZwjfxUP1IxaHE+8= +k8s.io/klog/v2 v2.120.1 h1:QXU6cPEOIslTGvZaXvFWiP9VKyeet3sawzTOvdXb4Vw= +k8s.io/klog/v2 v2.120.1/go.mod h1:3Jpz1GvMt720eyJH1ckRHK1EDfpxISzJ7I9OYgaDtPE= +k8s.io/kube-openapi v0.0.0-20240228011516-70dd3763d340 h1:BZqlfIlq5YbRMFko6/PM7FjZpUb45WallggurYhKGag= +k8s.io/kube-openapi v0.0.0-20240228011516-70dd3763d340/go.mod h1:yD4MZYeKMBwQKVht279WycxKyM84kkAx2DPrTXaeb98= +k8s.io/utils v0.0.0-20230726121419-3b25d923346b h1:sgn3ZU783SCgtaSJjpcVVlRqd6GSnlTLKgpAAttJvpI= +k8s.io/utils v0.0.0-20230726121419-3b25d923346b/go.mod h1:OLgZIPagt7ERELqWJFomSt595RzquPNLL48iOWgYOg0= +sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd h1:EDPBXCAspyGV4jQlpZSudPeMmr1bNJefnuqLsRAsHZo= +sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd/go.mod h1:B8JuhiUyNFVKdsE8h686QcCxMaH6HrOAZj4vswFpcB0= +sigs.k8s.io/structured-merge-diff/v4 v4.4.1 h1:150L+0vs/8DA78h1u02ooW1/fFq/Lwr+sGiqlzvrtq4= +sigs.k8s.io/structured-merge-diff/v4 v4.4.1/go.mod h1:N8hJocpFajUSSeSJ9bOZ77VzejKZaXsTtZo4/u7Io08= +sigs.k8s.io/yaml v1.3.0 h1:a2VclLzOGrwOHDiV8EfBGhvjHvP46CtW5j6POvhYGGo= +sigs.k8s.io/yaml v1.3.0/go.mod h1:GeOyir5tyXNByN85N/dRIT9es5UQNerPYEKK56eTBm8= diff --git a/cluster-autoscaler/provisioningrequest/apis/autoscaling.x-k8s.io/v1beta1/doc.go b/cluster-autoscaler/apis/provisioningrequest/autoscaling.x-k8s.io/v1beta1/doc.go similarity index 100% rename from cluster-autoscaler/provisioningrequest/apis/autoscaling.x-k8s.io/v1beta1/doc.go rename to cluster-autoscaler/apis/provisioningrequest/autoscaling.x-k8s.io/v1beta1/doc.go diff --git a/cluster-autoscaler/provisioningrequest/apis/autoscaling.x-k8s.io/v1beta1/register.go b/cluster-autoscaler/apis/provisioningrequest/autoscaling.x-k8s.io/v1beta1/register.go similarity index 100% rename from cluster-autoscaler/provisioningrequest/apis/autoscaling.x-k8s.io/v1beta1/register.go rename to cluster-autoscaler/apis/provisioningrequest/autoscaling.x-k8s.io/v1beta1/register.go diff --git a/cluster-autoscaler/apis/provisioningrequest/autoscaling.x-k8s.io/v1beta1/types.go b/cluster-autoscaler/apis/provisioningrequest/autoscaling.x-k8s.io/v1beta1/types.go new file mode 100644 index 000000000000..31cb27e901bc --- /dev/null +++ b/cluster-autoscaler/apis/provisioningrequest/autoscaling.x-k8s.io/v1beta1/types.go @@ -0,0 +1,203 @@ +/* +Copyright 2023 The Kubernetes Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// Package v1beta1 contains definitions of Provisioning Request related objects. +package v1beta1 + +import ( + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + + // Dependencies for the generation of the code: + _ "github.com/onsi/ginkgo/v2" + _ "github.com/onsi/gomega" + _ "k8s.io/code-generator" +) + +// +genclient +// +kubebuilder:storageversions +// +kubebuilder:resource:shortName=provreq;provreqs + +// ProvisioningRequest is a way to express additional capacity +// that we would like to provision in the cluster. Cluster Autoscaler +// can use this information in its calculations and signal if the capacity +// is available in the cluster or actively add capacity if needed. +// +// +kubebuilder:subresource:status +// +kubebuilder:printcolumn:name="Age",type="date",JSONPath=".metadata.creationTimestamp" +// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object +type ProvisioningRequest struct { + metav1.TypeMeta `json:",inline"` + // Standard object metadata. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#metadata + // + // +optional + metav1.ObjectMeta `json:"metadata,omitempty"` + // Spec contains specification of the ProvisioningRequest object. + // More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#spec-and-status. + // The spec is immutable, to make changes to the request users are expected to delete an existing + // and create a new object with the corrected fields. + // + // +kubebuilder:validation:Required + Spec ProvisioningRequestSpec `json:"spec"` + // Status of the ProvisioningRequest. CA constantly reconciles this field. + // + // +optional + Status ProvisioningRequestStatus `json:"status,omitempty"` +} + +// ProvisioningRequestList is a object for list of ProvisioningRequest. +// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object +type ProvisioningRequestList struct { + metav1.TypeMeta `json:",inline"` + // Standard list metadata. + // + // +optional + metav1.ListMeta `json:"metadata"` + // Items, list of ProvisioningRequest returned from API. + // + // +optional + Items []ProvisioningRequest `json:"items"` +} + +// ProvisioningRequestSpec is a specification of additional pods for which we +// would like to provision additional resources in the cluster. +type ProvisioningRequestSpec struct { + // PodSets lists groups of pods for which we would like to provision + // resources. + // + // +kubebuilder:validation:Required + // +kubebuilder:validation:MinItems=1 + // +kubebuilder:validation:MaxItems=32 + // +kubebuilder:validation:XValidation:rule="self == oldSelf",message="Value is immutable" + PodSets []PodSet `json:"podSets"` + + // ProvisioningClassName describes the different modes of provisioning the resources. + // Currently there is no support for 'ProvisioningClass' objects. + // Supported values: + // * check-capacity.kubernetes.io - check if current cluster state can fullfil this request, + // do not reserve the capacity. Users should provide a reference to a valid PodTemplate object. + // CA will check if there is enough capacity in cluster to fulfill the request and put + // the answer in 'CapacityAvailable' condition. + // * atomic-scale-up.kubernetes.io - provision the resources in an atomic manner. + // Users should provide a reference to a valid PodTemplate object. + // CA will try to create the VMs in an atomic manner, clean any partially provisioned VMs + // and re-try the operation in a exponential back-off manner. Users can configure the timeout + // duration after which the request will fail by 'ValidUntilSeconds' key in 'Parameters'. + // CA will set 'Failed=true' or 'Provisioned=true' condition according to the outcome. + // * ... - potential other classes that are specific to the cloud providers. + // 'kubernetes.io' suffix is reserved for the modes defined in Kubernetes projects. + // + // +kubebuilder:validation:Required + // +kubebuilder:validation:XValidation:rule="self == oldSelf",message="Value is immutable" + // +kubebuilder:validation:Pattern=`^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$` + // +kubebuilder:validation:MaxLength=253 + ProvisioningClassName string `json:"provisioningClassName"` + + // Parameters contains all other parameters classes may require. + // 'atomic-scale-up.kubernetes.io' supports 'ValidUntilSeconds' parameter, which should contain + // a string denoting duration for which we should retry (measured since creation fo the CR). + // + // +optional + // +kubebuilder:validation:XValidation:rule="self == oldSelf",message="Value is immutable" + // +kubebuilder:validation:MaxProperties=100 + Parameters map[string]Parameter `json:"parameters"` +} + +// Parameter is limited to 255 characters. +// +kubebuilder:validation:MaxLength=255 +type Parameter string + +// PodSet represents one group of pods for Provisioning Request to provision capacity. +type PodSet struct { + // PodTemplateRef is a reference to a PodTemplate object that is representing pods + // that will consume this reservation (must be within the same namespace). + // Users need to make sure that the fields relevant to scheduler (e.g. node selector tolerations) + // are consistent between this template and actual pods consuming the Provisioning Request. + // + // +kubebuilder:validation:Required + PodTemplateRef Reference `json:"podTemplateRef"` + // Count contains the number of pods that will be created with a given + // template. + // + // +kubebuilder:validation:Minimum=1 + Count int32 `json:"count"` +} + +// Reference represents reference to an object within the same namespace. +type Reference struct { + // Name of the referenced object. + // More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names#names + // + // +kubebuilder:validation:Required + // +kubebuilder:validation:Pattern=`^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$` + // +kubebuilder:validation:MaxLength=253 + Name string `json:"name,omitempty"` +} + +// ProvisioningRequestStatus represents the status of the resource reservation. +type ProvisioningRequestStatus struct { + // Conditions represent the observations of a Provisioning Request's + // current state. Those will contain information whether the capacity + // was found/created or if there were any issues. The condition types + // may differ between different provisioning classes. + // + // +listType=map + // +listMapKey=type + // +patchStrategy=merge + // +patchMergeKey=type + // +optional + Conditions []metav1.Condition `json:"conditions"` + + // ProvisioningClassDetails contains all other values custom provisioning classes may + // want to pass to end users. + // + // +optional + // +kubebuilder:validation:MaxProperties=64 + ProvisioningClassDetails map[string]Detail `json:"provisioningClassDetails"` +} + +// Detail is limited to 32768 characters. +// +kubebuilder:validation:MaxLength=32768 +type Detail string + +// The following constants list all currently available Conditions Type values. +// See: https://pkg.go.dev/k8s.io/apimachinery/pkg/apis/meta/v1#Condition +const ( + // Accepted indicates that the ProvisioningRequest was accepted by ClusterAutoscaler, + // so ClusterAutoscaler will attempt to provision the nodes for it. + Accepted string = "Accepted" + // BookingExpired indicates that the ProvisioningRequest had Provisioned condition before + // and capacity reservation time is expired. + BookingExpired string = "BookingExpired" + // CapacityRevoked indicates that requested resources are not longer valid. + CapacityRevoked string = "CapacityRevoked" + // Provisioned indicates that all of the requested resources were created + // and registered in the cluster. CA will set this condition when the + // VM creation finishes successfully. + Provisioned string = "Provisioned" + // Failed indicates that it is impossible to obtain resources to fulfill + // this ProvisioningRequest. + // Condition Reason and Message will contain more details about what failed. + Failed string = "Failed" +) + +const ( + // ProvisioningClassCheckCapacity denotes that CA will check if current cluster state can fulfill this request, + // and reserve the capacity for a specified time. + ProvisioningClassCheckCapacity string = "check-capacity.autoscaling.x-k8s.io" + // ProvisioningClassAtomicScaleUp denotes that CA try to provision the capacity + // in an atomic manner. + ProvisioningClassAtomicScaleUp string = "atomic-scale-up.autoscaling.x-k8s.io" +) diff --git a/cluster-autoscaler/provisioningrequest/apis/autoscaling.x-k8s.io/v1beta1/zz_generated.deepcopy.go b/cluster-autoscaler/apis/provisioningrequest/autoscaling.x-k8s.io/v1beta1/zz_generated.deepcopy.go similarity index 100% rename from cluster-autoscaler/provisioningrequest/apis/autoscaling.x-k8s.io/v1beta1/zz_generated.deepcopy.go rename to cluster-autoscaler/apis/provisioningrequest/autoscaling.x-k8s.io/v1beta1/zz_generated.deepcopy.go diff --git a/cluster-autoscaler/provisioningrequest/client/applyconfiguration/autoscaling.x-k8s.io/v1beta1/podset.go b/cluster-autoscaler/apis/provisioningrequest/client/applyconfiguration/autoscaling.x-k8s.io/v1beta1/podset.go similarity index 100% rename from cluster-autoscaler/provisioningrequest/client/applyconfiguration/autoscaling.x-k8s.io/v1beta1/podset.go rename to cluster-autoscaler/apis/provisioningrequest/client/applyconfiguration/autoscaling.x-k8s.io/v1beta1/podset.go diff --git a/cluster-autoscaler/provisioningrequest/client/applyconfiguration/autoscaling.x-k8s.io/v1beta1/provisioningrequest.go b/cluster-autoscaler/apis/provisioningrequest/client/applyconfiguration/autoscaling.x-k8s.io/v1beta1/provisioningrequest.go similarity index 100% rename from cluster-autoscaler/provisioningrequest/client/applyconfiguration/autoscaling.x-k8s.io/v1beta1/provisioningrequest.go rename to cluster-autoscaler/apis/provisioningrequest/client/applyconfiguration/autoscaling.x-k8s.io/v1beta1/provisioningrequest.go diff --git a/cluster-autoscaler/apis/provisioningrequest/client/applyconfiguration/autoscaling.x-k8s.io/v1beta1/provisioningrequestspec.go b/cluster-autoscaler/apis/provisioningrequest/client/applyconfiguration/autoscaling.x-k8s.io/v1beta1/provisioningrequestspec.go new file mode 100644 index 000000000000..1c523729260e --- /dev/null +++ b/cluster-autoscaler/apis/provisioningrequest/client/applyconfiguration/autoscaling.x-k8s.io/v1beta1/provisioningrequestspec.go @@ -0,0 +1,72 @@ +/* +Copyright The Kubernetes Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// Code generated by applyconfiguration-gen. DO NOT EDIT. + +package v1beta1 + +import ( + autoscalingxk8siov1beta1 "k8s.io/autoscaler/cluster-autoscaler/apis/provisioningrequest/autoscaling.x-k8s.io/v1beta1" +) + +// ProvisioningRequestSpecApplyConfiguration represents an declarative configuration of the ProvisioningRequestSpec type for use +// with apply. +type ProvisioningRequestSpecApplyConfiguration struct { + PodSets []PodSetApplyConfiguration `json:"podSets,omitempty"` + ProvisioningClassName *string `json:"provisioningClassName,omitempty"` + Parameters map[string]autoscalingxk8siov1beta1.Parameter `json:"parameters,omitempty"` +} + +// ProvisioningRequestSpecApplyConfiguration constructs an declarative configuration of the ProvisioningRequestSpec type for use with +// apply. +func ProvisioningRequestSpec() *ProvisioningRequestSpecApplyConfiguration { + return &ProvisioningRequestSpecApplyConfiguration{} +} + +// WithPodSets adds the given value to the PodSets field in the declarative configuration +// and returns the receiver, so that objects can be build by chaining "With" function invocations. +// If called multiple times, values provided by each call will be appended to the PodSets field. +func (b *ProvisioningRequestSpecApplyConfiguration) WithPodSets(values ...*PodSetApplyConfiguration) *ProvisioningRequestSpecApplyConfiguration { + for i := range values { + if values[i] == nil { + panic("nil value passed to WithPodSets") + } + b.PodSets = append(b.PodSets, *values[i]) + } + return b +} + +// WithProvisioningClassName sets the ProvisioningClassName field in the declarative configuration to the given value +// and returns the receiver, so that objects can be built by chaining "With" function invocations. +// If called multiple times, the ProvisioningClassName field is set to the value of the last call. +func (b *ProvisioningRequestSpecApplyConfiguration) WithProvisioningClassName(value string) *ProvisioningRequestSpecApplyConfiguration { + b.ProvisioningClassName = &value + return b +} + +// WithParameters puts the entries into the Parameters field in the declarative configuration +// and returns the receiver, so that objects can be build by chaining "With" function invocations. +// If called multiple times, the entries provided by each call will be put on the Parameters field, +// overwriting an existing map entries in Parameters field with the same key. +func (b *ProvisioningRequestSpecApplyConfiguration) WithParameters(entries map[string]autoscalingxk8siov1beta1.Parameter) *ProvisioningRequestSpecApplyConfiguration { + if b.Parameters == nil && len(entries) > 0 { + b.Parameters = make(map[string]autoscalingxk8siov1beta1.Parameter, len(entries)) + } + for k, v := range entries { + b.Parameters[k] = v + } + return b +} diff --git a/cluster-autoscaler/apis/provisioningrequest/client/applyconfiguration/autoscaling.x-k8s.io/v1beta1/provisioningrequeststatus.go b/cluster-autoscaler/apis/provisioningrequest/client/applyconfiguration/autoscaling.x-k8s.io/v1beta1/provisioningrequeststatus.go new file mode 100644 index 000000000000..f949fffafd22 --- /dev/null +++ b/cluster-autoscaler/apis/provisioningrequest/client/applyconfiguration/autoscaling.x-k8s.io/v1beta1/provisioningrequeststatus.go @@ -0,0 +1,61 @@ +/* +Copyright The Kubernetes Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// Code generated by applyconfiguration-gen. DO NOT EDIT. + +package v1beta1 + +import ( + v1 "k8s.io/apimachinery/pkg/apis/meta/v1" + v1beta1 "k8s.io/autoscaler/cluster-autoscaler/apis/provisioningrequest/autoscaling.x-k8s.io/v1beta1" +) + +// ProvisioningRequestStatusApplyConfiguration represents an declarative configuration of the ProvisioningRequestStatus type for use +// with apply. +type ProvisioningRequestStatusApplyConfiguration struct { + Conditions []v1.Condition `json:"conditions,omitempty"` + ProvisioningClassDetails map[string]v1beta1.Detail `json:"provisioningClassDetails,omitempty"` +} + +// ProvisioningRequestStatusApplyConfiguration constructs an declarative configuration of the ProvisioningRequestStatus type for use with +// apply. +func ProvisioningRequestStatus() *ProvisioningRequestStatusApplyConfiguration { + return &ProvisioningRequestStatusApplyConfiguration{} +} + +// WithConditions adds the given value to the Conditions field in the declarative configuration +// and returns the receiver, so that objects can be build by chaining "With" function invocations. +// If called multiple times, values provided by each call will be appended to the Conditions field. +func (b *ProvisioningRequestStatusApplyConfiguration) WithConditions(values ...v1.Condition) *ProvisioningRequestStatusApplyConfiguration { + for i := range values { + b.Conditions = append(b.Conditions, values[i]) + } + return b +} + +// WithProvisioningClassDetails puts the entries into the ProvisioningClassDetails field in the declarative configuration +// and returns the receiver, so that objects can be build by chaining "With" function invocations. +// If called multiple times, the entries provided by each call will be put on the ProvisioningClassDetails field, +// overwriting an existing map entries in ProvisioningClassDetails field with the same key. +func (b *ProvisioningRequestStatusApplyConfiguration) WithProvisioningClassDetails(entries map[string]v1beta1.Detail) *ProvisioningRequestStatusApplyConfiguration { + if b.ProvisioningClassDetails == nil && len(entries) > 0 { + b.ProvisioningClassDetails = make(map[string]v1beta1.Detail, len(entries)) + } + for k, v := range entries { + b.ProvisioningClassDetails[k] = v + } + return b +} diff --git a/cluster-autoscaler/provisioningrequest/client/applyconfiguration/autoscaling.x-k8s.io/v1beta1/reference.go b/cluster-autoscaler/apis/provisioningrequest/client/applyconfiguration/autoscaling.x-k8s.io/v1beta1/reference.go similarity index 100% rename from cluster-autoscaler/provisioningrequest/client/applyconfiguration/autoscaling.x-k8s.io/v1beta1/reference.go rename to cluster-autoscaler/apis/provisioningrequest/client/applyconfiguration/autoscaling.x-k8s.io/v1beta1/reference.go diff --git a/cluster-autoscaler/provisioningrequest/client/applyconfiguration/internal/internal.go b/cluster-autoscaler/apis/provisioningrequest/client/applyconfiguration/internal/internal.go similarity index 100% rename from cluster-autoscaler/provisioningrequest/client/applyconfiguration/internal/internal.go rename to cluster-autoscaler/apis/provisioningrequest/client/applyconfiguration/internal/internal.go diff --git a/cluster-autoscaler/apis/provisioningrequest/client/applyconfiguration/utils.go b/cluster-autoscaler/apis/provisioningrequest/client/applyconfiguration/utils.go new file mode 100644 index 000000000000..894c18a44998 --- /dev/null +++ b/cluster-autoscaler/apis/provisioningrequest/client/applyconfiguration/utils.go @@ -0,0 +1,45 @@ +/* +Copyright The Kubernetes Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// Code generated by applyconfiguration-gen. DO NOT EDIT. + +package applyconfiguration + +import ( + schema "k8s.io/apimachinery/pkg/runtime/schema" + v1beta1 "k8s.io/autoscaler/cluster-autoscaler/apis/provisioningrequest/autoscaling.x-k8s.io/v1beta1" + autoscalingxk8siov1beta1 "k8s.io/autoscaler/cluster-autoscaler/apis/provisioningrequest/client/applyconfiguration/autoscaling.x-k8s.io/v1beta1" +) + +// ForKind returns an apply configuration type for the given GroupVersionKind, or nil if no +// apply configuration type exists for the given GroupVersionKind. +func ForKind(kind schema.GroupVersionKind) interface{} { + switch kind { + // Group=autoscaling.x-k8s.io, Version=v1beta1 + case v1beta1.SchemeGroupVersion.WithKind("PodSet"): + return &autoscalingxk8siov1beta1.PodSetApplyConfiguration{} + case v1beta1.SchemeGroupVersion.WithKind("ProvisioningRequest"): + return &autoscalingxk8siov1beta1.ProvisioningRequestApplyConfiguration{} + case v1beta1.SchemeGroupVersion.WithKind("ProvisioningRequestSpec"): + return &autoscalingxk8siov1beta1.ProvisioningRequestSpecApplyConfiguration{} + case v1beta1.SchemeGroupVersion.WithKind("ProvisioningRequestStatus"): + return &autoscalingxk8siov1beta1.ProvisioningRequestStatusApplyConfiguration{} + case v1beta1.SchemeGroupVersion.WithKind("Reference"): + return &autoscalingxk8siov1beta1.ReferenceApplyConfiguration{} + + } + return nil +} diff --git a/cluster-autoscaler/apis/provisioningrequest/client/clientset/versioned/clientset.go b/cluster-autoscaler/apis/provisioningrequest/client/clientset/versioned/clientset.go new file mode 100644 index 000000000000..2a940c1ac12d --- /dev/null +++ b/cluster-autoscaler/apis/provisioningrequest/client/clientset/versioned/clientset.go @@ -0,0 +1,120 @@ +/* +Copyright The Kubernetes Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// Code generated by client-gen. DO NOT EDIT. + +package versioned + +import ( + "fmt" + "net/http" + + autoscalingv1beta1 "k8s.io/autoscaler/cluster-autoscaler/apis/provisioningrequest/client/clientset/versioned/typed/autoscaling.x-k8s.io/v1beta1" + discovery "k8s.io/client-go/discovery" + rest "k8s.io/client-go/rest" + flowcontrol "k8s.io/client-go/util/flowcontrol" +) + +type Interface interface { + Discovery() discovery.DiscoveryInterface + AutoscalingV1beta1() autoscalingv1beta1.AutoscalingV1beta1Interface +} + +// Clientset contains the clients for groups. +type Clientset struct { + *discovery.DiscoveryClient + autoscalingV1beta1 *autoscalingv1beta1.AutoscalingV1beta1Client +} + +// AutoscalingV1beta1 retrieves the AutoscalingV1beta1Client +func (c *Clientset) AutoscalingV1beta1() autoscalingv1beta1.AutoscalingV1beta1Interface { + return c.autoscalingV1beta1 +} + +// Discovery retrieves the DiscoveryClient +func (c *Clientset) Discovery() discovery.DiscoveryInterface { + if c == nil { + return nil + } + return c.DiscoveryClient +} + +// NewForConfig creates a new Clientset for the given config. +// If config's RateLimiter is not set and QPS and Burst are acceptable, +// NewForConfig will generate a rate-limiter in configShallowCopy. +// NewForConfig is equivalent to NewForConfigAndClient(c, httpClient), +// where httpClient was generated with rest.HTTPClientFor(c). +func NewForConfig(c *rest.Config) (*Clientset, error) { + configShallowCopy := *c + + if configShallowCopy.UserAgent == "" { + configShallowCopy.UserAgent = rest.DefaultKubernetesUserAgent() + } + + // share the transport between all clients + httpClient, err := rest.HTTPClientFor(&configShallowCopy) + if err != nil { + return nil, err + } + + return NewForConfigAndClient(&configShallowCopy, httpClient) +} + +// NewForConfigAndClient creates a new Clientset for the given config and http client. +// Note the http client provided takes precedence over the configured transport values. +// If config's RateLimiter is not set and QPS and Burst are acceptable, +// NewForConfigAndClient will generate a rate-limiter in configShallowCopy. +func NewForConfigAndClient(c *rest.Config, httpClient *http.Client) (*Clientset, error) { + configShallowCopy := *c + if configShallowCopy.RateLimiter == nil && configShallowCopy.QPS > 0 { + if configShallowCopy.Burst <= 0 { + return nil, fmt.Errorf("burst is required to be greater than 0 when RateLimiter is not set and QPS is set to greater than 0") + } + configShallowCopy.RateLimiter = flowcontrol.NewTokenBucketRateLimiter(configShallowCopy.QPS, configShallowCopy.Burst) + } + + var cs Clientset + var err error + cs.autoscalingV1beta1, err = autoscalingv1beta1.NewForConfigAndClient(&configShallowCopy, httpClient) + if err != nil { + return nil, err + } + + cs.DiscoveryClient, err = discovery.NewDiscoveryClientForConfigAndClient(&configShallowCopy, httpClient) + if err != nil { + return nil, err + } + return &cs, nil +} + +// NewForConfigOrDie creates a new Clientset for the given config and +// panics if there is an error in the config. +func NewForConfigOrDie(c *rest.Config) *Clientset { + cs, err := NewForConfig(c) + if err != nil { + panic(err) + } + return cs +} + +// New creates a new Clientset for the given RESTClient. +func New(c rest.Interface) *Clientset { + var cs Clientset + cs.autoscalingV1beta1 = autoscalingv1beta1.New(c) + + cs.DiscoveryClient = discovery.NewDiscoveryClient(c) + return &cs +} diff --git a/cluster-autoscaler/apis/provisioningrequest/client/clientset/versioned/fake/clientset_generated.go b/cluster-autoscaler/apis/provisioningrequest/client/clientset/versioned/fake/clientset_generated.go new file mode 100644 index 000000000000..eb8263030374 --- /dev/null +++ b/cluster-autoscaler/apis/provisioningrequest/client/clientset/versioned/fake/clientset_generated.go @@ -0,0 +1,85 @@ +/* +Copyright The Kubernetes Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// Code generated by client-gen. DO NOT EDIT. + +package fake + +import ( + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/watch" + clientset "k8s.io/autoscaler/cluster-autoscaler/apis/provisioningrequest/client/clientset/versioned" + autoscalingv1beta1 "k8s.io/autoscaler/cluster-autoscaler/apis/provisioningrequest/client/clientset/versioned/typed/autoscaling.x-k8s.io/v1beta1" + fakeautoscalingv1beta1 "k8s.io/autoscaler/cluster-autoscaler/apis/provisioningrequest/client/clientset/versioned/typed/autoscaling.x-k8s.io/v1beta1/fake" + "k8s.io/client-go/discovery" + fakediscovery "k8s.io/client-go/discovery/fake" + "k8s.io/client-go/testing" +) + +// NewSimpleClientset returns a clientset that will respond with the provided objects. +// It's backed by a very simple object tracker that processes creates, updates and deletions as-is, +// without applying any validations and/or defaults. It shouldn't be considered a replacement +// for a real clientset and is mostly useful in simple unit tests. +func NewSimpleClientset(objects ...runtime.Object) *Clientset { + o := testing.NewObjectTracker(scheme, codecs.UniversalDecoder()) + for _, obj := range objects { + if err := o.Add(obj); err != nil { + panic(err) + } + } + + cs := &Clientset{tracker: o} + cs.discovery = &fakediscovery.FakeDiscovery{Fake: &cs.Fake} + cs.AddReactor("*", "*", testing.ObjectReaction(o)) + cs.AddWatchReactor("*", func(action testing.Action) (handled bool, ret watch.Interface, err error) { + gvr := action.GetResource() + ns := action.GetNamespace() + watch, err := o.Watch(gvr, ns) + if err != nil { + return false, nil, err + } + return true, watch, nil + }) + + return cs +} + +// Clientset implements clientset.Interface. Meant to be embedded into a +// struct to get a default implementation. This makes faking out just the method +// you want to test easier. +type Clientset struct { + testing.Fake + discovery *fakediscovery.FakeDiscovery + tracker testing.ObjectTracker +} + +func (c *Clientset) Discovery() discovery.DiscoveryInterface { + return c.discovery +} + +func (c *Clientset) Tracker() testing.ObjectTracker { + return c.tracker +} + +var ( + _ clientset.Interface = &Clientset{} + _ testing.FakeClient = &Clientset{} +) + +// AutoscalingV1beta1 retrieves the AutoscalingV1beta1Client +func (c *Clientset) AutoscalingV1beta1() autoscalingv1beta1.AutoscalingV1beta1Interface { + return &fakeautoscalingv1beta1.FakeAutoscalingV1beta1{Fake: &c.Fake} +} diff --git a/cluster-autoscaler/provisioningrequest/client/clientset/versioned/fake/doc.go b/cluster-autoscaler/apis/provisioningrequest/client/clientset/versioned/fake/doc.go similarity index 100% rename from cluster-autoscaler/provisioningrequest/client/clientset/versioned/fake/doc.go rename to cluster-autoscaler/apis/provisioningrequest/client/clientset/versioned/fake/doc.go diff --git a/cluster-autoscaler/apis/provisioningrequest/client/clientset/versioned/fake/register.go b/cluster-autoscaler/apis/provisioningrequest/client/clientset/versioned/fake/register.go new file mode 100644 index 000000000000..63b931ff9509 --- /dev/null +++ b/cluster-autoscaler/apis/provisioningrequest/client/clientset/versioned/fake/register.go @@ -0,0 +1,56 @@ +/* +Copyright The Kubernetes Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// Code generated by client-gen. DO NOT EDIT. + +package fake + +import ( + v1 "k8s.io/apimachinery/pkg/apis/meta/v1" + runtime "k8s.io/apimachinery/pkg/runtime" + schema "k8s.io/apimachinery/pkg/runtime/schema" + serializer "k8s.io/apimachinery/pkg/runtime/serializer" + utilruntime "k8s.io/apimachinery/pkg/util/runtime" + autoscalingv1beta1 "k8s.io/autoscaler/cluster-autoscaler/apis/provisioningrequest/autoscaling.x-k8s.io/v1beta1" +) + +var scheme = runtime.NewScheme() +var codecs = serializer.NewCodecFactory(scheme) + +var localSchemeBuilder = runtime.SchemeBuilder{ + autoscalingv1beta1.AddToScheme, +} + +// AddToScheme adds all types of this clientset into the given scheme. This allows composition +// of clientsets, like in: +// +// import ( +// "k8s.io/client-go/kubernetes" +// clientsetscheme "k8s.io/client-go/kubernetes/scheme" +// aggregatorclientsetscheme "k8s.io/kube-aggregator/pkg/client/clientset_generated/clientset/scheme" +// ) +// +// kclientset, _ := kubernetes.NewForConfig(c) +// _ = aggregatorclientsetscheme.AddToScheme(clientsetscheme.Scheme) +// +// After this, RawExtensions in Kubernetes types will serialize kube-aggregator types +// correctly. +var AddToScheme = localSchemeBuilder.AddToScheme + +func init() { + v1.AddToGroupVersion(scheme, schema.GroupVersion{Version: "v1"}) + utilruntime.Must(AddToScheme(scheme)) +} diff --git a/cluster-autoscaler/provisioningrequest/client/clientset/versioned/scheme/doc.go b/cluster-autoscaler/apis/provisioningrequest/client/clientset/versioned/scheme/doc.go similarity index 100% rename from cluster-autoscaler/provisioningrequest/client/clientset/versioned/scheme/doc.go rename to cluster-autoscaler/apis/provisioningrequest/client/clientset/versioned/scheme/doc.go diff --git a/cluster-autoscaler/apis/provisioningrequest/client/clientset/versioned/scheme/register.go b/cluster-autoscaler/apis/provisioningrequest/client/clientset/versioned/scheme/register.go new file mode 100644 index 000000000000..ee8998ec2364 --- /dev/null +++ b/cluster-autoscaler/apis/provisioningrequest/client/clientset/versioned/scheme/register.go @@ -0,0 +1,56 @@ +/* +Copyright The Kubernetes Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// Code generated by client-gen. DO NOT EDIT. + +package scheme + +import ( + v1 "k8s.io/apimachinery/pkg/apis/meta/v1" + runtime "k8s.io/apimachinery/pkg/runtime" + schema "k8s.io/apimachinery/pkg/runtime/schema" + serializer "k8s.io/apimachinery/pkg/runtime/serializer" + utilruntime "k8s.io/apimachinery/pkg/util/runtime" + autoscalingv1beta1 "k8s.io/autoscaler/cluster-autoscaler/apis/provisioningrequest/autoscaling.x-k8s.io/v1beta1" +) + +var Scheme = runtime.NewScheme() +var Codecs = serializer.NewCodecFactory(Scheme) +var ParameterCodec = runtime.NewParameterCodec(Scheme) +var localSchemeBuilder = runtime.SchemeBuilder{ + autoscalingv1beta1.AddToScheme, +} + +// AddToScheme adds all types of this clientset into the given scheme. This allows composition +// of clientsets, like in: +// +// import ( +// "k8s.io/client-go/kubernetes" +// clientsetscheme "k8s.io/client-go/kubernetes/scheme" +// aggregatorclientsetscheme "k8s.io/kube-aggregator/pkg/client/clientset_generated/clientset/scheme" +// ) +// +// kclientset, _ := kubernetes.NewForConfig(c) +// _ = aggregatorclientsetscheme.AddToScheme(clientsetscheme.Scheme) +// +// After this, RawExtensions in Kubernetes types will serialize kube-aggregator types +// correctly. +var AddToScheme = localSchemeBuilder.AddToScheme + +func init() { + v1.AddToGroupVersion(Scheme, schema.GroupVersion{Version: "v1"}) + utilruntime.Must(AddToScheme(Scheme)) +} diff --git a/cluster-autoscaler/apis/provisioningrequest/client/clientset/versioned/typed/autoscaling.x-k8s.io/v1beta1/autoscaling.x-k8s.io_client.go b/cluster-autoscaler/apis/provisioningrequest/client/clientset/versioned/typed/autoscaling.x-k8s.io/v1beta1/autoscaling.x-k8s.io_client.go new file mode 100644 index 000000000000..8f7a4c3497af --- /dev/null +++ b/cluster-autoscaler/apis/provisioningrequest/client/clientset/versioned/typed/autoscaling.x-k8s.io/v1beta1/autoscaling.x-k8s.io_client.go @@ -0,0 +1,107 @@ +/* +Copyright The Kubernetes Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// Code generated by client-gen. DO NOT EDIT. + +package v1beta1 + +import ( + "net/http" + + v1beta1 "k8s.io/autoscaler/cluster-autoscaler/apis/provisioningrequest/autoscaling.x-k8s.io/v1beta1" + "k8s.io/autoscaler/cluster-autoscaler/apis/provisioningrequest/client/clientset/versioned/scheme" + rest "k8s.io/client-go/rest" +) + +type AutoscalingV1beta1Interface interface { + RESTClient() rest.Interface + ProvisioningRequestsGetter +} + +// AutoscalingV1beta1Client is used to interact with features provided by the autoscaling.x-k8s.io group. +type AutoscalingV1beta1Client struct { + restClient rest.Interface +} + +func (c *AutoscalingV1beta1Client) ProvisioningRequests(namespace string) ProvisioningRequestInterface { + return newProvisioningRequests(c, namespace) +} + +// NewForConfig creates a new AutoscalingV1beta1Client for the given config. +// NewForConfig is equivalent to NewForConfigAndClient(c, httpClient), +// where httpClient was generated with rest.HTTPClientFor(c). +func NewForConfig(c *rest.Config) (*AutoscalingV1beta1Client, error) { + config := *c + if err := setConfigDefaults(&config); err != nil { + return nil, err + } + httpClient, err := rest.HTTPClientFor(&config) + if err != nil { + return nil, err + } + return NewForConfigAndClient(&config, httpClient) +} + +// NewForConfigAndClient creates a new AutoscalingV1beta1Client for the given config and http client. +// Note the http client provided takes precedence over the configured transport values. +func NewForConfigAndClient(c *rest.Config, h *http.Client) (*AutoscalingV1beta1Client, error) { + config := *c + if err := setConfigDefaults(&config); err != nil { + return nil, err + } + client, err := rest.RESTClientForConfigAndClient(&config, h) + if err != nil { + return nil, err + } + return &AutoscalingV1beta1Client{client}, nil +} + +// NewForConfigOrDie creates a new AutoscalingV1beta1Client for the given config and +// panics if there is an error in the config. +func NewForConfigOrDie(c *rest.Config) *AutoscalingV1beta1Client { + client, err := NewForConfig(c) + if err != nil { + panic(err) + } + return client +} + +// New creates a new AutoscalingV1beta1Client for the given RESTClient. +func New(c rest.Interface) *AutoscalingV1beta1Client { + return &AutoscalingV1beta1Client{c} +} + +func setConfigDefaults(config *rest.Config) error { + gv := v1beta1.SchemeGroupVersion + config.GroupVersion = &gv + config.APIPath = "/apis" + config.NegotiatedSerializer = scheme.Codecs.WithoutConversion() + + if config.UserAgent == "" { + config.UserAgent = rest.DefaultKubernetesUserAgent() + } + + return nil +} + +// RESTClient returns a RESTClient that is used to communicate +// with API server by this client implementation. +func (c *AutoscalingV1beta1Client) RESTClient() rest.Interface { + if c == nil { + return nil + } + return c.restClient +} diff --git a/cluster-autoscaler/provisioningrequest/client/clientset/versioned/typed/autoscaling.x-k8s.io/v1beta1/doc.go b/cluster-autoscaler/apis/provisioningrequest/client/clientset/versioned/typed/autoscaling.x-k8s.io/v1beta1/doc.go similarity index 100% rename from cluster-autoscaler/provisioningrequest/client/clientset/versioned/typed/autoscaling.x-k8s.io/v1beta1/doc.go rename to cluster-autoscaler/apis/provisioningrequest/client/clientset/versioned/typed/autoscaling.x-k8s.io/v1beta1/doc.go diff --git a/cluster-autoscaler/provisioningrequest/client/clientset/versioned/typed/autoscaling.x-k8s.io/v1beta1/fake/doc.go b/cluster-autoscaler/apis/provisioningrequest/client/clientset/versioned/typed/autoscaling.x-k8s.io/v1beta1/fake/doc.go similarity index 100% rename from cluster-autoscaler/provisioningrequest/client/clientset/versioned/typed/autoscaling.x-k8s.io/v1beta1/fake/doc.go rename to cluster-autoscaler/apis/provisioningrequest/client/clientset/versioned/typed/autoscaling.x-k8s.io/v1beta1/fake/doc.go diff --git a/cluster-autoscaler/apis/provisioningrequest/client/clientset/versioned/typed/autoscaling.x-k8s.io/v1beta1/fake/fake_autoscaling.x-k8s.io_client.go b/cluster-autoscaler/apis/provisioningrequest/client/clientset/versioned/typed/autoscaling.x-k8s.io/v1beta1/fake/fake_autoscaling.x-k8s.io_client.go new file mode 100644 index 000000000000..33d0774402fc --- /dev/null +++ b/cluster-autoscaler/apis/provisioningrequest/client/clientset/versioned/typed/autoscaling.x-k8s.io/v1beta1/fake/fake_autoscaling.x-k8s.io_client.go @@ -0,0 +1,40 @@ +/* +Copyright The Kubernetes Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// Code generated by client-gen. DO NOT EDIT. + +package fake + +import ( + v1beta1 "k8s.io/autoscaler/cluster-autoscaler/apis/provisioningrequest/client/clientset/versioned/typed/autoscaling.x-k8s.io/v1beta1" + rest "k8s.io/client-go/rest" + testing "k8s.io/client-go/testing" +) + +type FakeAutoscalingV1beta1 struct { + *testing.Fake +} + +func (c *FakeAutoscalingV1beta1) ProvisioningRequests(namespace string) v1beta1.ProvisioningRequestInterface { + return &FakeProvisioningRequests{c, namespace} +} + +// RESTClient returns a RESTClient that is used to communicate +// with API server by this client implementation. +func (c *FakeAutoscalingV1beta1) RESTClient() rest.Interface { + var ret *rest.RESTClient + return ret +} diff --git a/cluster-autoscaler/apis/provisioningrequest/client/clientset/versioned/typed/autoscaling.x-k8s.io/v1beta1/fake/fake_provisioningrequest.go b/cluster-autoscaler/apis/provisioningrequest/client/clientset/versioned/typed/autoscaling.x-k8s.io/v1beta1/fake/fake_provisioningrequest.go new file mode 100644 index 000000000000..0ec6755ad4ba --- /dev/null +++ b/cluster-autoscaler/apis/provisioningrequest/client/clientset/versioned/typed/autoscaling.x-k8s.io/v1beta1/fake/fake_provisioningrequest.go @@ -0,0 +1,189 @@ +/* +Copyright The Kubernetes Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// Code generated by client-gen. DO NOT EDIT. + +package fake + +import ( + "context" + json "encoding/json" + "fmt" + + v1 "k8s.io/apimachinery/pkg/apis/meta/v1" + labels "k8s.io/apimachinery/pkg/labels" + types "k8s.io/apimachinery/pkg/types" + watch "k8s.io/apimachinery/pkg/watch" + v1beta1 "k8s.io/autoscaler/cluster-autoscaler/apis/provisioningrequest/autoscaling.x-k8s.io/v1beta1" + autoscalingxk8siov1beta1 "k8s.io/autoscaler/cluster-autoscaler/apis/provisioningrequest/client/applyconfiguration/autoscaling.x-k8s.io/v1beta1" + testing "k8s.io/client-go/testing" +) + +// FakeProvisioningRequests implements ProvisioningRequestInterface +type FakeProvisioningRequests struct { + Fake *FakeAutoscalingV1beta1 + ns string +} + +var provisioningrequestsResource = v1beta1.SchemeGroupVersion.WithResource("provisioningrequests") + +var provisioningrequestsKind = v1beta1.SchemeGroupVersion.WithKind("ProvisioningRequest") + +// Get takes name of the provisioningRequest, and returns the corresponding provisioningRequest object, and an error if there is any. +func (c *FakeProvisioningRequests) Get(ctx context.Context, name string, options v1.GetOptions) (result *v1beta1.ProvisioningRequest, err error) { + obj, err := c.Fake. + Invokes(testing.NewGetAction(provisioningrequestsResource, c.ns, name), &v1beta1.ProvisioningRequest{}) + + if obj == nil { + return nil, err + } + return obj.(*v1beta1.ProvisioningRequest), err +} + +// List takes label and field selectors, and returns the list of ProvisioningRequests that match those selectors. +func (c *FakeProvisioningRequests) List(ctx context.Context, opts v1.ListOptions) (result *v1beta1.ProvisioningRequestList, err error) { + obj, err := c.Fake. + Invokes(testing.NewListAction(provisioningrequestsResource, provisioningrequestsKind, c.ns, opts), &v1beta1.ProvisioningRequestList{}) + + if obj == nil { + return nil, err + } + + label, _, _ := testing.ExtractFromListOptions(opts) + if label == nil { + label = labels.Everything() + } + list := &v1beta1.ProvisioningRequestList{ListMeta: obj.(*v1beta1.ProvisioningRequestList).ListMeta} + for _, item := range obj.(*v1beta1.ProvisioningRequestList).Items { + if label.Matches(labels.Set(item.Labels)) { + list.Items = append(list.Items, item) + } + } + return list, err +} + +// Watch returns a watch.Interface that watches the requested provisioningRequests. +func (c *FakeProvisioningRequests) Watch(ctx context.Context, opts v1.ListOptions) (watch.Interface, error) { + return c.Fake. + InvokesWatch(testing.NewWatchAction(provisioningrequestsResource, c.ns, opts)) + +} + +// Create takes the representation of a provisioningRequest and creates it. Returns the server's representation of the provisioningRequest, and an error, if there is any. +func (c *FakeProvisioningRequests) Create(ctx context.Context, provisioningRequest *v1beta1.ProvisioningRequest, opts v1.CreateOptions) (result *v1beta1.ProvisioningRequest, err error) { + obj, err := c.Fake. + Invokes(testing.NewCreateAction(provisioningrequestsResource, c.ns, provisioningRequest), &v1beta1.ProvisioningRequest{}) + + if obj == nil { + return nil, err + } + return obj.(*v1beta1.ProvisioningRequest), err +} + +// Update takes the representation of a provisioningRequest and updates it. Returns the server's representation of the provisioningRequest, and an error, if there is any. +func (c *FakeProvisioningRequests) Update(ctx context.Context, provisioningRequest *v1beta1.ProvisioningRequest, opts v1.UpdateOptions) (result *v1beta1.ProvisioningRequest, err error) { + obj, err := c.Fake. + Invokes(testing.NewUpdateAction(provisioningrequestsResource, c.ns, provisioningRequest), &v1beta1.ProvisioningRequest{}) + + if obj == nil { + return nil, err + } + return obj.(*v1beta1.ProvisioningRequest), err +} + +// UpdateStatus was generated because the type contains a Status member. +// Add a +genclient:noStatus comment above the type to avoid generating UpdateStatus(). +func (c *FakeProvisioningRequests) UpdateStatus(ctx context.Context, provisioningRequest *v1beta1.ProvisioningRequest, opts v1.UpdateOptions) (*v1beta1.ProvisioningRequest, error) { + obj, err := c.Fake. + Invokes(testing.NewUpdateSubresourceAction(provisioningrequestsResource, "status", c.ns, provisioningRequest), &v1beta1.ProvisioningRequest{}) + + if obj == nil { + return nil, err + } + return obj.(*v1beta1.ProvisioningRequest), err +} + +// Delete takes name of the provisioningRequest and deletes it. Returns an error if one occurs. +func (c *FakeProvisioningRequests) Delete(ctx context.Context, name string, opts v1.DeleteOptions) error { + _, err := c.Fake. + Invokes(testing.NewDeleteActionWithOptions(provisioningrequestsResource, c.ns, name, opts), &v1beta1.ProvisioningRequest{}) + + return err +} + +// DeleteCollection deletes a collection of objects. +func (c *FakeProvisioningRequests) DeleteCollection(ctx context.Context, opts v1.DeleteOptions, listOpts v1.ListOptions) error { + action := testing.NewDeleteCollectionAction(provisioningrequestsResource, c.ns, listOpts) + + _, err := c.Fake.Invokes(action, &v1beta1.ProvisioningRequestList{}) + return err +} + +// Patch applies the patch and returns the patched provisioningRequest. +func (c *FakeProvisioningRequests) Patch(ctx context.Context, name string, pt types.PatchType, data []byte, opts v1.PatchOptions, subresources ...string) (result *v1beta1.ProvisioningRequest, err error) { + obj, err := c.Fake. + Invokes(testing.NewPatchSubresourceAction(provisioningrequestsResource, c.ns, name, pt, data, subresources...), &v1beta1.ProvisioningRequest{}) + + if obj == nil { + return nil, err + } + return obj.(*v1beta1.ProvisioningRequest), err +} + +// Apply takes the given apply declarative configuration, applies it and returns the applied provisioningRequest. +func (c *FakeProvisioningRequests) Apply(ctx context.Context, provisioningRequest *autoscalingxk8siov1beta1.ProvisioningRequestApplyConfiguration, opts v1.ApplyOptions) (result *v1beta1.ProvisioningRequest, err error) { + if provisioningRequest == nil { + return nil, fmt.Errorf("provisioningRequest provided to Apply must not be nil") + } + data, err := json.Marshal(provisioningRequest) + if err != nil { + return nil, err + } + name := provisioningRequest.Name + if name == nil { + return nil, fmt.Errorf("provisioningRequest.Name must be provided to Apply") + } + obj, err := c.Fake. + Invokes(testing.NewPatchSubresourceAction(provisioningrequestsResource, c.ns, *name, types.ApplyPatchType, data), &v1beta1.ProvisioningRequest{}) + + if obj == nil { + return nil, err + } + return obj.(*v1beta1.ProvisioningRequest), err +} + +// ApplyStatus was generated because the type contains a Status member. +// Add a +genclient:noStatus comment above the type to avoid generating ApplyStatus(). +func (c *FakeProvisioningRequests) ApplyStatus(ctx context.Context, provisioningRequest *autoscalingxk8siov1beta1.ProvisioningRequestApplyConfiguration, opts v1.ApplyOptions) (result *v1beta1.ProvisioningRequest, err error) { + if provisioningRequest == nil { + return nil, fmt.Errorf("provisioningRequest provided to Apply must not be nil") + } + data, err := json.Marshal(provisioningRequest) + if err != nil { + return nil, err + } + name := provisioningRequest.Name + if name == nil { + return nil, fmt.Errorf("provisioningRequest.Name must be provided to Apply") + } + obj, err := c.Fake. + Invokes(testing.NewPatchSubresourceAction(provisioningrequestsResource, c.ns, *name, types.ApplyPatchType, data, "status"), &v1beta1.ProvisioningRequest{}) + + if obj == nil { + return nil, err + } + return obj.(*v1beta1.ProvisioningRequest), err +} diff --git a/cluster-autoscaler/provisioningrequest/client/clientset/versioned/typed/autoscaling.x-k8s.io/v1beta1/generated_expansion.go b/cluster-autoscaler/apis/provisioningrequest/client/clientset/versioned/typed/autoscaling.x-k8s.io/v1beta1/generated_expansion.go similarity index 100% rename from cluster-autoscaler/provisioningrequest/client/clientset/versioned/typed/autoscaling.x-k8s.io/v1beta1/generated_expansion.go rename to cluster-autoscaler/apis/provisioningrequest/client/clientset/versioned/typed/autoscaling.x-k8s.io/v1beta1/generated_expansion.go diff --git a/cluster-autoscaler/apis/provisioningrequest/client/clientset/versioned/typed/autoscaling.x-k8s.io/v1beta1/provisioningrequest.go b/cluster-autoscaler/apis/provisioningrequest/client/clientset/versioned/typed/autoscaling.x-k8s.io/v1beta1/provisioningrequest.go new file mode 100644 index 000000000000..df19e1075a67 --- /dev/null +++ b/cluster-autoscaler/apis/provisioningrequest/client/clientset/versioned/typed/autoscaling.x-k8s.io/v1beta1/provisioningrequest.go @@ -0,0 +1,256 @@ +/* +Copyright The Kubernetes Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// Code generated by client-gen. DO NOT EDIT. + +package v1beta1 + +import ( + "context" + json "encoding/json" + "fmt" + "time" + + v1 "k8s.io/apimachinery/pkg/apis/meta/v1" + types "k8s.io/apimachinery/pkg/types" + watch "k8s.io/apimachinery/pkg/watch" + v1beta1 "k8s.io/autoscaler/cluster-autoscaler/apis/provisioningrequest/autoscaling.x-k8s.io/v1beta1" + autoscalingxk8siov1beta1 "k8s.io/autoscaler/cluster-autoscaler/apis/provisioningrequest/client/applyconfiguration/autoscaling.x-k8s.io/v1beta1" + scheme "k8s.io/autoscaler/cluster-autoscaler/apis/provisioningrequest/client/clientset/versioned/scheme" + rest "k8s.io/client-go/rest" +) + +// ProvisioningRequestsGetter has a method to return a ProvisioningRequestInterface. +// A group's client should implement this interface. +type ProvisioningRequestsGetter interface { + ProvisioningRequests(namespace string) ProvisioningRequestInterface +} + +// ProvisioningRequestInterface has methods to work with ProvisioningRequest resources. +type ProvisioningRequestInterface interface { + Create(ctx context.Context, provisioningRequest *v1beta1.ProvisioningRequest, opts v1.CreateOptions) (*v1beta1.ProvisioningRequest, error) + Update(ctx context.Context, provisioningRequest *v1beta1.ProvisioningRequest, opts v1.UpdateOptions) (*v1beta1.ProvisioningRequest, error) + UpdateStatus(ctx context.Context, provisioningRequest *v1beta1.ProvisioningRequest, opts v1.UpdateOptions) (*v1beta1.ProvisioningRequest, error) + Delete(ctx context.Context, name string, opts v1.DeleteOptions) error + DeleteCollection(ctx context.Context, opts v1.DeleteOptions, listOpts v1.ListOptions) error + Get(ctx context.Context, name string, opts v1.GetOptions) (*v1beta1.ProvisioningRequest, error) + List(ctx context.Context, opts v1.ListOptions) (*v1beta1.ProvisioningRequestList, error) + Watch(ctx context.Context, opts v1.ListOptions) (watch.Interface, error) + Patch(ctx context.Context, name string, pt types.PatchType, data []byte, opts v1.PatchOptions, subresources ...string) (result *v1beta1.ProvisioningRequest, err error) + Apply(ctx context.Context, provisioningRequest *autoscalingxk8siov1beta1.ProvisioningRequestApplyConfiguration, opts v1.ApplyOptions) (result *v1beta1.ProvisioningRequest, err error) + ApplyStatus(ctx context.Context, provisioningRequest *autoscalingxk8siov1beta1.ProvisioningRequestApplyConfiguration, opts v1.ApplyOptions) (result *v1beta1.ProvisioningRequest, err error) + ProvisioningRequestExpansion +} + +// provisioningRequests implements ProvisioningRequestInterface +type provisioningRequests struct { + client rest.Interface + ns string +} + +// newProvisioningRequests returns a ProvisioningRequests +func newProvisioningRequests(c *AutoscalingV1beta1Client, namespace string) *provisioningRequests { + return &provisioningRequests{ + client: c.RESTClient(), + ns: namespace, + } +} + +// Get takes name of the provisioningRequest, and returns the corresponding provisioningRequest object, and an error if there is any. +func (c *provisioningRequests) Get(ctx context.Context, name string, options v1.GetOptions) (result *v1beta1.ProvisioningRequest, err error) { + result = &v1beta1.ProvisioningRequest{} + err = c.client.Get(). + Namespace(c.ns). + Resource("provisioningrequests"). + Name(name). + VersionedParams(&options, scheme.ParameterCodec). + Do(ctx). + Into(result) + return +} + +// List takes label and field selectors, and returns the list of ProvisioningRequests that match those selectors. +func (c *provisioningRequests) List(ctx context.Context, opts v1.ListOptions) (result *v1beta1.ProvisioningRequestList, err error) { + var timeout time.Duration + if opts.TimeoutSeconds != nil { + timeout = time.Duration(*opts.TimeoutSeconds) * time.Second + } + result = &v1beta1.ProvisioningRequestList{} + err = c.client.Get(). + Namespace(c.ns). + Resource("provisioningrequests"). + VersionedParams(&opts, scheme.ParameterCodec). + Timeout(timeout). + Do(ctx). + Into(result) + return +} + +// Watch returns a watch.Interface that watches the requested provisioningRequests. +func (c *provisioningRequests) Watch(ctx context.Context, opts v1.ListOptions) (watch.Interface, error) { + var timeout time.Duration + if opts.TimeoutSeconds != nil { + timeout = time.Duration(*opts.TimeoutSeconds) * time.Second + } + opts.Watch = true + return c.client.Get(). + Namespace(c.ns). + Resource("provisioningrequests"). + VersionedParams(&opts, scheme.ParameterCodec). + Timeout(timeout). + Watch(ctx) +} + +// Create takes the representation of a provisioningRequest and creates it. Returns the server's representation of the provisioningRequest, and an error, if there is any. +func (c *provisioningRequests) Create(ctx context.Context, provisioningRequest *v1beta1.ProvisioningRequest, opts v1.CreateOptions) (result *v1beta1.ProvisioningRequest, err error) { + result = &v1beta1.ProvisioningRequest{} + err = c.client.Post(). + Namespace(c.ns). + Resource("provisioningrequests"). + VersionedParams(&opts, scheme.ParameterCodec). + Body(provisioningRequest). + Do(ctx). + Into(result) + return +} + +// Update takes the representation of a provisioningRequest and updates it. Returns the server's representation of the provisioningRequest, and an error, if there is any. +func (c *provisioningRequests) Update(ctx context.Context, provisioningRequest *v1beta1.ProvisioningRequest, opts v1.UpdateOptions) (result *v1beta1.ProvisioningRequest, err error) { + result = &v1beta1.ProvisioningRequest{} + err = c.client.Put(). + Namespace(c.ns). + Resource("provisioningrequests"). + Name(provisioningRequest.Name). + VersionedParams(&opts, scheme.ParameterCodec). + Body(provisioningRequest). + Do(ctx). + Into(result) + return +} + +// UpdateStatus was generated because the type contains a Status member. +// Add a +genclient:noStatus comment above the type to avoid generating UpdateStatus(). +func (c *provisioningRequests) UpdateStatus(ctx context.Context, provisioningRequest *v1beta1.ProvisioningRequest, opts v1.UpdateOptions) (result *v1beta1.ProvisioningRequest, err error) { + result = &v1beta1.ProvisioningRequest{} + err = c.client.Put(). + Namespace(c.ns). + Resource("provisioningrequests"). + Name(provisioningRequest.Name). + SubResource("status"). + VersionedParams(&opts, scheme.ParameterCodec). + Body(provisioningRequest). + Do(ctx). + Into(result) + return +} + +// Delete takes name of the provisioningRequest and deletes it. Returns an error if one occurs. +func (c *provisioningRequests) Delete(ctx context.Context, name string, opts v1.DeleteOptions) error { + return c.client.Delete(). + Namespace(c.ns). + Resource("provisioningrequests"). + Name(name). + Body(&opts). + Do(ctx). + Error() +} + +// DeleteCollection deletes a collection of objects. +func (c *provisioningRequests) DeleteCollection(ctx context.Context, opts v1.DeleteOptions, listOpts v1.ListOptions) error { + var timeout time.Duration + if listOpts.TimeoutSeconds != nil { + timeout = time.Duration(*listOpts.TimeoutSeconds) * time.Second + } + return c.client.Delete(). + Namespace(c.ns). + Resource("provisioningrequests"). + VersionedParams(&listOpts, scheme.ParameterCodec). + Timeout(timeout). + Body(&opts). + Do(ctx). + Error() +} + +// Patch applies the patch and returns the patched provisioningRequest. +func (c *provisioningRequests) Patch(ctx context.Context, name string, pt types.PatchType, data []byte, opts v1.PatchOptions, subresources ...string) (result *v1beta1.ProvisioningRequest, err error) { + result = &v1beta1.ProvisioningRequest{} + err = c.client.Patch(pt). + Namespace(c.ns). + Resource("provisioningrequests"). + Name(name). + SubResource(subresources...). + VersionedParams(&opts, scheme.ParameterCodec). + Body(data). + Do(ctx). + Into(result) + return +} + +// Apply takes the given apply declarative configuration, applies it and returns the applied provisioningRequest. +func (c *provisioningRequests) Apply(ctx context.Context, provisioningRequest *autoscalingxk8siov1beta1.ProvisioningRequestApplyConfiguration, opts v1.ApplyOptions) (result *v1beta1.ProvisioningRequest, err error) { + if provisioningRequest == nil { + return nil, fmt.Errorf("provisioningRequest provided to Apply must not be nil") + } + patchOpts := opts.ToPatchOptions() + data, err := json.Marshal(provisioningRequest) + if err != nil { + return nil, err + } + name := provisioningRequest.Name + if name == nil { + return nil, fmt.Errorf("provisioningRequest.Name must be provided to Apply") + } + result = &v1beta1.ProvisioningRequest{} + err = c.client.Patch(types.ApplyPatchType). + Namespace(c.ns). + Resource("provisioningrequests"). + Name(*name). + VersionedParams(&patchOpts, scheme.ParameterCodec). + Body(data). + Do(ctx). + Into(result) + return +} + +// ApplyStatus was generated because the type contains a Status member. +// Add a +genclient:noStatus comment above the type to avoid generating ApplyStatus(). +func (c *provisioningRequests) ApplyStatus(ctx context.Context, provisioningRequest *autoscalingxk8siov1beta1.ProvisioningRequestApplyConfiguration, opts v1.ApplyOptions) (result *v1beta1.ProvisioningRequest, err error) { + if provisioningRequest == nil { + return nil, fmt.Errorf("provisioningRequest provided to Apply must not be nil") + } + patchOpts := opts.ToPatchOptions() + data, err := json.Marshal(provisioningRequest) + if err != nil { + return nil, err + } + + name := provisioningRequest.Name + if name == nil { + return nil, fmt.Errorf("provisioningRequest.Name must be provided to Apply") + } + + result = &v1beta1.ProvisioningRequest{} + err = c.client.Patch(types.ApplyPatchType). + Namespace(c.ns). + Resource("provisioningrequests"). + Name(*name). + SubResource("status"). + VersionedParams(&patchOpts, scheme.ParameterCodec). + Body(data). + Do(ctx). + Into(result) + return +} diff --git a/cluster-autoscaler/apis/provisioningrequest/client/informers/externalversions/autoscaling.x-k8s.io/interface.go b/cluster-autoscaler/apis/provisioningrequest/client/informers/externalversions/autoscaling.x-k8s.io/interface.go new file mode 100644 index 000000000000..ff99b5a2d9db --- /dev/null +++ b/cluster-autoscaler/apis/provisioningrequest/client/informers/externalversions/autoscaling.x-k8s.io/interface.go @@ -0,0 +1,46 @@ +/* +Copyright The Kubernetes Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// Code generated by informer-gen. DO NOT EDIT. + +package autoscaling + +import ( + v1beta1 "k8s.io/autoscaler/cluster-autoscaler/apis/provisioningrequest/client/informers/externalversions/autoscaling.x-k8s.io/v1beta1" + internalinterfaces "k8s.io/autoscaler/cluster-autoscaler/apis/provisioningrequest/client/informers/externalversions/internalinterfaces" +) + +// Interface provides access to each of this group's versions. +type Interface interface { + // V1beta1 provides access to shared informers for resources in V1beta1. + V1beta1() v1beta1.Interface +} + +type group struct { + factory internalinterfaces.SharedInformerFactory + namespace string + tweakListOptions internalinterfaces.TweakListOptionsFunc +} + +// New returns a new Interface. +func New(f internalinterfaces.SharedInformerFactory, namespace string, tweakListOptions internalinterfaces.TweakListOptionsFunc) Interface { + return &group{factory: f, namespace: namespace, tweakListOptions: tweakListOptions} +} + +// V1beta1 returns a new v1beta1.Interface. +func (g *group) V1beta1() v1beta1.Interface { + return v1beta1.New(g.factory, g.namespace, g.tweakListOptions) +} diff --git a/cluster-autoscaler/apis/provisioningrequest/client/informers/externalversions/autoscaling.x-k8s.io/v1beta1/interface.go b/cluster-autoscaler/apis/provisioningrequest/client/informers/externalversions/autoscaling.x-k8s.io/v1beta1/interface.go new file mode 100644 index 000000000000..75d067779e80 --- /dev/null +++ b/cluster-autoscaler/apis/provisioningrequest/client/informers/externalversions/autoscaling.x-k8s.io/v1beta1/interface.go @@ -0,0 +1,45 @@ +/* +Copyright The Kubernetes Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// Code generated by informer-gen. DO NOT EDIT. + +package v1beta1 + +import ( + internalinterfaces "k8s.io/autoscaler/cluster-autoscaler/apis/provisioningrequest/client/informers/externalversions/internalinterfaces" +) + +// Interface provides access to all the informers in this group version. +type Interface interface { + // ProvisioningRequests returns a ProvisioningRequestInformer. + ProvisioningRequests() ProvisioningRequestInformer +} + +type version struct { + factory internalinterfaces.SharedInformerFactory + namespace string + tweakListOptions internalinterfaces.TweakListOptionsFunc +} + +// New returns a new Interface. +func New(f internalinterfaces.SharedInformerFactory, namespace string, tweakListOptions internalinterfaces.TweakListOptionsFunc) Interface { + return &version{factory: f, namespace: namespace, tweakListOptions: tweakListOptions} +} + +// ProvisioningRequests returns a ProvisioningRequestInformer. +func (v *version) ProvisioningRequests() ProvisioningRequestInformer { + return &provisioningRequestInformer{factory: v.factory, namespace: v.namespace, tweakListOptions: v.tweakListOptions} +} diff --git a/cluster-autoscaler/apis/provisioningrequest/client/informers/externalversions/autoscaling.x-k8s.io/v1beta1/provisioningrequest.go b/cluster-autoscaler/apis/provisioningrequest/client/informers/externalversions/autoscaling.x-k8s.io/v1beta1/provisioningrequest.go new file mode 100644 index 000000000000..0cfad3976415 --- /dev/null +++ b/cluster-autoscaler/apis/provisioningrequest/client/informers/externalversions/autoscaling.x-k8s.io/v1beta1/provisioningrequest.go @@ -0,0 +1,90 @@ +/* +Copyright The Kubernetes Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// Code generated by informer-gen. DO NOT EDIT. + +package v1beta1 + +import ( + "context" + time "time" + + v1 "k8s.io/apimachinery/pkg/apis/meta/v1" + runtime "k8s.io/apimachinery/pkg/runtime" + watch "k8s.io/apimachinery/pkg/watch" + autoscalingxk8siov1beta1 "k8s.io/autoscaler/cluster-autoscaler/apis/provisioningrequest/autoscaling.x-k8s.io/v1beta1" + versioned "k8s.io/autoscaler/cluster-autoscaler/apis/provisioningrequest/client/clientset/versioned" + internalinterfaces "k8s.io/autoscaler/cluster-autoscaler/apis/provisioningrequest/client/informers/externalversions/internalinterfaces" + v1beta1 "k8s.io/autoscaler/cluster-autoscaler/apis/provisioningrequest/client/listers/autoscaling.x-k8s.io/v1beta1" + cache "k8s.io/client-go/tools/cache" +) + +// ProvisioningRequestInformer provides access to a shared informer and lister for +// ProvisioningRequests. +type ProvisioningRequestInformer interface { + Informer() cache.SharedIndexInformer + Lister() v1beta1.ProvisioningRequestLister +} + +type provisioningRequestInformer struct { + factory internalinterfaces.SharedInformerFactory + tweakListOptions internalinterfaces.TweakListOptionsFunc + namespace string +} + +// NewProvisioningRequestInformer constructs a new informer for ProvisioningRequest type. +// Always prefer using an informer factory to get a shared informer instead of getting an independent +// one. This reduces memory footprint and number of connections to the server. +func NewProvisioningRequestInformer(client versioned.Interface, namespace string, resyncPeriod time.Duration, indexers cache.Indexers) cache.SharedIndexInformer { + return NewFilteredProvisioningRequestInformer(client, namespace, resyncPeriod, indexers, nil) +} + +// NewFilteredProvisioningRequestInformer constructs a new informer for ProvisioningRequest type. +// Always prefer using an informer factory to get a shared informer instead of getting an independent +// one. This reduces memory footprint and number of connections to the server. +func NewFilteredProvisioningRequestInformer(client versioned.Interface, namespace string, resyncPeriod time.Duration, indexers cache.Indexers, tweakListOptions internalinterfaces.TweakListOptionsFunc) cache.SharedIndexInformer { + return cache.NewSharedIndexInformer( + &cache.ListWatch{ + ListFunc: func(options v1.ListOptions) (runtime.Object, error) { + if tweakListOptions != nil { + tweakListOptions(&options) + } + return client.AutoscalingV1beta1().ProvisioningRequests(namespace).List(context.TODO(), options) + }, + WatchFunc: func(options v1.ListOptions) (watch.Interface, error) { + if tweakListOptions != nil { + tweakListOptions(&options) + } + return client.AutoscalingV1beta1().ProvisioningRequests(namespace).Watch(context.TODO(), options) + }, + }, + &autoscalingxk8siov1beta1.ProvisioningRequest{}, + resyncPeriod, + indexers, + ) +} + +func (f *provisioningRequestInformer) defaultInformer(client versioned.Interface, resyncPeriod time.Duration) cache.SharedIndexInformer { + return NewFilteredProvisioningRequestInformer(client, f.namespace, resyncPeriod, cache.Indexers{cache.NamespaceIndex: cache.MetaNamespaceIndexFunc}, f.tweakListOptions) +} + +func (f *provisioningRequestInformer) Informer() cache.SharedIndexInformer { + return f.factory.InformerFor(&autoscalingxk8siov1beta1.ProvisioningRequest{}, f.defaultInformer) +} + +func (f *provisioningRequestInformer) Lister() v1beta1.ProvisioningRequestLister { + return v1beta1.NewProvisioningRequestLister(f.Informer().GetIndexer()) +} diff --git a/cluster-autoscaler/apis/provisioningrequest/client/informers/externalversions/factory.go b/cluster-autoscaler/apis/provisioningrequest/client/informers/externalversions/factory.go new file mode 100644 index 000000000000..57b06e6c58d2 --- /dev/null +++ b/cluster-autoscaler/apis/provisioningrequest/client/informers/externalversions/factory.go @@ -0,0 +1,261 @@ +/* +Copyright The Kubernetes Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// Code generated by informer-gen. DO NOT EDIT. + +package externalversions + +import ( + reflect "reflect" + sync "sync" + time "time" + + v1 "k8s.io/apimachinery/pkg/apis/meta/v1" + runtime "k8s.io/apimachinery/pkg/runtime" + schema "k8s.io/apimachinery/pkg/runtime/schema" + versioned "k8s.io/autoscaler/cluster-autoscaler/apis/provisioningrequest/client/clientset/versioned" + autoscalingxk8sio "k8s.io/autoscaler/cluster-autoscaler/apis/provisioningrequest/client/informers/externalversions/autoscaling.x-k8s.io" + internalinterfaces "k8s.io/autoscaler/cluster-autoscaler/apis/provisioningrequest/client/informers/externalversions/internalinterfaces" + cache "k8s.io/client-go/tools/cache" +) + +// SharedInformerOption defines the functional option type for SharedInformerFactory. +type SharedInformerOption func(*sharedInformerFactory) *sharedInformerFactory + +type sharedInformerFactory struct { + client versioned.Interface + namespace string + tweakListOptions internalinterfaces.TweakListOptionsFunc + lock sync.Mutex + defaultResync time.Duration + customResync map[reflect.Type]time.Duration + transform cache.TransformFunc + + informers map[reflect.Type]cache.SharedIndexInformer + // startedInformers is used for tracking which informers have been started. + // This allows Start() to be called multiple times safely. + startedInformers map[reflect.Type]bool + // wg tracks how many goroutines were started. + wg sync.WaitGroup + // shuttingDown is true when Shutdown has been called. It may still be running + // because it needs to wait for goroutines. + shuttingDown bool +} + +// WithCustomResyncConfig sets a custom resync period for the specified informer types. +func WithCustomResyncConfig(resyncConfig map[v1.Object]time.Duration) SharedInformerOption { + return func(factory *sharedInformerFactory) *sharedInformerFactory { + for k, v := range resyncConfig { + factory.customResync[reflect.TypeOf(k)] = v + } + return factory + } +} + +// WithTweakListOptions sets a custom filter on all listers of the configured SharedInformerFactory. +func WithTweakListOptions(tweakListOptions internalinterfaces.TweakListOptionsFunc) SharedInformerOption { + return func(factory *sharedInformerFactory) *sharedInformerFactory { + factory.tweakListOptions = tweakListOptions + return factory + } +} + +// WithNamespace limits the SharedInformerFactory to the specified namespace. +func WithNamespace(namespace string) SharedInformerOption { + return func(factory *sharedInformerFactory) *sharedInformerFactory { + factory.namespace = namespace + return factory + } +} + +// WithTransform sets a transform on all informers. +func WithTransform(transform cache.TransformFunc) SharedInformerOption { + return func(factory *sharedInformerFactory) *sharedInformerFactory { + factory.transform = transform + return factory + } +} + +// NewSharedInformerFactory constructs a new instance of sharedInformerFactory for all namespaces. +func NewSharedInformerFactory(client versioned.Interface, defaultResync time.Duration) SharedInformerFactory { + return NewSharedInformerFactoryWithOptions(client, defaultResync) +} + +// NewFilteredSharedInformerFactory constructs a new instance of sharedInformerFactory. +// Listers obtained via this SharedInformerFactory will be subject to the same filters +// as specified here. +// Deprecated: Please use NewSharedInformerFactoryWithOptions instead +func NewFilteredSharedInformerFactory(client versioned.Interface, defaultResync time.Duration, namespace string, tweakListOptions internalinterfaces.TweakListOptionsFunc) SharedInformerFactory { + return NewSharedInformerFactoryWithOptions(client, defaultResync, WithNamespace(namespace), WithTweakListOptions(tweakListOptions)) +} + +// NewSharedInformerFactoryWithOptions constructs a new instance of a SharedInformerFactory with additional options. +func NewSharedInformerFactoryWithOptions(client versioned.Interface, defaultResync time.Duration, options ...SharedInformerOption) SharedInformerFactory { + factory := &sharedInformerFactory{ + client: client, + namespace: v1.NamespaceAll, + defaultResync: defaultResync, + informers: make(map[reflect.Type]cache.SharedIndexInformer), + startedInformers: make(map[reflect.Type]bool), + customResync: make(map[reflect.Type]time.Duration), + } + + // Apply all options + for _, opt := range options { + factory = opt(factory) + } + + return factory +} + +func (f *sharedInformerFactory) Start(stopCh <-chan struct{}) { + f.lock.Lock() + defer f.lock.Unlock() + + if f.shuttingDown { + return + } + + for informerType, informer := range f.informers { + if !f.startedInformers[informerType] { + f.wg.Add(1) + // We need a new variable in each loop iteration, + // otherwise the goroutine would use the loop variable + // and that keeps changing. + informer := informer + go func() { + defer f.wg.Done() + informer.Run(stopCh) + }() + f.startedInformers[informerType] = true + } + } +} + +func (f *sharedInformerFactory) Shutdown() { + f.lock.Lock() + f.shuttingDown = true + f.lock.Unlock() + + // Will return immediately if there is nothing to wait for. + f.wg.Wait() +} + +func (f *sharedInformerFactory) WaitForCacheSync(stopCh <-chan struct{}) map[reflect.Type]bool { + informers := func() map[reflect.Type]cache.SharedIndexInformer { + f.lock.Lock() + defer f.lock.Unlock() + + informers := map[reflect.Type]cache.SharedIndexInformer{} + for informerType, informer := range f.informers { + if f.startedInformers[informerType] { + informers[informerType] = informer + } + } + return informers + }() + + res := map[reflect.Type]bool{} + for informType, informer := range informers { + res[informType] = cache.WaitForCacheSync(stopCh, informer.HasSynced) + } + return res +} + +// InformerFor returns the SharedIndexInformer for obj using an internal +// client. +func (f *sharedInformerFactory) InformerFor(obj runtime.Object, newFunc internalinterfaces.NewInformerFunc) cache.SharedIndexInformer { + f.lock.Lock() + defer f.lock.Unlock() + + informerType := reflect.TypeOf(obj) + informer, exists := f.informers[informerType] + if exists { + return informer + } + + resyncPeriod, exists := f.customResync[informerType] + if !exists { + resyncPeriod = f.defaultResync + } + + informer = newFunc(f.client, resyncPeriod) + informer.SetTransform(f.transform) + f.informers[informerType] = informer + + return informer +} + +// SharedInformerFactory provides shared informers for resources in all known +// API group versions. +// +// It is typically used like this: +// +// ctx, cancel := context.Background() +// defer cancel() +// factory := NewSharedInformerFactory(client, resyncPeriod) +// defer factory.WaitForStop() // Returns immediately if nothing was started. +// genericInformer := factory.ForResource(resource) +// typedInformer := factory.SomeAPIGroup().V1().SomeType() +// factory.Start(ctx.Done()) // Start processing these informers. +// synced := factory.WaitForCacheSync(ctx.Done()) +// for v, ok := range synced { +// if !ok { +// fmt.Fprintf(os.Stderr, "caches failed to sync: %v", v) +// return +// } +// } +// +// // Creating informers can also be created after Start, but then +// // Start must be called again: +// anotherGenericInformer := factory.ForResource(resource) +// factory.Start(ctx.Done()) +type SharedInformerFactory interface { + internalinterfaces.SharedInformerFactory + + // Start initializes all requested informers. They are handled in goroutines + // which run until the stop channel gets closed. + Start(stopCh <-chan struct{}) + + // Shutdown marks a factory as shutting down. At that point no new + // informers can be started anymore and Start will return without + // doing anything. + // + // In addition, Shutdown blocks until all goroutines have terminated. For that + // to happen, the close channel(s) that they were started with must be closed, + // either before Shutdown gets called or while it is waiting. + // + // Shutdown may be called multiple times, even concurrently. All such calls will + // block until all goroutines have terminated. + Shutdown() + + // WaitForCacheSync blocks until all started informers' caches were synced + // or the stop channel gets closed. + WaitForCacheSync(stopCh <-chan struct{}) map[reflect.Type]bool + + // ForResource gives generic access to a shared informer of the matching type. + ForResource(resource schema.GroupVersionResource) (GenericInformer, error) + + // InformerFor returns the SharedIndexInformer for obj using an internal + // client. + InformerFor(obj runtime.Object, newFunc internalinterfaces.NewInformerFunc) cache.SharedIndexInformer + + Autoscaling() autoscalingxk8sio.Interface +} + +func (f *sharedInformerFactory) Autoscaling() autoscalingxk8sio.Interface { + return autoscalingxk8sio.New(f, f.namespace, f.tweakListOptions) +} diff --git a/cluster-autoscaler/apis/provisioningrequest/client/informers/externalversions/generic.go b/cluster-autoscaler/apis/provisioningrequest/client/informers/externalversions/generic.go new file mode 100644 index 000000000000..94ac828160a2 --- /dev/null +++ b/cluster-autoscaler/apis/provisioningrequest/client/informers/externalversions/generic.go @@ -0,0 +1,62 @@ +/* +Copyright The Kubernetes Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// Code generated by informer-gen. DO NOT EDIT. + +package externalversions + +import ( + "fmt" + + schema "k8s.io/apimachinery/pkg/runtime/schema" + v1beta1 "k8s.io/autoscaler/cluster-autoscaler/apis/provisioningrequest/autoscaling.x-k8s.io/v1beta1" + cache "k8s.io/client-go/tools/cache" +) + +// GenericInformer is type of SharedIndexInformer which will locate and delegate to other +// sharedInformers based on type +type GenericInformer interface { + Informer() cache.SharedIndexInformer + Lister() cache.GenericLister +} + +type genericInformer struct { + informer cache.SharedIndexInformer + resource schema.GroupResource +} + +// Informer returns the SharedIndexInformer. +func (f *genericInformer) Informer() cache.SharedIndexInformer { + return f.informer +} + +// Lister returns the GenericLister. +func (f *genericInformer) Lister() cache.GenericLister { + return cache.NewGenericLister(f.Informer().GetIndexer(), f.resource) +} + +// ForResource gives generic access to a shared informer of the matching type +// TODO extend this to unknown resources with a client pool +func (f *sharedInformerFactory) ForResource(resource schema.GroupVersionResource) (GenericInformer, error) { + switch resource { + // Group=autoscaling.x-k8s.io, Version=v1beta1 + case v1beta1.SchemeGroupVersion.WithResource("provisioningrequests"): + return &genericInformer{resource: resource.GroupResource(), informer: f.Autoscaling().V1beta1().ProvisioningRequests().Informer()}, nil + + } + + return nil, fmt.Errorf("no informer found for %v", resource) +} diff --git a/cluster-autoscaler/apis/provisioningrequest/client/informers/externalversions/internalinterfaces/factory_interfaces.go b/cluster-autoscaler/apis/provisioningrequest/client/informers/externalversions/internalinterfaces/factory_interfaces.go new file mode 100644 index 000000000000..1c9d3baed19d --- /dev/null +++ b/cluster-autoscaler/apis/provisioningrequest/client/informers/externalversions/internalinterfaces/factory_interfaces.go @@ -0,0 +1,40 @@ +/* +Copyright The Kubernetes Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// Code generated by informer-gen. DO NOT EDIT. + +package internalinterfaces + +import ( + time "time" + + v1 "k8s.io/apimachinery/pkg/apis/meta/v1" + runtime "k8s.io/apimachinery/pkg/runtime" + versioned "k8s.io/autoscaler/cluster-autoscaler/apis/provisioningrequest/client/clientset/versioned" + cache "k8s.io/client-go/tools/cache" +) + +// NewInformerFunc takes versioned.Interface and time.Duration to return a SharedIndexInformer. +type NewInformerFunc func(versioned.Interface, time.Duration) cache.SharedIndexInformer + +// SharedInformerFactory a small interface to allow for adding an informer without an import cycle +type SharedInformerFactory interface { + Start(stopCh <-chan struct{}) + InformerFor(obj runtime.Object, newFunc NewInformerFunc) cache.SharedIndexInformer +} + +// TweakListOptionsFunc is a function that transforms a v1.ListOptions. +type TweakListOptionsFunc func(*v1.ListOptions) diff --git a/cluster-autoscaler/provisioningrequest/client/listers/autoscaling.x-k8s.io/v1beta1/expansion_generated.go b/cluster-autoscaler/apis/provisioningrequest/client/listers/autoscaling.x-k8s.io/v1beta1/expansion_generated.go similarity index 100% rename from cluster-autoscaler/provisioningrequest/client/listers/autoscaling.x-k8s.io/v1beta1/expansion_generated.go rename to cluster-autoscaler/apis/provisioningrequest/client/listers/autoscaling.x-k8s.io/v1beta1/expansion_generated.go diff --git a/cluster-autoscaler/apis/provisioningrequest/client/listers/autoscaling.x-k8s.io/v1beta1/provisioningrequest.go b/cluster-autoscaler/apis/provisioningrequest/client/listers/autoscaling.x-k8s.io/v1beta1/provisioningrequest.go new file mode 100644 index 000000000000..4073183123cc --- /dev/null +++ b/cluster-autoscaler/apis/provisioningrequest/client/listers/autoscaling.x-k8s.io/v1beta1/provisioningrequest.go @@ -0,0 +1,99 @@ +/* +Copyright The Kubernetes Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// Code generated by lister-gen. DO NOT EDIT. + +package v1beta1 + +import ( + "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/labels" + v1beta1 "k8s.io/autoscaler/cluster-autoscaler/apis/provisioningrequest/autoscaling.x-k8s.io/v1beta1" + "k8s.io/client-go/tools/cache" +) + +// ProvisioningRequestLister helps list ProvisioningRequests. +// All objects returned here must be treated as read-only. +type ProvisioningRequestLister interface { + // List lists all ProvisioningRequests in the indexer. + // Objects returned here must be treated as read-only. + List(selector labels.Selector) (ret []*v1beta1.ProvisioningRequest, err error) + // ProvisioningRequests returns an object that can list and get ProvisioningRequests. + ProvisioningRequests(namespace string) ProvisioningRequestNamespaceLister + ProvisioningRequestListerExpansion +} + +// provisioningRequestLister implements the ProvisioningRequestLister interface. +type provisioningRequestLister struct { + indexer cache.Indexer +} + +// NewProvisioningRequestLister returns a new ProvisioningRequestLister. +func NewProvisioningRequestLister(indexer cache.Indexer) ProvisioningRequestLister { + return &provisioningRequestLister{indexer: indexer} +} + +// List lists all ProvisioningRequests in the indexer. +func (s *provisioningRequestLister) List(selector labels.Selector) (ret []*v1beta1.ProvisioningRequest, err error) { + err = cache.ListAll(s.indexer, selector, func(m interface{}) { + ret = append(ret, m.(*v1beta1.ProvisioningRequest)) + }) + return ret, err +} + +// ProvisioningRequests returns an object that can list and get ProvisioningRequests. +func (s *provisioningRequestLister) ProvisioningRequests(namespace string) ProvisioningRequestNamespaceLister { + return provisioningRequestNamespaceLister{indexer: s.indexer, namespace: namespace} +} + +// ProvisioningRequestNamespaceLister helps list and get ProvisioningRequests. +// All objects returned here must be treated as read-only. +type ProvisioningRequestNamespaceLister interface { + // List lists all ProvisioningRequests in the indexer for a given namespace. + // Objects returned here must be treated as read-only. + List(selector labels.Selector) (ret []*v1beta1.ProvisioningRequest, err error) + // Get retrieves the ProvisioningRequest from the indexer for a given namespace and name. + // Objects returned here must be treated as read-only. + Get(name string) (*v1beta1.ProvisioningRequest, error) + ProvisioningRequestNamespaceListerExpansion +} + +// provisioningRequestNamespaceLister implements the ProvisioningRequestNamespaceLister +// interface. +type provisioningRequestNamespaceLister struct { + indexer cache.Indexer + namespace string +} + +// List lists all ProvisioningRequests in the indexer for a given namespace. +func (s provisioningRequestNamespaceLister) List(selector labels.Selector) (ret []*v1beta1.ProvisioningRequest, err error) { + err = cache.ListAllByNamespace(s.indexer, s.namespace, selector, func(m interface{}) { + ret = append(ret, m.(*v1beta1.ProvisioningRequest)) + }) + return ret, err +} + +// Get retrieves the ProvisioningRequest from the indexer for a given namespace and name. +func (s provisioningRequestNamespaceLister) Get(name string) (*v1beta1.ProvisioningRequest, error) { + obj, exists, err := s.indexer.GetByKey(s.namespace + "/" + name) + if err != nil { + return nil, err + } + if !exists { + return nil, errors.NewNotFound(v1beta1.Resource("provisioningrequest"), name) + } + return obj.(*v1beta1.ProvisioningRequest), nil +} diff --git a/cluster-autoscaler/cloudprovider/alicloud/alibaba-cloud-sdk-go/sdk/utils/utils.go b/cluster-autoscaler/cloudprovider/alicloud/alibaba-cloud-sdk-go/sdk/utils/utils.go index 68da41409fa9..32a18e607a4a 100644 --- a/cluster-autoscaler/cloudprovider/alicloud/alibaba-cloud-sdk-go/sdk/utils/utils.go +++ b/cluster-autoscaler/cloudprovider/alicloud/alibaba-cloud-sdk-go/sdk/utils/utils.go @@ -22,7 +22,7 @@ import ( "encoding/hex" "encoding/json" "fmt" - "github.com/satori/go.uuid" + "github.com/google/uuid" "net/url" "reflect" "strconv" @@ -37,8 +37,9 @@ var ( // GetUUIDV4 returns uuidHex func GetUUIDV4() (uuidHex string) { - uuidV4 := uuid.NewV4() - uuidHex = hex.EncodeToString(uuidV4.Bytes()) + uuidV4 := uuid.New() + binaryUUID, _ := uuidV4.MarshalBinary() + uuidHex = hex.EncodeToString(binaryUUID) return } diff --git a/cluster-autoscaler/cloudprovider/alicloud/alicloud_auto_scaling_group.go b/cluster-autoscaler/cloudprovider/alicloud/alicloud_auto_scaling_group.go index bc1fd7be798e..dd0c9a7eb478 100644 --- a/cluster-autoscaler/cloudprovider/alicloud/alicloud_auto_scaling_group.go +++ b/cluster-autoscaler/cloudprovider/alicloud/alicloud_auto_scaling_group.go @@ -69,6 +69,11 @@ func (asg *Asg) IncreaseSize(delta int) error { return asg.manager.SetAsgSize(asg, size+int64(delta)) } +// AtomicIncreaseSize is not implemented. +func (asg *Asg) AtomicIncreaseSize(delta int) error { + return cloudprovider.ErrNotImplemented +} + // DecreaseTargetSize decreases the target size of the node group. This function // doesn't permit to delete any existing node and can be used only to reduce the // request for new nodes that have not been yet fulfilled. Delta should be negative. diff --git a/cluster-autoscaler/cloudprovider/alicloud/alicloud_cloud_provider.go b/cluster-autoscaler/cloudprovider/alicloud/alicloud_cloud_provider.go index 09cc002e7ecd..c8513c7bf964 100644 --- a/cluster-autoscaler/cloudprovider/alicloud/alicloud_cloud_provider.go +++ b/cluster-autoscaler/cloudprovider/alicloud/alicloud_cloud_provider.go @@ -126,6 +126,10 @@ func (ali *aliCloudProvider) NodeGroups() []cloudprovider.NodeGroup { // NodeGroupForNode returns the node group for the given node. func (ali *aliCloudProvider) NodeGroupForNode(node *apiv1.Node) (cloudprovider.NodeGroup, error) { + if len(node.Spec.ProviderID) == 0 { + klog.Warningf("Node %v has no providerId", node.Name) + return nil, nil + } instanceId, err := ecsInstanceIdFromProviderId(node.Spec.ProviderID) if err != nil { klog.Errorf("failed to get instance Id from provider Id:%s,because of %s", node.Spec.ProviderID, err.Error()) diff --git a/cluster-autoscaler/cloudprovider/aws/README.md b/cluster-autoscaler/cloudprovider/aws/README.md index 8f12fc0a05a5..ea6cc0a7d7eb 100644 --- a/cluster-autoscaler/cloudprovider/aws/README.md +++ b/cluster-autoscaler/cloudprovider/aws/README.md @@ -49,8 +49,11 @@ should be updated to restrict the resources/add conditionals: "autoscaling:DescribeLaunchConfigurations", "autoscaling:DescribeScalingActivities", "autoscaling:DescribeTags", + "ec2:DescribeImages", "ec2:DescribeInstanceTypes", - "ec2:DescribeLaunchTemplateVersions" + "ec2:DescribeLaunchTemplateVersions", + "ec2:GetInstanceTypesFromInstanceRequirements", + "eks:DescribeNodegroup" ], "Resource": ["*"] }, @@ -58,10 +61,7 @@ should be updated to restrict the resources/add conditionals: "Effect": "Allow", "Action": [ "autoscaling:SetDesiredCapacity", - "autoscaling:TerminateInstanceInAutoScalingGroup", - "ec2:DescribeImages", - "ec2:GetInstanceTypesFromInstanceRequirements", - "eks:DescribeNodegroup" + "autoscaling:TerminateInstanceInAutoScalingGroup" ], "Resource": ["*"] } @@ -247,7 +247,7 @@ as string). Currently supported autoscaling options (and example values) are: * `k8s.io/cluster-autoscaler/node-template/autoscaling-options/scaledownunreadytime`: `20m0s` (overrides `--scale-down-unready-time` value for that specific ASG) * `k8s.io/cluster-autoscaler/node-template/autoscaling-options/ignoredaemonsetsutilization`: `true` - (overrides `--ignore-daemonsets-utilization` value for that specific ASG) + (overrides `--ignore-daemonsets-utilization` value for that specific ASG) **NOTE:** It is your responsibility to ensure such labels and/or taints are applied via the node's kubelet configuration at startup. Cluster Autoscaler will not set the node taints for you. @@ -507,12 +507,14 @@ Please note: it is also possible to mount the cloud config file from host: `--scale-down-delay-after-delete`, and `--scale-down-delay-after-failure` flag. E.g. `--scale-down-delay-after-add=5m` to decrease the scale down delay to 5 minutes after a node has been added. -- If you're running multiple ASGs, the `--expander` flag supports three options: - `random`, `most-pods` and `least-waste`. `random` will expand a random ASG on - scale up. `most-pods` will scale up the ASG that will schedule the most amount - of pods. `least-waste` will expand the ASG that will waste the least amount of - CPU/MEM resources. In the event of a tie, cluster autoscaler will fall back to - `random`. +- If you're running multiple ASGs, the `--expander` flag supports five options: + `random`, `most-pods`, `least-waste`, `priority`, and `grpc`. `random` will + expand a random ASG on scale up. `most-pods` will scale up the ASG that will + schedule the most amount of pods. `least-waste` will expand the ASG that will + waste the least amount of CPU/MEM resources. In the event of a tie, cluster + autoscaler will fall back to`random`. The `priority` expander lets you define + a custom priority ranking in a ConfigMap for selecting ASGs, and the `grpc` + expander allows you to write your own expansion logic. - If you're managing your own kubelets, they need to be started with the `--provider-id` flag. The provider id has the format `aws:////`, e.g. diff --git a/cluster-autoscaler/cloudprovider/aws/auto_scaling_groups.go b/cluster-autoscaler/cloudprovider/aws/auto_scaling_groups.go index 284b5b5c0b32..4667e285b153 100644 --- a/cluster-autoscaler/cloudprovider/aws/auto_scaling_groups.go +++ b/cluster-autoscaler/cloudprovider/aws/auto_scaling_groups.go @@ -141,8 +141,6 @@ func (m *asgCache) register(asg *asg) *asg { return existing } - klog.V(4).Infof("Updating ASG %s", asg.AwsRef.Name) - // Explicit registered groups should always use the manually provided min/max // values and the not the ones returned by the API if !m.explicitlyConfigured[asg.AwsRef] { diff --git a/cluster-autoscaler/cloudprovider/aws/aws_cloud_provider.go b/cluster-autoscaler/cloudprovider/aws/aws_cloud_provider.go index f5ce3f8199de..f515530b7ccb 100644 --- a/cluster-autoscaler/cloudprovider/aws/aws_cloud_provider.go +++ b/cluster-autoscaler/cloudprovider/aws/aws_cloud_provider.go @@ -280,6 +280,11 @@ func (ng *AwsNodeGroup) IncreaseSize(delta int) error { return ng.awsManager.SetAsgSize(ng.asg, size+delta) } +// AtomicIncreaseSize is not implemented. +func (ng *AwsNodeGroup) AtomicIncreaseSize(delta int) error { + return cloudprovider.ErrNotImplemented +} + // DecreaseTargetSize decreases the target size of the node group. This function // doesn't permit to delete any existing node and can be used only to reduce the // request for new nodes that have not been yet fulfilled. Delta should be negative. diff --git a/cluster-autoscaler/cloudprovider/aws/ec2_instance_types.go b/cluster-autoscaler/cloudprovider/aws/ec2_instance_types.go index 4781bbbfd073..f71770c82d1b 100644 --- a/cluster-autoscaler/cloudprovider/aws/ec2_instance_types.go +++ b/cluster-autoscaler/cloudprovider/aws/ec2_instance_types.go @@ -28,7 +28,7 @@ type InstanceType struct { } // StaticListLastUpdateTime is a string declaring the last time the static list was updated. -var StaticListLastUpdateTime = "2023-11-29" +var StaticListLastUpdateTime = "2024-04-08" // InstanceTypes is a map of ec2 resources var InstanceTypes = map[string]*InstanceType{ @@ -1600,6 +1600,76 @@ var InstanceTypes = map[string]*InstanceType{ GPU: 1, Architecture: "arm64", }, + "g6.12xlarge": { + InstanceType: "g6.12xlarge", + VCPU: 48, + MemoryMb: 196608, + GPU: 4, + Architecture: "amd64", + }, + "g6.16xlarge": { + InstanceType: "g6.16xlarge", + VCPU: 64, + MemoryMb: 262144, + GPU: 1, + Architecture: "amd64", + }, + "g6.24xlarge": { + InstanceType: "g6.24xlarge", + VCPU: 96, + MemoryMb: 393216, + GPU: 4, + Architecture: "amd64", + }, + "g6.2xlarge": { + InstanceType: "g6.2xlarge", + VCPU: 8, + MemoryMb: 32768, + GPU: 1, + Architecture: "amd64", + }, + "g6.48xlarge": { + InstanceType: "g6.48xlarge", + VCPU: 192, + MemoryMb: 786432, + GPU: 8, + Architecture: "amd64", + }, + "g6.4xlarge": { + InstanceType: "g6.4xlarge", + VCPU: 16, + MemoryMb: 65536, + GPU: 1, + Architecture: "amd64", + }, + "g6.8xlarge": { + InstanceType: "g6.8xlarge", + VCPU: 32, + MemoryMb: 131072, + GPU: 1, + Architecture: "amd64", + }, + "g6.xlarge": { + InstanceType: "g6.xlarge", + VCPU: 4, + MemoryMb: 16384, + GPU: 1, + Architecture: "amd64", + }, + "gr6.4xlarge": { + InstanceType: "gr6.4xlarge", + VCPU: 16, + MemoryMb: 131072, + GPU: 1, + Architecture: "amd64", + }, + "gr6.8xlarge": { + InstanceType: "gr6.8xlarge", + VCPU: 32, + MemoryMb: 262144, + GPU: 1, + Architecture: "amd64", + }, "h1.16xlarge": { InstanceType: "h1.16xlarge", VCPU: 64, diff --git a/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-run-on-control-plane.yaml b/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-run-on-control-plane.yaml index 9cda4af03e5b..1bc986ce19d5 100644 --- a/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-run-on-control-plane.yaml +++ b/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-run-on-control-plane.yaml @@ -149,9 +149,9 @@ spec: - effect: NoSchedule operator: "Equal" value: "true" - key: node-role.kubernetes.io/master + key: node-role.kubernetes.io/control-plane nodeSelector: - kubernetes.io/role: master + kubernetes.io/role: control-plane containers: - image: registry.k8s.io/autoscaling/cluster-autoscaler:v1.26.2 name: cluster-autoscaler diff --git a/cluster-autoscaler/cloudprovider/azure/azure_agent_pool.go b/cluster-autoscaler/cloudprovider/azure/azure_agent_pool.go index f8d60aa99968..a3e714fc07bc 100644 --- a/cluster-autoscaler/cloudprovider/azure/azure_agent_pool.go +++ b/cluster-autoscaler/cloudprovider/azure/azure_agent_pool.go @@ -335,6 +335,11 @@ func (as *AgentPool) IncreaseSize(delta int) error { return realError } +// AtomicIncreaseSize is not implemented. +func (as *AgentPool) AtomicIncreaseSize(delta int) error { + return cloudprovider.ErrNotImplemented +} + // DecreaseTargetSize decreases the target size of the node group. This function // doesn't permit to delete any existing node and can be used only to reduce the // request for new nodes that have not been yet fulfilled. Delta should be negative. diff --git a/cluster-autoscaler/cloudprovider/azure/azure_cache.go b/cluster-autoscaler/cloudprovider/azure/azure_cache.go index 1beb7b2b8feb..711e98cb8236 100644 --- a/cluster-autoscaler/cloudprovider/azure/azure_cache.go +++ b/cluster-autoscaler/cloudprovider/azure/azure_cache.go @@ -51,7 +51,6 @@ type azureCache struct { // Cache content. resourceGroup string vmType string - vmsPoolSet map[string]struct{} // track the nodepools that're vms pool scaleSets map[string]compute.VirtualMachineScaleSet virtualMachines map[string][]compute.VirtualMachine registeredNodeGroups []cloudprovider.NodeGroup @@ -68,7 +67,6 @@ func newAzureCache(client *azClient, cacheTTL time.Duration, resourceGroup, vmTy refreshInterval: cacheTTL, resourceGroup: resourceGroup, vmType: vmType, - vmsPoolSet: make(map[string]struct{}), scaleSets: make(map[string]compute.VirtualMachineScaleSet), virtualMachines: make(map[string][]compute.VirtualMachine), registeredNodeGroups: make([]cloudprovider.NodeGroup, 0), @@ -89,13 +87,6 @@ func newAzureCache(client *azClient, cacheTTL time.Duration, resourceGroup, vmTy return cache, nil } -func (m *azureCache) getVMsPoolSet() map[string]struct{} { - m.mutex.Lock() - defer m.mutex.Unlock() - - return m.vmsPoolSet -} - func (m *azureCache) getVirtualMachines() map[string][]compute.VirtualMachine { m.mutex.Lock() defer m.mutex.Unlock() @@ -174,78 +165,54 @@ func (m *azureCache) fetchAzureResources() error { m.mutex.Lock() defer m.mutex.Unlock() - // fetch all the resources since CAS may be operating on mixed nodepools - // including both VMSS and VMs pools - vmssResult, err := m.fetchScaleSets() - if err == nil { - m.scaleSets = vmssResult - } else { - return err - } - - vmResult, vmsPoolSet, err := m.fetchVirtualMachines() - if err == nil { - m.virtualMachines = vmResult - m.vmsPoolSet = vmsPoolSet - } else { - return err + switch m.vmType { + case vmTypeVMSS: + // List all VMSS in the RG. + vmssResult, err := m.fetchScaleSets() + if err == nil { + m.scaleSets = vmssResult + } else { + return err + } + case vmTypeStandard: + // List all VMs in the RG. + vmResult, err := m.fetchVirtualMachines() + if err == nil { + m.virtualMachines = vmResult + } else { + return err + } } return nil } -const ( - legacyAgentpoolNameTag = "poolName" - agentpoolNameTag = "aks-managed-poolName" - agentpoolTypeTag = "aks-managed-agentpool-type" - vmsPoolType = "VirtualMachines" -) - // fetchVirtualMachines returns the updated list of virtual machines in the config resource group using the Azure API. -func (m *azureCache) fetchVirtualMachines() (map[string][]compute.VirtualMachine, map[string]struct{}, error) { +func (m *azureCache) fetchVirtualMachines() (map[string][]compute.VirtualMachine, error) { ctx, cancel := getContextWithCancel() defer cancel() result, err := m.azClient.virtualMachinesClient.List(ctx, m.resourceGroup) if err != nil { klog.Errorf("VirtualMachinesClient.List in resource group %q failed: %v", m.resourceGroup, err) - return nil, nil, err.Error() + return nil, err.Error() } instances := make(map[string][]compute.VirtualMachine) - // track the nodepools that're vms pools - vmsPoolSet := make(map[string]struct{}) for _, instance := range result { if instance.Tags == nil { continue } tags := instance.Tags - vmPoolName := tags[agentpoolNameTag] - // fall back to legacy tag name if not found - if vmPoolName == nil { - vmPoolName = tags[legacyAgentpoolNameTag] - } - + vmPoolName := tags["poolName"] if vmPoolName == nil { continue } instances[to.String(vmPoolName)] = append(instances[to.String(vmPoolName)], instance) - - // if the nodepool is already in the map, skip it - if _, ok := vmsPoolSet[to.String(vmPoolName)]; ok { - continue - } - - // nodes from vms pool will have tag "aks-managed-agentpool-type" set to "VirtualMachines" - if agnetpoolType := tags[agentpoolTypeTag]; agnetpoolType != nil { - if strings.EqualFold(to.String(agnetpoolType), vmsPoolType) { - vmsPoolSet[to.String(vmPoolName)] = struct{}{} - } - } } - return instances, vmsPoolSet, nil + return instances, nil } // fetchScaleSets returns the updated list of scale sets in the config resource group using the Azure API. @@ -356,7 +323,6 @@ func (m *azureCache) getAutoscalingOptions(ref azureRef) map[string]string { // FindForInstance returns node group of the given Instance func (m *azureCache) FindForInstance(instance *azureRef, vmType string) (cloudprovider.NodeGroup, error) { - vmsPoolSet := m.getVMsPoolSet() m.mutex.Lock() defer m.mutex.Unlock() @@ -374,8 +340,7 @@ func (m *azureCache) FindForInstance(instance *azureRef, vmType string) (cloudpr return nil, nil } - // cluster with vmss pool only - if vmType == vmTypeVMSS && len(vmsPoolSet) == 0 { + if vmType == vmTypeVMSS { if m.areAllScaleSetsUniform() { // Omit virtual machines not managed by vmss only in case of uniform scale set. if ok := virtualMachineRE.Match([]byte(inst.Name)); ok { diff --git a/cluster-autoscaler/cloudprovider/azure/azure_cloud_provider.go b/cluster-autoscaler/cloudprovider/azure/azure_cloud_provider.go index 98b0ed185da3..744873284e6d 100644 --- a/cluster-autoscaler/cloudprovider/azure/azure_cloud_provider.go +++ b/cluster-autoscaler/cloudprovider/azure/azure_cloud_provider.go @@ -19,6 +19,7 @@ package azure import ( "io" "os" + "strings" apiv1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" @@ -105,6 +106,12 @@ func (azure *AzureCloudProvider) NodeGroupForNode(node *apiv1.Node) (cloudprovid klog.V(6).Infof("Skipping the search for node group for the node '%s' because it has no spec.ProviderID", node.ObjectMeta.Name) return nil, nil } + + if !strings.HasPrefix(node.Spec.ProviderID, "azure://") { + klog.V(6).Infof("Wrong azure ProviderID for node %v, skipped", node.Name) + return nil, nil + } + klog.V(6).Infof("Searching for node group for the node: %s\n", node.Spec.ProviderID) ref := &azureRef{ Name: node.Spec.ProviderID, diff --git a/cluster-autoscaler/cloudprovider/azure/azure_cloud_provider_test.go b/cluster-autoscaler/cloudprovider/azure/azure_cloud_provider_test.go index a7c52f56d11e..6038db4b5c43 100644 --- a/cluster-autoscaler/cloudprovider/azure/azure_cloud_provider_test.go +++ b/cluster-autoscaler/cloudprovider/azure/azure_cloud_provider_test.go @@ -44,9 +44,6 @@ func newTestAzureManager(t *testing.T) *AzureManager { mockVMSSClient.EXPECT().List(gomock.Any(), "rg").Return(expectedScaleSets, nil).AnyTimes() mockVMSSVMClient := mockvmssvmclient.NewMockInterface(ctrl) mockVMSSVMClient.EXPECT().List(gomock.Any(), "rg", "test-vmss", gomock.Any()).Return(expectedVMSSVMs, nil).AnyTimes() - mockVMClient := mockvmclient.NewMockInterface(ctrl) - expectedVMs := newTestVMList(3) - mockVMClient.EXPECT().List(gomock.Any(), "rg").Return(expectedVMs, nil).AnyTimes() manager := &AzureManager{ env: azure.PublicCloud, @@ -56,11 +53,11 @@ func newTestAzureManager(t *testing.T) *AzureManager { VMType: vmTypeVMSS, MaxDeploymentsCount: 2, Deployment: "deployment", + EnableForceDelete: true, }, azClient: &azClient{ virtualMachineScaleSetsClient: mockVMSSClient, virtualMachineScaleSetVMsClient: mockVMSSVMClient, - virtualMachinesClient: mockVMClient, deploymentsClient: &DeploymentsClientMock{ FakeStore: map[string]resources.DeploymentExtended{ "deployment": { @@ -119,68 +116,9 @@ func TestNodeGroups(t *testing.T) { assert.Equal(t, len(provider.NodeGroups()), 0) registered := provider.azureManager.RegisterNodeGroup( - newTestScaleSet(provider.azureManager, "test-asg"), - ) - assert.True(t, registered) - registered = provider.azureManager.RegisterNodeGroup( - newTestVMsPool(provider.azureManager, "test-vms-pool"), - ) - assert.True(t, registered) - assert.Equal(t, len(provider.NodeGroups()), 2) -} - -func TestMixedNodeGroups(t *testing.T) { - ctrl := gomock.NewController(t) - provider := newTestProvider(t) - mockVMSSClient := mockvmssclient.NewMockInterface(ctrl) - mockVMClient := mockvmclient.NewMockInterface(ctrl) - mockVMSSVMClient := mockvmssvmclient.NewMockInterface(ctrl) - provider.azureManager.azClient.virtualMachinesClient = mockVMClient - provider.azureManager.azClient.virtualMachineScaleSetsClient = mockVMSSClient - provider.azureManager.azClient.virtualMachineScaleSetVMsClient = mockVMSSVMClient - - expectedScaleSets := newTestVMSSList(3, "test-asg", "eastus", compute.Uniform) - expectedVMsPoolVMs := newTestVMsPoolVMList(3) - expectedVMSSVMs := newTestVMSSVMList(3) - - mockVMSSClient.EXPECT().List(gomock.Any(), provider.azureManager.config.ResourceGroup).Return(expectedScaleSets, nil).AnyTimes() - mockVMClient.EXPECT().List(gomock.Any(), provider.azureManager.config.ResourceGroup).Return(expectedVMsPoolVMs, nil).AnyTimes() - mockVMSSVMClient.EXPECT().List(gomock.Any(), provider.azureManager.config.ResourceGroup, "test-asg", gomock.Any()).Return(expectedVMSSVMs, nil).AnyTimes() - - assert.Equal(t, len(provider.NodeGroups()), 0) - registered := provider.azureManager.RegisterNodeGroup( - newTestScaleSet(provider.azureManager, "test-asg"), - ) - provider.azureManager.explicitlyConfigured["test-asg"] = true - assert.True(t, registered) - - registered = provider.azureManager.RegisterNodeGroup( - newTestVMsPool(provider.azureManager, "test-vms-pool"), - ) - provider.azureManager.explicitlyConfigured["test-vms-pool"] = true + newTestScaleSet(provider.azureManager, "test-asg")) assert.True(t, registered) - assert.Equal(t, len(provider.NodeGroups()), 2) - - // refresh cache - provider.azureManager.forceRefresh() - - // node from vmss pool - node := newApiNode(compute.Uniform, 0) - group, err := provider.NodeGroupForNode(node) - assert.NoError(t, err) - assert.NotNil(t, group, "Group should not be nil") - assert.Equal(t, group.Id(), "test-asg") - assert.Equal(t, group.MinSize(), 1) - assert.Equal(t, group.MaxSize(), 5) - - // node from vms pool - vmsPoolNode := newVMsNode(0) - group, err = provider.NodeGroupForNode(vmsPoolNode) - assert.NoError(t, err) - assert.NotNil(t, group, "Group should not be nil") - assert.Equal(t, group.Id(), "test-vms-pool") - assert.Equal(t, group.MinSize(), 3) - assert.Equal(t, group.MaxSize(), 10) + assert.Equal(t, len(provider.NodeGroups()), 1) } func TestNodeGroupForNode(t *testing.T) { @@ -198,9 +136,6 @@ func TestNodeGroupForNode(t *testing.T) { mockVMSSClient := mockvmssclient.NewMockInterface(ctrl) mockVMSSClient.EXPECT().List(gomock.Any(), provider.azureManager.config.ResourceGroup).Return(expectedScaleSets, nil) provider.azureManager.azClient.virtualMachineScaleSetsClient = mockVMSSClient - mockVMClient := mockvmclient.NewMockInterface(ctrl) - provider.azureManager.azClient.virtualMachinesClient = mockVMClient - mockVMClient.EXPECT().List(gomock.Any(), provider.azureManager.config.ResourceGroup).Return(expectedVMs, nil).AnyTimes() if orchMode == compute.Uniform { @@ -209,8 +144,11 @@ func TestNodeGroupForNode(t *testing.T) { provider.azureManager.azClient.virtualMachineScaleSetVMsClient = mockVMSSVMClient } else { + mockVMClient := mockvmclient.NewMockInterface(ctrl) provider.azureManager.config.EnableVmssFlex = true mockVMClient.EXPECT().ListVmssFlexVMsWithoutInstanceView(gomock.Any(), "test-asg").Return(expectedVMs, nil).AnyTimes() + provider.azureManager.azClient.virtualMachinesClient = mockVMClient + } registered := provider.azureManager.RegisterNodeGroup( diff --git a/cluster-autoscaler/cloudprovider/azure/azure_config.go b/cluster-autoscaler/cloudprovider/azure/azure_config.go index 9fe559d07e14..2f6f5f279ab2 100644 --- a/cluster-autoscaler/cloudprovider/azure/azure_config.go +++ b/cluster-autoscaler/cloudprovider/azure/azure_config.go @@ -130,6 +130,9 @@ type Config struct { CloudProviderBackoffDuration int `json:"cloudProviderBackoffDuration,omitempty" yaml:"cloudProviderBackoffDuration,omitempty"` CloudProviderBackoffJitter float64 `json:"cloudProviderBackoffJitter,omitempty" yaml:"cloudProviderBackoffJitter,omitempty"` + // EnableForceDelete defines whether to enable force deletion on the APIs + EnableForceDelete bool `json:"enableForceDelete,omitempty" yaml:"enableForceDelete,omitempty"` + // EnableDynamicInstanceList defines whether to enable dynamic instance workflow for instance information check EnableDynamicInstanceList bool `json:"enableDynamicInstanceList,omitempty" yaml:"enableDynamicInstanceList,omitempty"` @@ -303,6 +306,13 @@ func BuildAzureConfig(configReader io.Reader) (*Config, error) { } } + if enableForceDelete := os.Getenv("AZURE_ENABLE_FORCE_DELETE"); enableForceDelete != "" { + cfg.EnableForceDelete, err = strconv.ParseBool(enableForceDelete) + if err != nil { + return nil, fmt.Errorf("failed to parse AZURE_ENABLE_FORCE_DELETE: %q, %v", enableForceDelete, err) + } + } + err = initializeCloudProviderRateLimitConfig(&cfg.CloudProviderRateLimitConfig) if err != nil { return nil, err diff --git a/cluster-autoscaler/cloudprovider/azure/azure_manager.go b/cluster-autoscaler/cloudprovider/azure/azure_manager.go index 637b5a897805..daa449f3cf09 100644 --- a/cluster-autoscaler/cloudprovider/azure/azure_manager.go +++ b/cluster-autoscaler/cloudprovider/azure/azure_manager.go @@ -26,10 +26,13 @@ import ( "time" "github.com/Azure/go-autorest/autorest/azure" + "k8s.io/apimachinery/pkg/util/wait" "k8s.io/autoscaler/cluster-autoscaler/cloudprovider" "k8s.io/autoscaler/cluster-autoscaler/config" "k8s.io/autoscaler/cluster-autoscaler/config/dynamic" + kretry "k8s.io/client-go/util/retry" klog "k8s.io/klog/v2" + "sigs.k8s.io/cloud-provider-azure/pkg/retry" ) const ( @@ -106,7 +109,18 @@ func createAzureManagerInternal(configReader io.Reader, discoveryOpts cloudprovi return nil, err } - if err := manager.forceRefresh(); err != nil { + retryBackoff := wait.Backoff{ + Duration: 2 * time.Minute, + Factor: 1.0, + Jitter: 0.1, + Steps: 6, + Cap: 10 * time.Minute, + } + + err = kretry.OnError(retryBackoff, retry.IsErrorRetriable, func() (err error) { + return manager.forceRefresh() + }) + if err != nil { return nil, err } @@ -147,11 +161,6 @@ func (m *AzureManager) buildNodeGroupFromSpec(spec string) (cloudprovider.NodeGr return nil, fmt.Errorf("failed to parse node group spec: %v", err) } - vmsPoolSet := m.azureCache.getVMsPoolSet() - if _, ok := vmsPoolSet[s.Name]; ok { - return NewVMsPool(s, m), nil - } - switch m.config.VMType { case vmTypeStandard: return NewAgentPool(s, m) diff --git a/cluster-autoscaler/cloudprovider/azure/azure_manager_test.go b/cluster-autoscaler/cloudprovider/azure/azure_manager_test.go index 83ca223e059b..3bad46343f50 100644 --- a/cluster-autoscaler/cloudprovider/azure/azure_manager_test.go +++ b/cluster-autoscaler/cloudprovider/azure/azure_manager_test.go @@ -144,7 +144,6 @@ func TestCreateAzureManagerValidConfig(t *testing.T) { mockVMClient := mockvmclient.NewMockInterface(ctrl) mockVMSSClient := mockvmssclient.NewMockInterface(ctrl) mockVMSSClient.EXPECT().List(gomock.Any(), "fakeId").Return([]compute.VirtualMachineScaleSet{}, nil).Times(2) - mockVMClient.EXPECT().List(gomock.Any(), "fakeId").Return([]compute.VirtualMachine{}, nil).Times(2) mockAzClient := &azClient{ virtualMachinesClient: mockVMClient, virtualMachineScaleSetsClient: mockVMSSClient, @@ -227,7 +226,6 @@ func TestCreateAzureManagerValidConfigForStandardVMType(t *testing.T) { mockVMClient := mockvmclient.NewMockInterface(ctrl) mockVMClient.EXPECT().List(gomock.Any(), "fakeId").Return([]compute.VirtualMachine{}, nil).Times(2) mockVMSSClient := mockvmssclient.NewMockInterface(ctrl) - mockVMSSClient.EXPECT().List(gomock.Any(), "fakeId").Return([]compute.VirtualMachineScaleSet{}, nil).Times(2) mockAzClient := &azClient{ virtualMachinesClient: mockVMClient, virtualMachineScaleSetsClient: mockVMSSClient, @@ -340,7 +338,6 @@ func TestCreateAzureManagerWithNilConfig(t *testing.T) { mockVMClient := mockvmclient.NewMockInterface(ctrl) mockVMSSClient := mockvmssclient.NewMockInterface(ctrl) mockVMSSClient.EXPECT().List(gomock.Any(), "resourceGroup").Return([]compute.VirtualMachineScaleSet{}, nil).AnyTimes() - mockVMClient.EXPECT().List(gomock.Any(), "resourceGroup").Return([]compute.VirtualMachine{}, nil).AnyTimes() mockAzClient := &azClient{ virtualMachinesClient: mockVMClient, virtualMachineScaleSetsClient: mockVMSSClient, @@ -666,10 +663,7 @@ func TestGetFilteredAutoscalingGroupsVmss(t *testing.T) { expectedScaleSets := []compute.VirtualMachineScaleSet{fakeVMSSWithTags(vmssName, map[string]*string{vmssTag: &vmssTagValue, "min": &min, "max": &max})} mockVMSSClient := mockvmssclient.NewMockInterface(ctrl) mockVMSSClient.EXPECT().List(gomock.Any(), manager.config.ResourceGroup).Return(expectedScaleSets, nil).AnyTimes() - mockVMClient := mockvmclient.NewMockInterface(ctrl) - mockVMClient.EXPECT().List(gomock.Any(), manager.config.ResourceGroup).Return([]compute.VirtualMachine{}, nil).AnyTimes() manager.azClient.virtualMachineScaleSetsClient = mockVMSSClient - manager.azClient.virtualMachinesClient = mockVMClient err := manager.forceRefresh() assert.NoError(t, err) @@ -685,6 +679,7 @@ func TestGetFilteredAutoscalingGroupsVmss(t *testing.T) { minSize: minVal, maxSize: maxVal, manager: manager, + enableForceDelete: manager.config.EnableForceDelete, curSize: 3, sizeRefreshPeriod: manager.azureCache.refreshInterval, instancesRefreshPeriod: defaultVmssInstancesRefreshPeriod, @@ -742,9 +737,6 @@ func TestFetchAutoAsgsVmss(t *testing.T) { mockVMSSVMClient := mockvmssvmclient.NewMockInterface(ctrl) mockVMSSVMClient.EXPECT().List(gomock.Any(), manager.config.ResourceGroup, vmssName, gomock.Any()).Return(expectedVMSSVMs, nil).AnyTimes() manager.azClient.virtualMachineScaleSetVMsClient = mockVMSSVMClient - mockVMClient := mockvmclient.NewMockInterface(ctrl) - manager.azClient.virtualMachinesClient = mockVMClient - mockVMClient.EXPECT().List(gomock.Any(), manager.config.ResourceGroup).Return([]compute.VirtualMachine{}, nil).AnyTimes() err := manager.forceRefresh() assert.NoError(t, err) diff --git a/cluster-autoscaler/cloudprovider/azure/azure_scale_set.go b/cluster-autoscaler/cloudprovider/azure/azure_scale_set.go index f0b2e83f07e8..e7c6079c2206 100644 --- a/cluster-autoscaler/cloudprovider/azure/azure_scale_set.go +++ b/cluster-autoscaler/cloudprovider/azure/azure_scale_set.go @@ -58,6 +58,8 @@ type ScaleSet struct { minSize int maxSize int + enableForceDelete bool + sizeMutex sync.Mutex curSize int64 @@ -87,6 +89,7 @@ func NewScaleSet(spec *dynamic.NodeGroupSpec, az *AzureManager, curSize int64) ( sizeRefreshPeriod: az.azureCache.refreshInterval, enableDynamicInstanceList: az.config.EnableDynamicInstanceList, instancesRefreshJitter: az.config.VmssVmsCacheJitter, + enableForceDelete: az.config.EnableForceDelete, } if az.config.VmssVmsCacheTTL != 0 { @@ -251,6 +254,16 @@ func (scaleSet *ScaleSet) SetScaleSetSize(size int64) error { Sku: vmssInfo.Sku, Location: vmssInfo.Location, } + + if vmssInfo.ExtendedLocation != nil { + op.ExtendedLocation = &compute.ExtendedLocation{ + Name: vmssInfo.ExtendedLocation.Name, + Type: vmssInfo.ExtendedLocation.Type, + } + + klog.V(3).Infof("Passing ExtendedLocation information if it is not nil, with Edge Zone name:(%s)", *op.ExtendedLocation.Name) + } + ctx, cancel := getContextWithTimeout(vmssContextTimeout) defer cancel() klog.V(3).Infof("Waiting for virtualMachineScaleSetsClient.CreateOrUpdateAsync(%s)", scaleSet.Name) @@ -297,6 +310,11 @@ func (scaleSet *ScaleSet) IncreaseSize(delta int) error { return scaleSet.SetScaleSetSize(size + int64(delta)) } +// AtomicIncreaseSize is not implemented. +func (scaleSet *ScaleSet) AtomicIncreaseSize(delta int) error { + return cloudprovider.ErrNotImplemented +} + // GetScaleSetVms returns list of nodes for the given scale set. func (scaleSet *ScaleSet) GetScaleSetVms() ([]compute.VirtualMachineScaleSetVM, *retry.Error) { klog.V(4).Infof("GetScaleSetVms: starts") @@ -432,8 +450,15 @@ func (scaleSet *ScaleSet) DeleteInstances(instances []*azureRef, hasUnregistered resourceGroup := scaleSet.manager.config.ResourceGroup scaleSet.instanceMutex.Lock() - klog.V(3).Infof("Calling virtualMachineScaleSetsClient.DeleteInstancesAsync(%v)", requiredIds.InstanceIds) - future, rerr := scaleSet.manager.azClient.virtualMachineScaleSetsClient.DeleteInstancesAsync(ctx, resourceGroup, commonAsg.Id(), *requiredIds, false) + klog.V(3).Infof("Calling virtualMachineScaleSetsClient.DeleteInstancesAsync(%v), force delete set to %v", requiredIds.InstanceIds, scaleSet.enableForceDelete) + future, rerr := scaleSet.manager.azClient.virtualMachineScaleSetsClient.DeleteInstancesAsync(ctx, resourceGroup, commonAsg.Id(), *requiredIds, scaleSet.enableForceDelete) + + if scaleSet.enableForceDelete && isOperationNotAllowed(rerr) { + klog.Infof("falling back to normal delete for instances %v for %s", requiredIds.InstanceIds, scaleSet.Name) + future, rerr = scaleSet.manager.azClient.virtualMachineScaleSetsClient.DeleteInstancesAsync(ctx, resourceGroup, + commonAsg.Id(), *requiredIds, false) + } + scaleSet.instanceMutex.Unlock() if rerr != nil { klog.Errorf("virtualMachineScaleSetsClient.DeleteInstancesAsync for instances %v failed: %v", requiredIds.InstanceIds, rerr) @@ -741,3 +766,7 @@ func (scaleSet *ScaleSet) getOrchestrationMode() (compute.OrchestrationMode, err } return vmss.OrchestrationMode, nil } + +func isOperationNotAllowed(rerr *retry.Error) bool { + return rerr != nil && rerr.ServiceErrorCode() == retry.OperationNotAllowed +} diff --git a/cluster-autoscaler/cloudprovider/azure/azure_scale_set_test.go b/cluster-autoscaler/cloudprovider/azure/azure_scale_set_test.go index 85ee14c4c483..edfdb1fa08eb 100644 --- a/cluster-autoscaler/cloudprovider/azure/azure_scale_set_test.go +++ b/cluster-autoscaler/cloudprovider/azure/azure_scale_set_test.go @@ -32,14 +32,32 @@ import ( "sigs.k8s.io/cloud-provider-azure/pkg/azureclients/vmssvmclient/mockvmssvmclient" ) +const ( + testASG = "test-asg" + testLocation = "eastus" +) + func newTestScaleSet(manager *AzureManager, name string) *ScaleSet { return &ScaleSet{ azureRef: azureRef{ Name: name, }, - manager: manager, - minSize: 1, - maxSize: 5, + manager: manager, + minSize: 1, + maxSize: 5, + enableForceDelete: manager.config.EnableForceDelete, + } +} + +func newTestScaleSetMinSizeZero(manager *AzureManager, name string) *ScaleSet { + return &ScaleSet{ + azureRef: azureRef{ + Name: name, + }, + manager: manager, + minSize: 0, + maxSize: 5, + enableForceDelete: manager.config.EnableForceDelete, } } @@ -60,6 +78,22 @@ func newTestVMSSList(cap int64, name, loc string, orchmode compute.Orchestration } } +func newTestVMSSListForEdgeZones(capacity int64, name string) *compute.VirtualMachineScaleSet { + return &compute.VirtualMachineScaleSet{ + Name: to.StringPtr(name), + Sku: &compute.Sku{ + Capacity: to.Int64Ptr(capacity), + Name: to.StringPtr("Standard_D4_v2"), + }, + VirtualMachineScaleSetProperties: &compute.VirtualMachineScaleSetProperties{}, + Location: to.StringPtr(testLocation), + ExtendedLocation: &compute.ExtendedLocation{ + Name: to.StringPtr("losangeles"), + Type: compute.ExtendedLocationTypes("EdgeZone"), + }, + } +} + func newTestVMSSVMList(count int) []compute.VirtualMachineScaleSetVM { var vmssVMList []compute.VirtualMachineScaleSetVM for i := 0; i < count; i++ { @@ -121,6 +155,15 @@ func TestMinSize(t *testing.T) { assert.Equal(t, provider.NodeGroups()[0].MinSize(), 1) } +func TestMinSizeZero(t *testing.T) { + provider := newTestProvider(t) + registered := provider.azureManager.RegisterNodeGroup( + newTestScaleSetMinSizeZero(provider.azureManager, testASG)) + assert.True(t, registered) + assert.Equal(t, len(provider.NodeGroups()), 1) + assert.Equal(t, provider.NodeGroups()[0].MinSize(), 0) +} + func TestTargetSize(t *testing.T) { ctrl := gomock.NewController(t) defer ctrl.Finish() @@ -136,9 +179,6 @@ func TestTargetSize(t *testing.T) { mockVMSSClient := mockvmssclient.NewMockInterface(ctrl) mockVMSSClient.EXPECT().List(gomock.Any(), provider.azureManager.config.ResourceGroup).Return(expectedScaleSets, nil).AnyTimes() provider.azureManager.azClient.virtualMachineScaleSetsClient = mockVMSSClient - mockVMClient := mockvmclient.NewMockInterface(ctrl) - mockVMClient.EXPECT().List(gomock.Any(), provider.azureManager.config.ResourceGroup).Return([]compute.VirtualMachine{}, nil).AnyTimes() - provider.azureManager.azClient.virtualMachinesClient = mockVMClient if orchMode == compute.Uniform { @@ -148,7 +188,9 @@ func TestTargetSize(t *testing.T) { } else { provider.azureManager.config.EnableVmssFlex = true + mockVMClient := mockvmclient.NewMockInterface(ctrl) mockVMClient.EXPECT().ListVmssFlexVMsWithoutInstanceView(gomock.Any(), "test-asg").Return(expectedVMs, nil).AnyTimes() + provider.azureManager.azClient.virtualMachinesClient = mockVMClient } err := provider.azureManager.forceRefresh() @@ -177,26 +219,34 @@ func TestIncreaseSize(t *testing.T) { for _, orchMode := range orchestrationModes { provider := newTestProvider(t) - expectedScaleSets := newTestVMSSList(3, "test-asg", "eastus", orchMode) + expectedScaleSets := newTestVMSSList(3, testASG, testLocation, orchMode) + + // Include Edge Zone scenario here, testing scale from 3 to 5 and scale from zero cases. + expectedEdgeZoneScaleSets := newTestVMSSListForEdgeZones(3, "edgezone-vmss") + expectedEdgeZoneMinZeroScaleSets := newTestVMSSListForEdgeZones(0, "edgezone-minzero-vmss") + expectedScaleSets = append(expectedScaleSets, *expectedEdgeZoneScaleSets, *expectedEdgeZoneMinZeroScaleSets) mockVMSSClient := mockvmssclient.NewMockInterface(ctrl) mockVMSSClient.EXPECT().List(gomock.Any(), provider.azureManager.config.ResourceGroup).Return(expectedScaleSets, nil).AnyTimes() - mockVMSSClient.EXPECT().CreateOrUpdateAsync(gomock.Any(), provider.azureManager.config.ResourceGroup, "test-asg", gomock.Any()).Return(nil, nil) + mockVMSSClient.EXPECT().CreateOrUpdateAsync(gomock.Any(), provider.azureManager.config.ResourceGroup, testASG, gomock.Any()).Return(nil, nil) + // This should be Anytimes() because the parent function of this call - updateVMSSCapacity() is a goroutine + // and this test doesn't wait on goroutine, hence, it is difficult to write exact expected number (which is 3 here) + // before we return from this this. + // This is a future TODO: sync.WaitGroup should be used in actual code and make code easily testable mockVMSSClient.EXPECT().WaitForCreateOrUpdateResult(gomock.Any(), gomock.Any(), provider.azureManager.config.ResourceGroup).Return(&http.Response{StatusCode: http.StatusOK}, nil).AnyTimes() provider.azureManager.azClient.virtualMachineScaleSetsClient = mockVMSSClient - mockVMClient := mockvmclient.NewMockInterface(ctrl) - mockVMClient.EXPECT().List(gomock.Any(), provider.azureManager.config.ResourceGroup).Return([]compute.VirtualMachine{}, nil).AnyTimes() - provider.azureManager.azClient.virtualMachinesClient = mockVMClient if orchMode == compute.Uniform { mockVMSSVMClient := mockvmssvmclient.NewMockInterface(ctrl) - mockVMSSVMClient.EXPECT().List(gomock.Any(), provider.azureManager.config.ResourceGroup, "test-asg", gomock.Any()).Return(expectedVMSSVMs, nil).AnyTimes() + mockVMSSVMClient.EXPECT().List(gomock.Any(), provider.azureManager.config.ResourceGroup, testASG, gomock.Any()).Return(expectedVMSSVMs, nil).AnyTimes() provider.azureManager.azClient.virtualMachineScaleSetVMsClient = mockVMSSVMClient } else { provider.azureManager.config.EnableVmssFlex = true - mockVMClient.EXPECT().ListVmssFlexVMsWithoutInstanceView(gomock.Any(), "test-asg").Return(expectedVMs, nil).AnyTimes() + mockVMClient := mockvmclient.NewMockInterface(ctrl) + mockVMClient.EXPECT().ListVmssFlexVMsWithoutInstanceView(gomock.Any(), testASG).Return(expectedVMs, nil).AnyTimes() + provider.azureManager.azClient.virtualMachinesClient = mockVMClient } err := provider.azureManager.forceRefresh() assert.NoError(t, err) @@ -207,23 +257,63 @@ func TestIncreaseSize(t *testing.T) { assert.Equal(t, expectedErr, err) registered := provider.azureManager.RegisterNodeGroup( - newTestScaleSet(provider.azureManager, "test-asg")) + newTestScaleSet(provider.azureManager, testASG)) assert.True(t, registered) assert.Equal(t, len(provider.NodeGroups()), 1) - // current target size is 2. + // Current target size is 3. targetSize, err := provider.NodeGroups()[0].TargetSize() assert.NoError(t, err) assert.Equal(t, 3, targetSize) - // increase 3 nodes. + // Increase 2 nodes. err = provider.NodeGroups()[0].IncreaseSize(2) assert.NoError(t, err) - // new target size should be 5. + // New target size should be 5. targetSize, err = provider.NodeGroups()[0].TargetSize() assert.NoError(t, err) assert.Equal(t, 5, targetSize) + + // Testing Edge Zone scenario. Scale from 3 to 5. + registeredForEdgeZone := provider.azureManager.RegisterNodeGroup( + newTestScaleSet(provider.azureManager, "edgezone-vmss")) + assert.True(t, registeredForEdgeZone) + assert.Equal(t, len(provider.NodeGroups()), 2) + + targetSizeForEdgeZone, err := provider.NodeGroups()[1].TargetSize() + assert.NoError(t, err) + assert.Equal(t, 3, targetSizeForEdgeZone) + + mockVMSSClient.EXPECT().CreateOrUpdateAsync(gomock.Any(), provider.azureManager.config.ResourceGroup, + "edgezone-vmss", gomock.Any()).Return(nil, nil) + err = provider.NodeGroups()[1].IncreaseSize(2) + assert.NoError(t, err) + + targetSizeForEdgeZone, err = provider.NodeGroups()[1].TargetSize() + assert.NoError(t, err) + assert.Equal(t, 5, targetSizeForEdgeZone) + + // Testing Edge Zone scenario scaleFromZero case. Scale from 0 to 2. + registeredForEdgeZoneMinZero := provider.azureManager.RegisterNodeGroup( + newTestScaleSetMinSizeZero(provider.azureManager, "edgezone-minzero-vmss")) + assert.True(t, registeredForEdgeZoneMinZero) + assert.Equal(t, len(provider.NodeGroups()), 3) + + // Current target size is 0. + targetSizeForEdgeZoneMinZero, err := provider.NodeGroups()[2].TargetSize() + assert.NoError(t, err) + assert.Equal(t, 0, targetSizeForEdgeZoneMinZero) + + mockVMSSClient.EXPECT().CreateOrUpdateAsync(gomock.Any(), provider.azureManager.config.ResourceGroup, + "edgezone-minzero-vmss", gomock.Any()).Return(nil, nil) + err = provider.NodeGroups()[2].IncreaseSize(2) + assert.NoError(t, err) + + // New target size should be 2. + targetSizeForEdgeZoneMinZero, err = provider.NodeGroups()[2].TargetSize() + assert.NoError(t, err) + assert.Equal(t, 2, targetSizeForEdgeZoneMinZero) } } @@ -259,7 +349,6 @@ func TestIncreaseSizeOnVMProvisioningFailed(t *testing.T) { expectedScaleSets := newTestVMSSList(3, "vmss-failed-upscale", "eastus", compute.Uniform) expectedVMSSVMs := newTestVMSSVMList(3) - expectedVMs := newTestVMList(3) expectedVMSSVMs[2].ProvisioningState = to.StringPtr(provisioningStateFailed) if !testCase.isMissingInstanceView { expectedVMSSVMs[2].InstanceView = &compute.VirtualMachineScaleSetVMInstanceView{Statuses: &testCase.statuses} @@ -273,9 +362,6 @@ func TestIncreaseSizeOnVMProvisioningFailed(t *testing.T) { mockVMSSVMClient := mockvmssvmclient.NewMockInterface(ctrl) mockVMSSVMClient.EXPECT().List(gomock.Any(), manager.config.ResourceGroup, "vmss-failed-upscale", gomock.Any()).Return(expectedVMSSVMs, nil).AnyTimes() manager.azClient.virtualMachineScaleSetVMsClient = mockVMSSVMClient - mockVMClient := mockvmclient.NewMockInterface(ctrl) - mockVMClient.EXPECT().List(gomock.Any(), manager.config.ResourceGroup).Return(expectedVMs, nil).AnyTimes() - manager.azClient.virtualMachinesClient = mockVMClient manager.explicitlyConfigured["vmss-failed-upscale"] = true registered := manager.RegisterNodeGroup(newTestScaleSet(manager, vmssName)) assert.True(t, registered) @@ -365,9 +451,6 @@ func TestBelongs(t *testing.T) { mockVMSSClient := mockvmssclient.NewMockInterface(ctrl) mockVMSSClient.EXPECT().List(gomock.Any(), provider.azureManager.config.ResourceGroup).Return(expectedScaleSets, nil) provider.azureManager.azClient.virtualMachineScaleSetsClient = mockVMSSClient - mockVMClient := mockvmclient.NewMockInterface(ctrl) - mockVMClient.EXPECT().List(gomock.Any(), provider.azureManager.config.ResourceGroup).Return([]compute.VirtualMachine{}, nil).AnyTimes() - provider.azureManager.azClient.virtualMachinesClient = mockVMClient if orchMode == compute.Uniform { @@ -378,7 +461,9 @@ func TestBelongs(t *testing.T) { } else { provider.azureManager.config.EnableVmssFlex = true + mockVMClient := mockvmclient.NewMockInterface(ctrl) mockVMClient.EXPECT().ListVmssFlexVMsWithoutInstanceView(gomock.Any(), "test-asg").Return(expectedVMs, nil).AnyTimes() + provider.azureManager.azClient.virtualMachinesClient = mockVMClient } registered := provider.azureManager.RegisterNodeGroup( @@ -412,25 +497,53 @@ func TestDeleteNodes(t *testing.T) { vmssName := "test-asg" var vmssCapacity int64 = 3 - orchestrationModes := [2]compute.OrchestrationMode{compute.Uniform, compute.Flexible} - expectedVMSSVMs := newTestVMSSVMList(3) - expectedVMs := newTestVMList(3) + cases := []struct { + name string + orchestrationMode compute.OrchestrationMode + enableForceDelete bool + }{ + { + name: "uniform, force delete enabled", + orchestrationMode: compute.Uniform, + enableForceDelete: true, + }, + { + name: "uniform, force delete disabled", + orchestrationMode: compute.Uniform, + enableForceDelete: false, + }, + { + name: "flexible, force delete enabled", + orchestrationMode: compute.Flexible, + enableForceDelete: true, + }, + { + name: "flexible, force delete disabled", + orchestrationMode: compute.Flexible, + enableForceDelete: false, + }, + } - for _, orchMode := range orchestrationModes { + for _, tc := range cases { + orchMode := tc.orchestrationMode + enableForceDelete := tc.enableForceDelete + + expectedVMSSVMs := newTestVMSSVMList(3) + expectedVMs := newTestVMList(3) manager := newTestAzureManager(t) + manager.config.EnableForceDelete = enableForceDelete expectedScaleSets := newTestVMSSList(vmssCapacity, vmssName, "eastus", orchMode) + fmt.Printf("orchMode: %s, enableForceDelete: %t\n", orchMode, enableForceDelete) mockVMSSClient := mockvmssclient.NewMockInterface(ctrl) mockVMSSClient.EXPECT().List(gomock.Any(), manager.config.ResourceGroup).Return(expectedScaleSets, nil).Times(2) - mockVMSSClient.EXPECT().DeleteInstancesAsync(gomock.Any(), manager.config.ResourceGroup, gomock.Any(), gomock.Any(), false).Return(nil, nil) + mockVMSSClient.EXPECT().DeleteInstancesAsync(gomock.Any(), manager.config.ResourceGroup, gomock.Any(), gomock.Any(), enableForceDelete).Return(nil, nil) mockVMSSClient.EXPECT().WaitForDeleteInstancesResult(gomock.Any(), gomock.Any(), manager.config.ResourceGroup).Return(&http.Response{StatusCode: http.StatusOK}, nil).AnyTimes() manager.azClient.virtualMachineScaleSetsClient = mockVMSSClient mockVMSSVMClient := mockvmssvmclient.NewMockInterface(ctrl) mockVMClient := mockvmclient.NewMockInterface(ctrl) - manager.azClient.virtualMachinesClient = mockVMClient - mockVMClient.EXPECT().List(gomock.Any(), manager.config.ResourceGroup).Return(expectedVMs, nil).AnyTimes() if orchMode == compute.Uniform { mockVMSSVMClient.EXPECT().List(gomock.Any(), manager.config.ResourceGroup, "test-asg", gomock.Any()).Return(expectedVMSSVMs, nil).AnyTimes() @@ -438,6 +551,7 @@ func TestDeleteNodes(t *testing.T) { } else { manager.config.EnableVmssFlex = true mockVMClient.EXPECT().ListVmssFlexVMsWithoutInstanceView(gomock.Any(), "test-asg").Return(expectedVMs, nil).AnyTimes() + manager.azClient.virtualMachinesClient = mockVMClient } @@ -505,7 +619,6 @@ func TestDeleteNodes(t *testing.T) { instance2, found := scaleSet.getInstanceByProviderID(nodesToDelete[1].Spec.ProviderID) assert.True(t, found, true) assert.Equal(t, instance2.Status.State, cloudprovider.InstanceDeleting) - } } @@ -513,25 +626,51 @@ func TestDeleteNodeUnregistered(t *testing.T) { ctrl := gomock.NewController(t) defer ctrl.Finish() - orchestrationModes := [2]compute.OrchestrationMode{compute.Uniform, compute.Flexible} - vmssName := "test-asg" var vmssCapacity int64 = 2 - expectedVMSSVMs := newTestVMSSVMList(2) - expectedVMs := newTestVMList(2) - for _, orchMode := range orchestrationModes { + cases := []struct { + name string + orchestrationMode compute.OrchestrationMode + enableForceDelete bool + }{ + { + name: "uniform, force delete enabled", + orchestrationMode: compute.Uniform, + enableForceDelete: true, + }, + { + name: "uniform, force delete disabled", + orchestrationMode: compute.Uniform, + enableForceDelete: false, + }, + { + name: "flexible, force delete enabled", + orchestrationMode: compute.Flexible, + enableForceDelete: true, + }, + { + name: "flexible, force delete disabled", + orchestrationMode: compute.Flexible, + enableForceDelete: false, + }, + } + + for _, tc := range cases { + orchMode := tc.orchestrationMode + enableForceDelete := tc.enableForceDelete + expectedVMSSVMs := newTestVMSSVMList(2) + expectedVMs := newTestVMList(2) + manager := newTestAzureManager(t) + manager.config.EnableForceDelete = enableForceDelete expectedScaleSets := newTestVMSSList(vmssCapacity, vmssName, "eastus", orchMode) mockVMSSClient := mockvmssclient.NewMockInterface(ctrl) mockVMSSClient.EXPECT().List(gomock.Any(), manager.config.ResourceGroup).Return(expectedScaleSets, nil).Times(2) - mockVMSSClient.EXPECT().DeleteInstancesAsync(gomock.Any(), manager.config.ResourceGroup, gomock.Any(), gomock.Any(), false).Return(nil, nil) + mockVMSSClient.EXPECT().DeleteInstancesAsync(gomock.Any(), manager.config.ResourceGroup, gomock.Any(), gomock.Any(), enableForceDelete).Return(nil, nil) mockVMSSClient.EXPECT().WaitForDeleteInstancesResult(gomock.Any(), gomock.Any(), manager.config.ResourceGroup).Return(&http.Response{StatusCode: http.StatusOK}, nil).AnyTimes() manager.azClient.virtualMachineScaleSetsClient = mockVMSSClient - mockVMClient := mockvmclient.NewMockInterface(ctrl) - mockVMClient.EXPECT().List(gomock.Any(), manager.config.ResourceGroup).Return(expectedVMs, nil).AnyTimes() - manager.azClient.virtualMachinesClient = mockVMClient if orchMode == compute.Uniform { @@ -541,7 +680,9 @@ func TestDeleteNodeUnregistered(t *testing.T) { } else { manager.config.EnableVmssFlex = true + mockVMClient := mockvmclient.NewMockInterface(ctrl) mockVMClient.EXPECT().ListVmssFlexVMsWithoutInstanceView(gomock.Any(), "test-asg").Return(expectedVMs, nil).AnyTimes() + manager.azClient.virtualMachinesClient = mockVMClient } err := manager.forceRefresh() assert.NoError(t, err) @@ -687,9 +828,6 @@ func TestScaleSetNodes(t *testing.T) { mockVMSSClient := mockvmssclient.NewMockInterface(ctrl) mockVMSSClient.EXPECT().List(gomock.Any(), provider.azureManager.config.ResourceGroup).Return(expectedScaleSets, nil).AnyTimes() provider.azureManager.azClient.virtualMachineScaleSetsClient = mockVMSSClient - mockVMClient := mockvmclient.NewMockInterface(ctrl) - mockVMClient.EXPECT().List(gomock.Any(), provider.azureManager.config.ResourceGroup).Return([]compute.VirtualMachine{}, nil).AnyTimes() - provider.azureManager.azClient.virtualMachinesClient = mockVMClient if orchMode == compute.Uniform { @@ -699,7 +837,9 @@ func TestScaleSetNodes(t *testing.T) { } else { provider.azureManager.config.EnableVmssFlex = true + mockVMClient := mockvmclient.NewMockInterface(ctrl) mockVMClient.EXPECT().ListVmssFlexVMsWithoutInstanceView(gomock.Any(), "test-asg").Return(expectedVMs, nil).AnyTimes() + provider.azureManager.azClient.virtualMachinesClient = mockVMClient } registered := provider.azureManager.RegisterNodeGroup( @@ -754,7 +894,6 @@ func TestEnableVmssFlexFlag(t *testing.T) { provider.azureManager.config.EnableVmssFlex = false provider.azureManager.azClient.virtualMachineScaleSetsClient = mockVMSSClient mockVMClient := mockvmclient.NewMockInterface(ctrl) - mockVMClient.EXPECT().List(gomock.Any(), provider.azureManager.config.ResourceGroup).Return([]compute.VirtualMachine{}, nil).AnyTimes() mockVMClient.EXPECT().ListVmssFlexVMsWithoutInstanceView(gomock.Any(), "test-asg").Return(expectedVMs, nil).AnyTimes() provider.azureManager.azClient.virtualMachinesClient = mockVMClient diff --git a/cluster-autoscaler/cloudprovider/azure/azure_vms_pool.go b/cluster-autoscaler/cloudprovider/azure/azure_vms_pool.go deleted file mode 100644 index 015eaec4b20c..000000000000 --- a/cluster-autoscaler/cloudprovider/azure/azure_vms_pool.go +++ /dev/null @@ -1,174 +0,0 @@ -/* -Copyright 2017 The Kubernetes Authors. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package azure - -import ( - "fmt" - - "github.com/Azure/azure-sdk-for-go/services/compute/mgmt/2022-08-01/compute" - apiv1 "k8s.io/api/core/v1" - "k8s.io/autoscaler/cluster-autoscaler/cloudprovider" - "k8s.io/autoscaler/cluster-autoscaler/config" - "k8s.io/autoscaler/cluster-autoscaler/config/dynamic" - schedulerframework "k8s.io/kubernetes/pkg/scheduler/framework" -) - -// VMsPool is single instance VM pool -// this is a placeholder for now, no real implementation -type VMsPool struct { - azureRef - manager *AzureManager - resourceGroup string - - minSize int - maxSize int - - curSize int64 - // sizeMutex sync.Mutex - // lastSizeRefresh time.Time -} - -// NewVMsPool creates a new VMsPool -func NewVMsPool(spec *dynamic.NodeGroupSpec, am *AzureManager) *VMsPool { - nodepool := &VMsPool{ - azureRef: azureRef{ - Name: spec.Name, - }, - - manager: am, - resourceGroup: am.config.ResourceGroup, - - curSize: -1, - minSize: spec.MinSize, - maxSize: spec.MaxSize, - } - - return nodepool -} - -// MinSize returns the minimum size the cluster is allowed to scaled down -// to as provided by the node spec in --node parameter. -func (agentPool *VMsPool) MinSize() int { - return agentPool.minSize -} - -// Exist is always true since we are initialized with an existing agentpool -func (agentPool *VMsPool) Exist() bool { - return true -} - -// Create creates the node group on the cloud provider side. -func (agentPool *VMsPool) Create() (cloudprovider.NodeGroup, error) { - return nil, cloudprovider.ErrAlreadyExist -} - -// Delete deletes the node group on the cloud provider side. -func (agentPool *VMsPool) Delete() error { - return cloudprovider.ErrNotImplemented -} - -// Autoprovisioned is always false since we are initialized with an existing agentpool -func (agentPool *VMsPool) Autoprovisioned() bool { - return false -} - -// GetOptions returns NodeGroupAutoscalingOptions that should be used for this particular -// NodeGroup. Returning a nil will result in using default options. -func (agentPool *VMsPool) GetOptions(defaults config.NodeGroupAutoscalingOptions) (*config.NodeGroupAutoscalingOptions, error) { - // TODO(wenxuan): Implement this method - return nil, cloudprovider.ErrNotImplemented -} - -// MaxSize returns the maximum size scale limit provided by --node -// parameter to the autoscaler main -func (agentPool *VMsPool) MaxSize() int { - return agentPool.maxSize -} - -// TargetSize returns the current TARGET size of the node group. It is possible that the -// number is different from the number of nodes registered in Kubernetes. -func (agentPool *VMsPool) TargetSize() (int, error) { - // TODO(wenxuan): Implement this method - return -1, cloudprovider.ErrNotImplemented -} - -// IncreaseSize increase the size through a PUT AP call. It calculates the expected size -// based on a delta provided as parameter -func (agentPool *VMsPool) IncreaseSize(delta int) error { - // TODO(wenxuan): Implement this method - return cloudprovider.ErrNotImplemented -} - -// DeleteNodes extracts the providerIDs from the node spec and -// delete or deallocate the nodes from the agent pool based on the scale down policy. -func (agentPool *VMsPool) DeleteNodes(nodes []*apiv1.Node) error { - // TODO(wenxuan): Implement this method - return cloudprovider.ErrNotImplemented -} - -// DecreaseTargetSize decreases the target size of the node group. -func (agentPool *VMsPool) DecreaseTargetSize(delta int) error { - // TODO(wenxuan): Implement this method - return cloudprovider.ErrNotImplemented -} - -// Id returns the name of the agentPool -func (agentPool *VMsPool) Id() string { - return agentPool.azureRef.Name -} - -// Debug returns a string with basic details of the agentPool -func (agentPool *VMsPool) Debug() string { - return fmt.Sprintf("%s (%d:%d)", agentPool.Id(), agentPool.MinSize(), agentPool.MaxSize()) -} - -func (agentPool *VMsPool) getVMsFromCache() ([]compute.VirtualMachine, error) { - // vmsPoolMap is a map of agent pool name to the list of virtual machines - vmsPoolMap := agentPool.manager.azureCache.getVirtualMachines() - if _, ok := vmsPoolMap[agentPool.Name]; !ok { - return []compute.VirtualMachine{}, fmt.Errorf("vms pool %s not found in the cache", agentPool.Name) - } - - return vmsPoolMap[agentPool.Name], nil -} - -// Nodes returns the list of nodes in the vms agentPool. -func (agentPool *VMsPool) Nodes() ([]cloudprovider.Instance, error) { - vms, err := agentPool.getVMsFromCache() - if err != nil { - return nil, err - } - - nodes := make([]cloudprovider.Instance, 0, len(vms)) - for _, vm := range vms { - if len(*vm.ID) == 0 { - continue - } - resourceID, err := convertResourceGroupNameToLower("azure://" + *vm.ID) - if err != nil { - return nil, err - } - nodes = append(nodes, cloudprovider.Instance{Id: resourceID}) - } - - return nodes, nil -} - -// TemplateNodeInfo is not implemented. -func (agentPool *VMsPool) TemplateNodeInfo() (*schedulerframework.NodeInfo, error) { - return nil, cloudprovider.ErrNotImplemented -} diff --git a/cluster-autoscaler/cloudprovider/azure/azure_vms_pool_test.go b/cluster-autoscaler/cloudprovider/azure/azure_vms_pool_test.go deleted file mode 100644 index a3b0ebe45e4a..000000000000 --- a/cluster-autoscaler/cloudprovider/azure/azure_vms_pool_test.go +++ /dev/null @@ -1,67 +0,0 @@ -/* -Copyright 2017 The Kubernetes Authors. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package azure - -import ( - "fmt" - - "github.com/Azure/azure-sdk-for-go/services/compute/mgmt/2022-08-01/compute" - "github.com/Azure/go-autorest/autorest/to" - apiv1 "k8s.io/api/core/v1" -) - -func newTestVMsPool(manager *AzureManager, name string) *VMsPool { - return &VMsPool{ - azureRef: azureRef{ - Name: name, - }, - manager: manager, - minSize: 3, - maxSize: 10, - } -} - -const ( - fakeVMsPoolVMID = "/subscriptions/test-subscription-id/resourceGroups/test-rg/providers/Microsoft.Compute/virtualMachines/%d" -) - -func newTestVMsPoolVMList(count int) []compute.VirtualMachine { - var vmList []compute.VirtualMachine - for i := 0; i < count; i++ { - vm := compute.VirtualMachine{ - ID: to.StringPtr(fmt.Sprintf(fakeVMsPoolVMID, i)), - VirtualMachineProperties: &compute.VirtualMachineProperties{ - VMID: to.StringPtr(fmt.Sprintf("123E4567-E89B-12D3-A456-426655440000-%d", i)), - }, - Tags: map[string]*string{ - agentpoolTypeTag: to.StringPtr("VirtualMachines"), - agentpoolNameTag: to.StringPtr("test-vms-pool"), - }, - } - vmList = append(vmList, vm) - } - return vmList -} - -func newVMsNode(vmID int64) *apiv1.Node { - node := &apiv1.Node{ - Spec: apiv1.NodeSpec{ - ProviderID: "azure://" + fmt.Sprintf(fakeVMsPoolVMID, vmID), - }, - } - return node -} diff --git a/cluster-autoscaler/cloudprovider/azure/examples/cluster-autoscaler-autodiscover.yaml b/cluster-autoscaler/cloudprovider/azure/examples/cluster-autoscaler-autodiscover.yaml index fee3dc20a305..d9652ad98043 100644 --- a/cluster-autoscaler/cloudprovider/azure/examples/cluster-autoscaler-autodiscover.yaml +++ b/cluster-autoscaler/cloudprovider/azure/examples/cluster-autoscaler-autodiscover.yaml @@ -154,9 +154,9 @@ spec: - effect: NoSchedule operator: "Equal" value: "true" - key: node-role.kubernetes.io/master + key: node-role.kubernetes.io/control-plane nodeSelector: - kubernetes.io/role: master + kubernetes.io/role: control-plane containers: - command: - ./cluster-autoscaler diff --git a/cluster-autoscaler/cloudprovider/azure/examples/cluster-autoscaler-standard-control-plane.yaml b/cluster-autoscaler/cloudprovider/azure/examples/cluster-autoscaler-standard-control-plane.yaml index b3d21d8d92a5..1a2bb5666082 100644 --- a/cluster-autoscaler/cloudprovider/azure/examples/cluster-autoscaler-standard-control-plane.yaml +++ b/cluster-autoscaler/cloudprovider/azure/examples/cluster-autoscaler-standard-control-plane.yaml @@ -156,9 +156,9 @@ spec: - effect: NoSchedule operator: "Equal" value: "true" - key: node-role.kubernetes.io/master + key: node-role.kubernetes.io/control-plane nodeSelector: - kubernetes.io/role: master + kubernetes.io/role: control-plane containers: - command: - ./cluster-autoscaler diff --git a/cluster-autoscaler/cloudprovider/azure/examples/cluster-autoscaler-standard-msi.yaml b/cluster-autoscaler/cloudprovider/azure/examples/cluster-autoscaler-standard-msi.yaml index 83d2bcccb7ac..3a3175972507 100644 --- a/cluster-autoscaler/cloudprovider/azure/examples/cluster-autoscaler-standard-msi.yaml +++ b/cluster-autoscaler/cloudprovider/azure/examples/cluster-autoscaler-standard-msi.yaml @@ -154,9 +154,9 @@ spec: - effect: NoSchedule operator: "Equal" value: "true" - key: node-role.kubernetes.io/master + key: node-role.kubernetes.io/control-plane nodeSelector: - kubernetes.io/role: master + kubernetes.io/role: control-plane containers: - command: - ./cluster-autoscaler diff --git a/cluster-autoscaler/cloudprovider/azure/examples/cluster-autoscaler-vmss-control-plane.yaml b/cluster-autoscaler/cloudprovider/azure/examples/cluster-autoscaler-vmss-control-plane.yaml index 9161887cb4db..1a1756b5b07e 100644 --- a/cluster-autoscaler/cloudprovider/azure/examples/cluster-autoscaler-vmss-control-plane.yaml +++ b/cluster-autoscaler/cloudprovider/azure/examples/cluster-autoscaler-vmss-control-plane.yaml @@ -155,9 +155,9 @@ spec: - effect: NoSchedule operator: "Equal" value: "true" - key: node-role.kubernetes.io/master + key: node-role.kubernetes.io/control-plane nodeSelector: - kubernetes.io/role: master + kubernetes.io/role: control-plane containers: - image: registry.k8s.io/autoscaling/cluster-autoscaler:{{ ca_version }} imagePullPolicy: Always diff --git a/cluster-autoscaler/cloudprovider/azure/examples/cluster-autoscaler-vmss-msi.yaml b/cluster-autoscaler/cloudprovider/azure/examples/cluster-autoscaler-vmss-msi.yaml index b28a8e3c4db0..aa113b32fdfd 100644 --- a/cluster-autoscaler/cloudprovider/azure/examples/cluster-autoscaler-vmss-msi.yaml +++ b/cluster-autoscaler/cloudprovider/azure/examples/cluster-autoscaler-vmss-msi.yaml @@ -153,9 +153,9 @@ spec: - effect: NoSchedule operator: "Equal" value: "true" - key: node-role.kubernetes.io/master + key: node-role.kubernetes.io/control-plane nodeSelector: - kubernetes.io/role: master + kubernetes.io/role: control-plane containers: - image: registry.k8s.io/autoscaling/cluster-autoscaler:{{ ca_version }} imagePullPolicy: Always diff --git a/cluster-autoscaler/cloudprovider/baiducloud/baiducloud_cloud_provider.go b/cluster-autoscaler/cloudprovider/baiducloud/baiducloud_cloud_provider.go index 11a81f322020..a38d2b86d46a 100644 --- a/cluster-autoscaler/cloudprovider/baiducloud/baiducloud_cloud_provider.go +++ b/cluster-autoscaler/cloudprovider/baiducloud/baiducloud_cloud_provider.go @@ -278,6 +278,11 @@ func (asg *Asg) IncreaseSize(delta int) error { return asg.baiducloudManager.ScaleUpCluster(delta, asg.Name) } +// AtomicIncreaseSize is not implemented. +func (asg *Asg) AtomicIncreaseSize(delta int) error { + return cloudprovider.ErrNotImplemented +} + // DeleteNodes deletes nodes from this node group. Error is returned either on // failure or if the given node doesn't belong to this node group. This function // should wait until node group size is updated. Implementation required. diff --git a/cluster-autoscaler/cloudprovider/bizflycloud/bizflycloud_node_group.go b/cluster-autoscaler/cloudprovider/bizflycloud/bizflycloud_node_group.go index 363d634e9e81..7ca070fffbc0 100644 --- a/cluster-autoscaler/cloudprovider/bizflycloud/bizflycloud_node_group.go +++ b/cluster-autoscaler/cloudprovider/bizflycloud/bizflycloud_node_group.go @@ -100,6 +100,11 @@ func (n *NodeGroup) IncreaseSize(delta int) error { return nil } +// AtomicIncreaseSize is not implemented. +func (n *NodeGroup) AtomicIncreaseSize(delta int) error { + return cloudprovider.ErrNotImplemented +} + // DeleteNodes deletes nodes from this node group (and also increasing the size // of the node group with that). Error is returned either on failure or if the // given node doesn't belong to this node group. This function should wait diff --git a/cluster-autoscaler/cloudprovider/brightbox/brightbox_node_group.go b/cluster-autoscaler/cloudprovider/brightbox/brightbox_node_group.go index e790fbe928a6..4b97f8779320 100644 --- a/cluster-autoscaler/cloudprovider/brightbox/brightbox_node_group.go +++ b/cluster-autoscaler/cloudprovider/brightbox/brightbox_node_group.go @@ -123,6 +123,11 @@ func (ng *brightboxNodeGroup) IncreaseSize(delta int) error { ) } +// AtomicIncreaseSize is not implemented. +func (ng *brightboxNodeGroup) AtomicIncreaseSize(delta int) error { + return cloudprovider.ErrNotImplemented +} + // DeleteNodes deletes nodes from this node group. Error is returned // either on failure or if the given node doesn't belong to this // node group. This function should wait until node group size is diff --git a/cluster-autoscaler/cloudprovider/builder/builder_all.go b/cluster-autoscaler/cloudprovider/builder/builder_all.go index 2a5549ff65d3..3e74c815ae3d 100644 --- a/cluster-autoscaler/cloudprovider/builder/builder_all.go +++ b/cluster-autoscaler/cloudprovider/builder/builder_all.go @@ -1,5 +1,5 @@ -//go:build !gce && !aws && !azure && !kubemark && !alicloud && !magnum && !digitalocean && !clusterapi && !huaweicloud && !ionoscloud && !linode && !hetzner && !bizflycloud && !brightbox && !packet && !oci && !vultr && !tencentcloud && !scaleway && !externalgrpc && !civo && !rancher && !volcengine && !baiducloud && !cherry && !cloudstack && !exoscale && !kamatera && !ovhcloud && !kwok -// +build !gce,!aws,!azure,!kubemark,!alicloud,!magnum,!digitalocean,!clusterapi,!huaweicloud,!ionoscloud,!linode,!hetzner,!bizflycloud,!brightbox,!packet,!oci,!vultr,!tencentcloud,!scaleway,!externalgrpc,!civo,!rancher,!volcengine,!baiducloud,!cherry,!cloudstack,!exoscale,!kamatera,!ovhcloud,!kwok +//go:build !gce && !aws && !azure && !kubemark && !alicloud && !magnum && !digitalocean && !clusterapi && !huaweicloud && !ionoscloud && !linode && !hetzner && !bizflycloud && !brightbox && !equinixmetal && !oci && !vultr && !tencentcloud && !scaleway && !externalgrpc && !civo && !rancher && !volcengine && !baiducloud && !cherry && !cloudstack && !exoscale && !kamatera && !ovhcloud +// +build !gce,!aws,!azure,!kubemark,!alicloud,!magnum,!digitalocean,!clusterapi,!huaweicloud,!ionoscloud,!linode,!hetzner,!bizflycloud,!brightbox,!equinixmetal,!oci,!vultr,!tencentcloud,!scaleway,!externalgrpc,!civo,!rancher,!volcengine,!baiducloud,!cherry,!cloudstack,!exoscale,!kamatera,!ovhcloud /* Copyright 2018 The Kubernetes Authors. @@ -32,6 +32,7 @@ import ( "k8s.io/autoscaler/cluster-autoscaler/cloudprovider/cloudstack" "k8s.io/autoscaler/cluster-autoscaler/cloudprovider/clusterapi" "k8s.io/autoscaler/cluster-autoscaler/cloudprovider/digitalocean" + "k8s.io/autoscaler/cluster-autoscaler/cloudprovider/equinixmetal" "k8s.io/autoscaler/cluster-autoscaler/cloudprovider/exoscale" "k8s.io/autoscaler/cluster-autoscaler/cloudprovider/externalgrpc" "k8s.io/autoscaler/cluster-autoscaler/cloudprovider/gce" @@ -45,7 +46,6 @@ import ( "k8s.io/autoscaler/cluster-autoscaler/cloudprovider/mcm" oci "k8s.io/autoscaler/cluster-autoscaler/cloudprovider/oci/instancepools" "k8s.io/autoscaler/cluster-autoscaler/cloudprovider/ovhcloud" - "k8s.io/autoscaler/cluster-autoscaler/cloudprovider/packet" "k8s.io/autoscaler/cluster-autoscaler/cloudprovider/rancher" "k8s.io/autoscaler/cluster-autoscaler/cloudprovider/scaleway" "k8s.io/autoscaler/cluster-autoscaler/cloudprovider/tencentcloud" @@ -79,7 +79,7 @@ var AvailableCloudProviders = []string{ cloudprovider.LinodeProviderName, cloudprovider.BizflyCloudProviderName, cloudprovider.BrightboxProviderName, - cloudprovider.PacketProviderName, + cloudprovider.EquinixMetalProviderName, cloudprovider.VultrProviderName, cloudprovider.TencentcloudProviderName, cloudprovider.CivoProviderName, @@ -129,7 +129,7 @@ func buildCloudProvider(opts config.AutoscalingOptions, case cloudprovider.HetznerProviderName: return hetzner.BuildHetzner(opts, do, rl) case cloudprovider.PacketProviderName, cloudprovider.EquinixMetalProviderName: - return packet.BuildCloudProvider(opts, do, rl) + return equinixmetal.BuildCloudProvider(opts, do, rl) case cloudprovider.ClusterAPIProviderName: return clusterapi.BuildClusterAPI(opts, do, rl) case mcm.ProviderName: diff --git a/cluster-autoscaler/cloudprovider/builder/builder_kwok.go b/cluster-autoscaler/cloudprovider/builder/builder_kwok.go index b79f7973b18d..a254ac4246bc 100644 --- a/cluster-autoscaler/cloudprovider/builder/builder_kwok.go +++ b/cluster-autoscaler/cloudprovider/builder/builder_kwok.go @@ -23,6 +23,8 @@ import ( "k8s.io/autoscaler/cluster-autoscaler/cloudprovider" "k8s.io/autoscaler/cluster-autoscaler/cloudprovider/kwok" "k8s.io/autoscaler/cluster-autoscaler/config" + + "k8s.io/client-go/informers" ) // AvailableCloudProviders supported by the cloud provider builder. @@ -33,10 +35,10 @@ var AvailableCloudProviders = []string{ // DefaultCloudProvider for Kwok-only build is Kwok. const DefaultCloudProvider = cloudprovider.KwokProviderName -func buildCloudProvider(opts config.AutoscalingOptions, do cloudprovider.NodeGroupDiscoveryOptions, rl *cloudprovider.ResourceLimiter) cloudprovider.CloudProvider { +func buildCloudProvider(opts config.AutoscalingOptions, do cloudprovider.NodeGroupDiscoveryOptions, rl *cloudprovider.ResourceLimiter, informerFactory informers.SharedInformerFactory) cloudprovider.CloudProvider { switch opts.CloudProviderName { case cloudprovider.KwokProviderName: - return kwok.BuildKwokCloudProvider(opts, do, rl)(opts, do, rl) + return kwok.BuildKwok(opts, do, rl, informerFactory) } return nil diff --git a/cluster-autoscaler/cloudprovider/builder/builder_packet.go b/cluster-autoscaler/cloudprovider/builder/builder_packet.go index 4a13533fc713..5fc36299e8c5 100644 --- a/cluster-autoscaler/cloudprovider/builder/builder_packet.go +++ b/cluster-autoscaler/cloudprovider/builder/builder_packet.go @@ -1,5 +1,5 @@ -//go:build packet -// +build packet +//go:build equinixmetal +// +build equinixmetal /* Copyright 2019 The Kubernetes Authors. @@ -21,14 +21,14 @@ package builder import ( "k8s.io/autoscaler/cluster-autoscaler/cloudprovider" - "k8s.io/autoscaler/cluster-autoscaler/cloudprovider/packet" + "k8s.io/autoscaler/cluster-autoscaler/cloudprovider/equinixmetal" "k8s.io/autoscaler/cluster-autoscaler/config" "k8s.io/client-go/informers" ) // AvailableCloudProviders supported by the cloud provider builder. var AvailableCloudProviders = []string{ - packet.ProviderName, + cloudprovider.PacketProviderName, cloudprovider.EquinixMetalProviderName, } @@ -37,8 +37,8 @@ const DefaultCloudProvider = cloudprovider.EquinixMetalProviderName func buildCloudProvider(opts config.AutoscalingOptions, do cloudprovider.NodeGroupDiscoveryOptions, rl *cloudprovider.ResourceLimiter, _ informers.SharedInformerFactory) cloudprovider.CloudProvider { switch opts.CloudProviderName { - case packet.ProviderName, cloudprovider.EquinixMetalProviderName: - return packet.BuildCloudProvider(opts, do, rl) + case cloudprovider.PacketProviderName, cloudprovider.EquinixMetalProviderName: + return equinixmetal.BuildCloudProvider(opts, do, rl) } return nil diff --git a/cluster-autoscaler/cloudprovider/cherryservers/cherry_node_group.go b/cluster-autoscaler/cloudprovider/cherryservers/cherry_node_group.go index 744bd52b66d3..4add7c862d7e 100644 --- a/cluster-autoscaler/cloudprovider/cherryservers/cherry_node_group.go +++ b/cluster-autoscaler/cloudprovider/cherryservers/cherry_node_group.go @@ -113,6 +113,11 @@ func (ng *cherryNodeGroup) IncreaseSize(delta int) error { return nil } +// AtomicIncreaseSize is not implemented. +func (ng *cherryNodeGroup) AtomicIncreaseSize(delta int) error { + return cloudprovider.ErrNotImplemented +} + // DeleteNodes deletes a set of nodes chosen by the autoscaler. func (ng *cherryNodeGroup) DeleteNodes(nodes []*apiv1.Node) error { // Batch simultaneous deletes on individual nodes diff --git a/cluster-autoscaler/cloudprovider/cherryservers/examples/cluster-autoscaler-deployment.yaml b/cluster-autoscaler/cloudprovider/cherryservers/examples/cluster-autoscaler-deployment.yaml index 183575b49c19..1cf37bc3e3f4 100644 --- a/cluster-autoscaler/cloudprovider/cherryservers/examples/cluster-autoscaler-deployment.yaml +++ b/cluster-autoscaler/cloudprovider/cherryservers/examples/cluster-autoscaler-deployment.yaml @@ -18,16 +18,16 @@ spec: spec: tolerations: - effect: NoSchedule - key: node-role.kubernetes.io/master + key: node-role.kubernetes.io/control-plane # Node affinity is used to force cluster-autoscaler to stick - # to the master node. This allows the cluster to reliably downscale + # to the macontrol-plane node. This allows the cluster to reliably downscale # to zero worker nodes when needed. affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - - key: node-role.kubernetes.io/master + - key: node-role.kubernetes.io/control-plane operator: Exists serviceAccountName: cluster-autoscaler containers: diff --git a/cluster-autoscaler/cloudprovider/civo/civo_node_group.go b/cluster-autoscaler/cloudprovider/civo/civo_node_group.go index d47abdbfbb4a..d20a8c4d5ebe 100644 --- a/cluster-autoscaler/cloudprovider/civo/civo_node_group.go +++ b/cluster-autoscaler/cloudprovider/civo/civo_node_group.go @@ -118,6 +118,11 @@ func (n *NodeGroup) IncreaseSize(delta int) error { return nil } +// AtomicIncreaseSize is not implemented. +func (n *NodeGroup) AtomicIncreaseSize(delta int) error { + return cloudprovider.ErrNotImplemented +} + // DeleteNodes deletes nodes from this node group (and also increasing the size // of the node group with that). Error is returned either on failure or if the // given node doesn't belong to this node group. This function should wait diff --git a/cluster-autoscaler/cloudprovider/cloud_provider.go b/cluster-autoscaler/cloudprovider/cloud_provider.go index e0bdc45b68fe..aa49b2e667c3 100644 --- a/cluster-autoscaler/cloudprovider/cloud_provider.go +++ b/cluster-autoscaler/cloudprovider/cloud_provider.go @@ -184,6 +184,17 @@ type NodeGroup interface { // node group size is updated. Implementation required. IncreaseSize(delta int) error + // AtomicIncreaseSize tries to increase the size of the node group atomically. + // - If the method returns nil, it guarantees that delta instances will be added to the node group + // within its MaxNodeProvisionTime. The function should wait until node group size is updated. + // The cloud provider is responsible for tracking and ensuring successful scale up asynchronously. + // - If the method returns an error, it guarantees that no new instances will be added to the node group + // as a result of this call. The cloud provider is responsible for ensuring that before returning from the method. + // Implementation is optional. If implemented, CA will take advantage of the method while scaling up + // GenericScaleUp ProvisioningClass, guaranteeing that all instances required for such a ProvisioningRequest + // are provisioned atomically. + AtomicIncreaseSize(delta int) error + // DeleteNodes deletes nodes from this node group. Error is returned either on // failure or if the given node doesn't belong to this node group. This function // should wait until node group size is updated. Implementation required. @@ -234,7 +245,7 @@ type NodeGroup interface { // GetOptions returns NodeGroupAutoscalingOptions that should be used for this particular // NodeGroup. Returning a nil will result in using default options. - // Implementation optional. + // Implementation optional. Callers MUST handle `cloudprovider.ErrNotImplemented`. GetOptions(defaults config.NodeGroupAutoscalingOptions) (*config.NodeGroupAutoscalingOptions, error) } diff --git a/cluster-autoscaler/cloudprovider/cloudstack/cloudstack_node_group.go b/cluster-autoscaler/cloudprovider/cloudstack/cloudstack_node_group.go index c4fdf5b805fb..9181b5cd82b9 100644 --- a/cluster-autoscaler/cloudprovider/cloudstack/cloudstack_node_group.go +++ b/cluster-autoscaler/cloudprovider/cloudstack/cloudstack_node_group.go @@ -71,6 +71,11 @@ func (asg *asg) IncreaseSize(delta int) error { return nil } +// AtomicIncreaseSize is not implemented. +func (asg *asg) AtomicIncreaseSize(delta int) error { + return cloudprovider.ErrNotImplemented +} + // DecreaseTargetSize decreases the target size of the node group. This function // doesn't permit to delete any existing node and can be used only to reduce the // request for new nodes that have not been yet fulfilled. Delta should be negative. diff --git a/cluster-autoscaler/cloudprovider/clusterapi/README.md b/cluster-autoscaler/cloudprovider/clusterapi/README.md index 2f4e247f4579..d6a73da14949 100644 --- a/cluster-autoscaler/cloudprovider/clusterapi/README.md +++ b/cluster-autoscaler/cloudprovider/clusterapi/README.md @@ -50,9 +50,7 @@ most likely need other command line flags. For more information you can invoke ## Configuring node group auto discovery -If you do not configure node group auto discovery, cluster autoscaler will attempt -to match nodes against any scalable resources found in any namespace and belonging -to any Cluster. +You must configure node group auto discovery to inform cluster autoscaler which cluster in which to find for scalable node groups. Limiting cluster autoscaler to only match against resources in the blue namespace @@ -323,13 +321,14 @@ set, the autoscaler will use the behavior described above. ## Sample manifest -A sample manifest that will create a deployment running the autoscaler is +A [sample manifest](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/clusterapi/examples/deployment.yaml) +that will create a deployment running the autoscaler is available. It can be deployed by passing it through `envsubst`, providing these environment variables to set the namespace to deploy into as well as the image and tag to use: ``` export AUTOSCALER_NS=kube-system -export AUTOSCALER_IMAGE=us.gcr.io/k8s-artifacts-prod/autoscaling/cluster-autoscaler:v1.20.0 +export AUTOSCALER_IMAGE=registry.k8s.io/autoscaling/cluster-autoscaler:v1.29.0 envsubst < examples/deployment.yaml | kubectl apply -f- ``` @@ -454,4 +453,3 @@ here as a reference for users who might be deploying on these infrastructures. | IBM Cloud | `ibm-cloud.kubernetes.io/worker-id` | Used by the IBM Cloud Cloud Controller Manager to identify the node | | IBM Cloud | `vpc-block-csi-driver-labels` | Used by the IBM Cloud CSI driver as a target for persistent volume node affinity | | IBM Cloud | `ibm-cloud.kubernetes.io/vpc-instance-id` | Used when a VPC is in use on IBM Cloud | - diff --git a/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_controller.go b/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_controller.go index 567bba5e0d1c..3869d1ac2baf 100644 --- a/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_controller.go +++ b/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_controller.go @@ -474,7 +474,9 @@ func newMachineController( Resource: resourceNameMachinePool, } machinePoolInformer = managementInformerFactory.ForResource(gvrMachinePool) - machinePoolInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{}) + if _, err := machinePoolInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{}); err != nil { + return nil, fmt.Errorf("failed to add event handler for resource %q: %w", resourceNameMachinePool, err) + } if err := machinePoolInformer.Informer().GetIndexer().AddIndexers(cache.Indexers{ machinePoolProviderIDIndex: indexMachinePoolByProviderID, diff --git a/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_controller_test.go b/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_controller_test.go index 359111be6bb1..fc89fdae577a 100644 --- a/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_controller_test.go +++ b/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_controller_test.go @@ -1043,9 +1043,8 @@ func TestControllerNodeGroupForNodeWithPositiveScalingBounds(t *testing.T) { if err != nil { t.Fatalf("unexpected error: %v", err) } - // We don't scale from 0 so nodes must belong to a - // nodegroup that has a scale size of at least 1. - if ng != nil { + // We allow scaling if minSize=maxSize + if ng == nil { t.Fatalf("unexpected nodegroup: %v", ng) } } @@ -1124,19 +1123,19 @@ func TestControllerNodeGroups(t *testing.T) { nodeGroupMaxSizeAnnotationKey: "1", } - // Test #5: machineset with no scaling bounds results in no nodegroups + // Test #5: 5 machineset with minSize=maxSize results in a five nodegroup machineSetConfigs = createMachineSetTestConfigs(namespace, clusterName, RandomString(6), 5, 1, annotations, nil) if err := addTestConfigs(t, controller, machineSetConfigs...); err != nil { t.Fatalf("unexpected error: %v", err) } - assertNodegroupLen(t, controller, 0) + assertNodegroupLen(t, controller, 5) - // Test #6: machinedeployment with no scaling bounds results in no nodegroups + // Test #6: add 2 machinedeployment with minSize=maxSize machineDeploymentConfigs = createMachineDeploymentTestConfigs(namespace, clusterName, RandomString(6), 2, 1, annotations, nil) if err := addTestConfigs(t, controller, machineDeploymentConfigs...); err != nil { t.Fatalf("unexpected error: %v", err) } - assertNodegroupLen(t, controller, 0) + assertNodegroupLen(t, controller, 7) annotations = map[string]string{ nodeGroupMinSizeAnnotationKey: "-1", diff --git a/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_nodegroup.go b/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_nodegroup.go index d28dc0a14f28..eb66ff8ee8ed 100644 --- a/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_nodegroup.go +++ b/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_nodegroup.go @@ -18,6 +18,7 @@ package clusterapi import ( "fmt" + "k8s.io/klog/v2" "math/rand" "github.com/pkg/errors" @@ -88,6 +89,11 @@ func (ng *nodegroup) IncreaseSize(delta int) error { return ng.scalableResource.SetSize(size + delta) } +// AtomicIncreaseSize is not implemented. +func (ng *nodegroup) AtomicIncreaseSize(delta int) error { + return cloudprovider.ErrNotImplemented +} + // DeleteNodes deletes nodes from this node group. Error is returned // either on failure or if the given node doesn't belong to this node // group. This function should wait until node group size is updated. @@ -355,7 +361,12 @@ func newNodeGroupFromScalableResource(controller *machineController, unstructure } // Ensure the node group would have the capacity to scale - if scalableResource.MaxSize()-scalableResource.MinSize() < 1 { + // allow MinSize = 0 + // allow MaxSize = MinSize + // don't allow MaxSize < MinSize + // don't allow MaxSize = MinSize = 0 + if scalableResource.MaxSize()-scalableResource.MinSize() < 0 || scalableResource.MaxSize() == 0 { + klog.V(4).Infof("nodegroup %s has no scaling capacity, skipping", scalableResource.Name()) return nil, nil } diff --git a/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_nodegroup_test.go b/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_nodegroup_test.go index 6da06bde3614..3012cdd6f1be 100644 --- a/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_nodegroup_test.go +++ b/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_nodegroup_test.go @@ -108,6 +108,18 @@ func TestNodeGroupNewNodeGroupConstructor(t *testing.T) { nodeCount: 5, errors: false, expectNil: true, + }, { + description: "no error and expect notNil: min=max=2", + annotations: map[string]string{ + nodeGroupMinSizeAnnotationKey: "2", + nodeGroupMaxSizeAnnotationKey: "2", + }, + nodeCount: 1, + minSize: 2, + maxSize: 2, + replicas: 1, + errors: false, + expectNil: false, }} newNodeGroup := func(controller *machineController, testConfig *testConfig) (*nodegroup, error) { diff --git a/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_utils.go b/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_utils.go index 58ae58915192..68c2be164436 100644 --- a/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_utils.go +++ b/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_utils.go @@ -201,6 +201,9 @@ func machineSetHasMachineDeploymentOwnerRef(machineSet *unstructured.Unstructure // normalizedProviderString splits s on '/' returning everything after // the last '/'. func normalizedProviderString(s string) normalizedProviderID { + if strings.HasPrefix(s, "azure://") && strings.Contains(s, "virtualMachineScaleSets") { + return normalizedProviderID(s) + } split := strings.Split(s, "/") return normalizedProviderID(split[len(split)-1]) } diff --git a/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_utils_test.go b/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_utils_test.go index c6bafd50a969..f56c767de209 100644 --- a/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_utils_test.go +++ b/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_utils_test.go @@ -422,6 +422,14 @@ func TestUtilNormalizedProviderID(t *testing.T) { description: "id with / characters", providerID: "aws:////i-12345678", expectedID: "i-12345678", + }, { + description: "azure standard vm", + providerID: "azure:///subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/resourceGroupName/providers/Microsoft.Compute/virtualMachines/control-plane-1cbe5-d4dx7", + expectedID: "control-plane-1cbe5-d4dx7", + }, { + description: "azure vmss", + providerID: "azure:///subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/resourceGroupName/providers/Microsoft.Compute/virtualMachineScaleSets/vmssName/virtualMachines/0", + expectedID: "azure:///subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/resourceGroupName/providers/Microsoft.Compute/virtualMachineScaleSets/vmssName/virtualMachines/0", }} { t.Run(tc.description, func(t *testing.T) { actualID := normalizedProviderString(tc.providerID) diff --git a/cluster-autoscaler/cloudprovider/clusterapi/examples/deployment.yaml b/cluster-autoscaler/cloudprovider/clusterapi/examples/deployment.yaml index 4046cea7f48b..bfee32fc1423 100644 --- a/cluster-autoscaler/cloudprovider/clusterapi/examples/deployment.yaml +++ b/cluster-autoscaler/cloudprovider/clusterapi/examples/deployment.yaml @@ -27,7 +27,7 @@ spec: terminationGracePeriodSeconds: 10 tolerations: - effect: NoSchedule - key: node-role.kubernetes.io/master + key: node-role.kubernetes.io/control-plane --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 @@ -165,6 +165,7 @@ rules: - machinedeployments/scale - machines - machinesets + - machinepools verbs: - get - list diff --git a/cluster-autoscaler/cloudprovider/digitalocean/digitalocean_node_group.go b/cluster-autoscaler/cloudprovider/digitalocean/digitalocean_node_group.go index 1b82b5e87628..421cdea1451c 100644 --- a/cluster-autoscaler/cloudprovider/digitalocean/digitalocean_node_group.go +++ b/cluster-autoscaler/cloudprovider/digitalocean/digitalocean_node_group.go @@ -106,6 +106,11 @@ func (n *NodeGroup) IncreaseSize(delta int) error { return nil } +// AtomicIncreaseSize is not implemented. +func (n *NodeGroup) AtomicIncreaseSize(delta int) error { + return cloudprovider.ErrNotImplemented +} + // DeleteNodes deletes nodes from this node group (and also increasing the size // of the node group with that). Error is returned either on failure or if the // given node doesn't belong to this node group. This function should wait diff --git a/cluster-autoscaler/cloudprovider/equinixmetal/OWNERS b/cluster-autoscaler/cloudprovider/equinixmetal/OWNERS new file mode 100644 index 000000000000..9fcd38cabf36 --- /dev/null +++ b/cluster-autoscaler/cloudprovider/equinixmetal/OWNERS @@ -0,0 +1,16 @@ +approvers: +- cprivitere +- d-mo +- deitch +- detiber +- displague +reviewers: +- cprivitere +- d-mo +- deitch +- detiber +- displague +- v-pap + +labels: +- area/provider/equinixmetal diff --git a/cluster-autoscaler/cloudprovider/equinixmetal/README.md b/cluster-autoscaler/cloudprovider/equinixmetal/README.md new file mode 100644 index 000000000000..a0b751d02c96 --- /dev/null +++ b/cluster-autoscaler/cloudprovider/equinixmetal/README.md @@ -0,0 +1,133 @@ + +# Cluster Autoscaler for Equinix Metal + +The cluster autoscaler for [Equinix Metal](https://metal.equinix.com) worker nodes performs +autoscaling within any specified nodepools. It will run as a `Deployment` in +your cluster. The nodepools are specified using tags on Equinix Metal. + +Note: Packet was acquired by Equinix in 2020 and renamed to Equinix Metal. + +This README will go over some of the necessary steps required to get +the cluster autoscaler up and running. + +## Permissions and credentials + +The autoscaler needs a `ServiceAccount` with permissions for Kubernetes and +requires credentials for interacting with Equinix Metal. + +An example `ServiceAccount` is given in [examples/cluster-autoscaler-svcaccount.yaml](examples/cluster-autoscaler-svcaccount.yaml). + +The credentials for authenticating with Equinix Metal are stored in a secret and +provided as an env var to the container. [examples/cluster-autoscaler-secret](examples/cluster-autoscaler-secret.yaml) +In the above file you can modify the following fields: + +| Secret | Key | Value | +|-------------------------------------|-------------------------|------------------------------------------------------------------------------------------------------------------------------------| +| cluster-autoscaler-equinixmetal | authtoken | Your Equinix Metal API token. It must be base64 encoded. | +| cluster-autoscaler-cloud-config | Global/project-id | Your Equinix Metal project id | +| cluster-autoscaler-cloud-config | Global/api-server | The ip:port for you cluster's k8s api (e.g. K8S_MASTER_PUBLIC_IP:6443) | +| cluster-autoscaler-cloud-config | Global/facility | The Equinix Metal facility for the devices in your nodepool (eg: sv15) | +| cluster-autoscaler-cloud-config | Global/plan | The Equinix Metal plan (aka size/flavor) for new nodes in the nodepool (eg: c3.small.x86) | +| cluster-autoscaler-cloud-config | Global/billing | The billing interval for new nodes (default: hourly) | +| cluster-autoscaler-cloud-config | Global/os | The OS image to use for new nodes (default: ubuntu_18_04). If you change this also update cloudinit. | +| cluster-autoscaler-cloud-config | Global/cloudinit | The base64 encoded [user data](https://metal.equinix.com/developers/docs/servers/user-data/) submitted when provisioning devices. In the example file, the default value has been tested with Ubuntu 18.04 to install Docker & kubelet and then to bootstrap the node into the cluster using kubeadm. The kubeadm, kubelet, kubectl are pinned to version 1.17.4. For a different base OS or bootstrap method, this needs to be customized accordingly| +| cluster-autoscaler-cloud-config | Global/reservation | The values "require" or "prefer" will request the next available hardware reservation for new devices in selected facility & plan. If no hardware reservations match, "require" will trigger a failure, while "prefer" will launch on-demand devices instead (default: none) | +| cluster-autoscaler-cloud-config | Global/hostname-pattern | The pattern for the names of new Equinix Metal devices (default: "k8s-{{.ClusterName}}-{{.NodeGroup}}-{{.RandString8}}" ) | + +You can always update the secret with more nodepool definitions (with different plans etc.) as shown in the example, but you should always provide a default nodepool configuration. + +## Configure nodepool and cluster names using Equinix Metal tags + +The Equinix Metal API does not yet have native support for groups or pools of devices. So we use tags to specify them. Each Equinix Metal device that's a member of the "cluster1" cluster should have the tag k8s-cluster-cluster1. The devices that are members of the "pool1" nodepool should also have the tag k8s-nodepool-pool1. Once you have a Kubernetes cluster running on Equinix Metal, use the Equinix Metal Portal or API to tag the nodes accordingly. + +## Autoscaler deployment + +The deployment in `examples/cluster-autoscaler-deployment.yaml` can be used, +but the arguments passed to the autoscaler will need to be changed +to match your cluster. + +| Argument | Usage | +|-----------------------|------------------------------------------------------------------------------------------------------------| +| --cluster-name | The name of your Kubernetes cluster. It should correspond to the tags that have been applied to the nodes. | +| --nodes | Of the form `min:max:NodepoolName`. For multiple nodepools you can add the same argument multiple times. E.g. for pool1, pool2 you would add `--nodes=0:10:pool1` and `--nodes=0:10:pool2`. In addition, each node provisioned by the autoscaler will have a label with key: `pool` and with value: `NodepoolName`. These labels can be useful when there is a need to target specific nodepools. | +| --expander=price | This is an optional argument which allows the cluster-autoscaler to take into account the pricing of the Equinix Metal nodes when scaling with multiple nodepools. | + +## Target Specific Nodepools (New!) + +In case you want to target a specific nodepool(s) for e.g. a deployment, you can add a `nodeAffinity` with the key `pool` and with value the nodepool name that you want to target. This functionality is not backwards compatible, which means that nodes provisioned with older cluster-autoscaler images won't have the key `pool`. But you can overcome this limitation by manually adding the correct labels. Here are some examples: + +Target a nodepool with a specific name: +``` +affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: pool + operator: In + values: + - pool3 +``` +Target a nodepool with a specific Equinix Metal instance: +``` +affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: beta.kubernetes.io/instance-type + operator: In + values: + - c3.small.x86 +``` + +## CCM and Controller node labels + +### CCM +By default, autoscaler assumes that you have an older deprecated version of `packet-ccm` installed in your +cluster. If however, that is not the case and you've migrated to the new `cloud-provider-equinix-metal` CCM, +then this must be told to autoscaler. This can be done via setting an environment variable in the deployment: +``` +env: + - name: INSTALLED_CCM + value: cloud-provider-equinix-metal +``` +**NOTE**: As a prerequisite, ensure that all worker nodes in your cluster have the prefix `equinixmetal://` in +the Node spec `.spec.providerID`. If there are any existing worker nodes with prefix `packet://`, then drain +the node, remove the node and restart the kubelet on that worker node to re-register the node in the cluster, +this would ensure that `cloud-provider-equinix-metal` CCM sets the uuid with prefix `equinixmetal://` to the +field `.spec.ProviderID`. + +### Controller node labels + +Autoscaler assumes that control plane nodes in your cluster are identified by the label +`node-role.kubernetes.io/master`. If for some reason, this assumption is not true in your case, then set the +environment variable in the deployment: + +``` +env: + - name: METAL_CONTROLLER_NODE_IDENTIFIER_LABEL + value: