Merge pull request #4566 from kgolab/kep-min-replicas

Enhancement proposal to add minReplicas per VPA Object (see #4560)

vertical-pod-autoscaler/enhancements/4566-min-replicas/README.md

@@ -0,0 +1,109 @@
+# KEP-4566: MinReplicas per VPA object
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+   - [Goals](#goals)
+   - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+- [Design Details](#design-details)
+   - [Test Plan](#test-plan)
+- [Implementation History](#implementation-history)
+- [Alternatives](#alternatives)
+   - [Existing Behaviour](#existing-behaviour)
+   - [Reuse Cluster Autoscaler Annotations](#reuse-cluster-autoscaler-annotations)
+<!-- /toc -->
+
+## Summary
+
+The default behaviour of VPA Updater is to allow Pod eviction only if there are
+at least 2 live replicas, in order to avoid temporary total unavailability of a
+workload under VPA in Auto mode. This can be changed globally with
+`--min-replicas` flag. However, such a change might be deemed dangerous in a
+centrally-managed multi-tenant cluster.
+
+This proposal addresses the problem by allowing to specify per VPA object a
+minimum number of replicas required to start Pod eviction (thus active VPA
+actuation). This allows to keep the current relatively safe default and override
+it explicitly only where needed.
+
+## Motivation
+
+The motivation behind the change is due to VPA users needing the configuration
+option in managed environments to allow for eviction of a single replica, see
+[issue #3986](https://github.com/kubernetes/autoscaler/issues/3986) or
+[issue #1828](https://github.com/kubernetes/autoscaler/issues/1828).
+
+### Goals
+
+- Main: allow workload owner to use VPA in Auto/Recreate mode on a single-replica
+  workloads,
+- Secondary: allow workload owner to specify a higher number of replicas required
+  to be kept alive. Note that this can also be achieved with
+  [PDB](https://kubernetes.io/docs/tasks/run-application/configure-pdb/).
+
+### Non-Goals
+
+- Create any advanced form of policy around VPA-introduced disruption.
+
+## Proposal
+
+The proposal is to add `minReplicas` field under VPA `spec.updatePolicy` and
+alter VPA Updater behaviour so it respects the new field:
+- if the field is set, use its value instead of the global `--min-replicas`
+  flag, only for this particular VPA object,
+- if the field is not set, use the global `--min-replicas` flag, keeping
+  backward compatibility.
+
+The field is ignored by other VPA components (Recommender, Admission
+Controller).
+
+Since the change is backward-compatible the suggestion is to simply extend `v1`
+version of VPA API, avoiding the hassle of introducing a new API version.
+
+## Design Details
+
+Suggested implementation is present in [PR
+4560](https://github.com/kubernetes/autoscaler/pull/4560).
+
+### Test Plan
+
+Add automated E2E tests that update VPA objects, altering the value of
+`spec.updatePolicy.minReplicas` flag and verifying that the behaviour of VPA
+Updater changes accordingly.
+
+## Implementation History
+
+- 2021-12-28: initial version
+
+## Alternatives
+
+### Existing Behaviour
+
+The only existing alternative to achieve the behaviour allowed by the proposed
+change is to set global `--min-replicas` flag to `1` and use Pod Disruption
+Budget to protect single-replica workloads from being disrupted, if needed.
+
+While this allows for the cluster to end up with a very similar (if not identical)
+behaviour, it changes the default behaviour to an unsafe one.
+As a result this alternative requires a cluster-wide coordination - PDBs might
+not be configured for all workloads needing protection.
+This might be hard to achieve in multitenant environments.
+
+Also the flag might not be accessible in managed environments (for example GKE).
+
+### Reuse Cluster Autoscaler Annotations
+
+An alternative suggested in
+[issue #3986](https://github.com/kubernetes/autoscaler/issues/3986) was to reuse
+Cluster Autoscaler annotation `cluster-autoscaler.kubernetes.io/safe-to-evict`.
+
+While it's not clear if there are use cases when one would like to allow VPA or
+CA to evict Pods but disallow it for the other controller, coupling them this
+way seems dangerous.
+
+Of course VPA could introduce its own annotation to express the same but it
+seems much more elegant to keep the information in the API object (a solution
+not possible for CA due to lack of such an object) instead of effectively
+spreading the API between VPA objects and Pod annotations.
+