vertical-pod-autoscaler: updater suddenly evicted a lot of pods after being redeployed

[milesbxf]

Raising an issue separately to #2178 to track what we've found

We had an issue with the VPA recently where we redeployed the recommender and updater; the updater immediately started evicting pods from nearly half our cluster (2-3k pods) in about 15-20 minutes. We're discussing how to mitigate the effects of pod/endpoint churn in #2178; this issue is about why the mass evictions happened.

What we did in the build up to the issue

We have the vertical pod autoscaler components set up to autoscale themselves (:inception:). The recommender had recommended over 1GiB memory, but the admission controller was unable to override the pod requests, as we still had pod resource limits set (to 1GiB).

We removed the limits and redeployed both components. The admission controller successfully updated both pods with new requests.

Recommender:
CPU 63m -> 203m
Memory 1000MiB -> 1GiB

Updater:
CPU 49m -> 126m
Memory 878Mi -> 1GiB

What we expected to happen

No change in pod eviction rate when redeploying the recommender/updater

What actually happened

Lots of pods were evicted:

Other interesting info

The recommender execution latency dropped by half after the pod was redeployed with the new resource requests:

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.