[local] NodeInfoPodTemplateProcessor implementation #47
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
clamoriniere
force-pushed
the
clamoriniere/PodTemplateProcessorInternal
branch
from
5424840 to
d97bbbc
Compare
bpineau
force-pushed
the
datadog-master-6.0
branch
from
9cf6596 to
5528a72
Compare
bpineau
force-pushed
the
datadog-master-6.0
branch
from
efa2cf1 to
d7d7ed1
Compare
Support per-VMSS (scaledown) settings as permited by the cloudprovider's interface `GetOptions()` method.
This is meant to isolate, contain and wrap access to our custom resource (storageclass/local-data) and label (/local-storage) in a single place, separated in our processors/datadog/ namespace. Avoid spreading local names and constraints over all the code base.
Main functional change compared to upstream's is the "storageclass/local-data" hack, meant to support pods claiming local-data (no-provisioner) volumes. filterOutSchedulable (+test) is mostly that of upstream (copied for later modifications), lightly modified to compose with older clusters where taint based eviction isn't enabled. The frontend is meant to hookin future sub-processors for pods. This gives a foothold for some local-only improvements, without touching upstream code; plans includes: * Runaway upscale prevention * Schedulable metrics (pending pods pressure, long pending pods) * Pods labels selectors (for dedicated autoscaler instances) * Possibly: unpriorize pods from cronjobs
Hooking it in the less intrusive way we could.
Goals of that PodListProcessor are twofold: * Lower presure on runonce loops by evaluating long pending pods less frequently * More importantly: free autoscaler cycles to recover from scaledown cooldown, so nodes created for pods causing infinite upscales get gc'ed rather than filling a cluster (and those pods are slowed down) The delay penalty could be made progressive (eg. 2m then 5m then 10m etc), but for now a static value makes evaluating benefits and impacts easier. Metrics to come in follow-up PR.
This is meant to replace a patch that was setting new nodes as NotReady until their lvp pod was there, with something bound to and contained in our podsListProcessor, not touching autoscaler core or cloudproviders at all anymore. Decision is entirely based of the local-data:true label: no guessing or per cloud provider instances types allowlists needed anymore. Instead of setting them NotReady, the new nodes that just joined are now considered as schedulable for pods requesting local-data once they are ready, which naturaly prevents spurious re-upscales. For now the change is restricted to new local-data nodes that just became ready for less than 5mn, as we're assessing wider impact. The downside is we need to modify the clusterSnapshot content before filterOutSchedulable runs scheduler predicates with those nodes, which happens later, also in our own podsListProcessor.
And stop trying to make that a nodeinfo processor: this was an attempt to follow upstream suggestions, but excessively intrusive (not rebase friendly) for a feature we might keep localy/forked for a long time. We're also submiting a "nodeinfos provider processor" to upstream, which (if accepted) will help integrate that kind of changes much more cleanly (eg. not breaking tests, not touching core/). For now, let's assume we might not have an upstream processor entry point for a long time.
Hooking it in the less intrusive way we could. We'll work with upstream for a proper "nodeinfos processor" entrypoint (and revisit this, depending on the outcome).
`FetchMigTemplate()` calls are costly, causing two cloud API calls (InstanceGroupManagersService.Get and InstanceTemplatesService.Get). Due to that, expiring all instance templates at once results in very noticeable latency spikes, and exposes to throttling, especially for clusters having many MIGs attached.
Support per-ASG (scaledown) settings as permited by the cloudprovider's interface GetOptions() method.
clamoriniere
force-pushed
the
clamoriniere/PodTemplateProcessorInternal
branch
from
d97bbbc to
4a00f15
Compare
clamoriniere
force-pushed
the
clamoriniere/PodTemplateProcessorInternal
branch
from
4a00f15 to
e7a47e3
Compare
|
I "rebased" the PR on top of the new |
bpineau
force-pushed
the
datadog-master-6.0
branch
2 times, most recently
from
618c5a3 to
b6de6ee
Compare
|
closed in favour of #50 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Based on kubernetes#3964, this PR only implement the mandatory elements to work in a Datadog environment.
I tried to build the commit to ease any futur rebase of https://github.com/kubernetes/autoscaler.