Fix templated nodeinfo names collisions in BinpackingNodeEstimator #4089
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Both upscale's `getUpcomingNodeInfos` and the binpacking estimator now uses the same shared DeepCopyTemplateNode function and inherits its naming pattern, which is great as that fixes a long standing bug. Due to that, `getUpcomingNodeInfos` will enrich the cluster snapshots with generated nodeinfos and nodes having predictable names (using template name + an incremental ordinal starting at 0) for upcoming nodes. Later, when it looks for fitting nodes for unschedulable pods (when upcoming nodes don't satisfy those (FitsAnyNodeMatching failing due to nodes capacity, or pods antiaffinity, ...), the binpacking estimator will also build virtual nodes and place them in a snapshot fork to evaluate scheduler predicates. Those temporary virtual nodes are built using the same pattern (template name and an index ordinal also starting at 0) as the one previously used by `getUpcomingNodeInfos`, which means it will generate the same nodeinfos/nodes names for nodegroups having upcoming nodes. But adding nodes by the same name in an existing cluster snapshot isn't allowed, and the evaluation attempt will fail. Practically this blocks re-upscales for nodegroups having upcoming nodes, which can cause a significant delay.
k8s-ci-robot
added
cncf-cla: yes
|
Nice catch! Sorry for introducing that issue :( /lgtm |
k8s-ci-robot
added
the
lgtm
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: bpineau, MaciekPytel The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
the
approved
k8s-ci-robot
added a commit
that referenced
this pull request
Sep 7, 2021
Cluster Autoscaler: Backport #4089 to 1.21
evansheng
pushed a commit
to airbnb/autoscaler
that referenced
this pull request
Aug 17, 2022
…sions Fix templated nodeinfo names collisions in BinpackingNodeEstimator
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Since a couple of months, both upscale's
getUpcomingNodeInfosand thebinpacking estimator uses the same shared
DeepCopyTemplateNodefunctionand inherits its naming pattern, which is great and fixes a long standing bug.
getUpcomingNodeInfoswill now enrich the cluster snapshots withgenerated nodeinfos and nodes having predictable names (using the nodegroup's
template name, and an incremental ordinal starting at 0) for upcoming nodes.
Later, when it looks for fitting nodes for unschedulable pods, the binpacking
estimator will also build virtual nodes and place them in a snapshot fork to
evaluate scheduler predicates.
Those temporary virtual nodes are built using the same pattern (template name
and an index ordinal also starting at 0) as the one previously used by
getUpcomingNodeInfos, which means it will generate the same nodeinfos/nodesnames for nodegroups having upcoming nodes.
But adding nodes by the same name in an existing cluster snapshot isn't allowed,
and the evaluation attempt will fail. So no nodeInfo from node groups having
upcoming nodes can ever be considered as fitting unschedulable pods.
Practically this blocks re-upscales for nodegroups having upcoming nodes,
which can cause a significant delay (or suboptimal choices among the
possible node groups options). This wasn't the case until recently, as
getUpcomingNodeInfosand the estimator used different naming schemes.