Invalidate resource requirements on extended resources with only request set #57170

jiayingz · 2017-12-14T02:00:18Z

What this PR does / why we need it:

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #57276

Special notes for your reviewer:

Release note:

Returns an error for non overcommitable resources if they don't have limit field set in container spec.

jiayingz · 2017-12-14T02:00:34Z

/sig-scheduling

jiayingz · 2017-12-14T02:00:48Z

/release-note-non

jiayingz · 2017-12-14T02:01:06Z

/assign @vishh @ConnorDoyle

ConnorDoyle · 2017-12-14T04:45:47Z

pkg/apis/core/helper/helpers.go

@@ -167,6 +167,7 @@ var overcommitBlacklist = sets.NewString(string(core.ResourceNvidiaGPU))
 func IsOvercommitAllowed(name core.ResourceName) bool {
 	return IsDefaultNamespaceResource(name) &&
 		!IsHugePageResourceName(name) &&
+		!IsExtendedResourceName(name) &&


This predicate is technically correct but redundant since extended resources are by definition disjoint from default namespace resources. Probably we only need one of the two, but it would be helpful to add a comment saying why this was correct.

Agree, comment makes more sense over redundant line of code.

Good catch! Removed.

ConnorDoyle · 2017-12-14T04:47:31Z

pkg/apis/core/validation/validation.go

 			if quantity.Cmp(limitQuantity) != 0 && !helper.IsOvercommitAllowed(resourceName) {
 				allErrs = append(allErrs, field.Invalid(reqPath, quantity.String(), fmt.Sprintf("must be equal to %s limit", resourceName)))
 			} else if quantity.Cmp(limitQuantity) > 0 {
 				allErrs = append(allErrs, field.Invalid(reqPath, quantity.String(), fmt.Sprintf("must be less than or equal to %s limit", resourceName)))
 			}
-		} else if resourceName == core.ResourceNvidiaGPU {
+		} else !helper.IsOvercommitAllowed(resourceName) {


Good catch! 👍 This was missed in the first extended resources pass.

However, this is a breaking API change right? I don’t think we can do this and maintain API compatibility with 1.8.

@ConnorDoyle do you have any suggestions on how we should roll out this API change? Can we first update the extended resource doc to document this side effect for 1.8 and 1.9 and mention our plan on invalidating this kind of resource spec in 1.10, and then have the validation logic take effect in 1.10?

I think we should consider not making this change.

The semantics of resource requests are: a minimum amount provided to a container. There's no indication implied that bursting is allowed for arbitrary resources, although we allow it for cpu and memory and that is documented. Also, we do already disallow limit ≠ request in validation.

Could you outline the specific user benefit of the validation change? Then we can weigh it against the cost.

jiayingz · 2017-12-15T19:26:55Z

/assign @davidopp @bsalamat

Would like to get some opinions from the scheduling folks on this PR. In short, for extended resources introduced in 1.8, we clearly document that "Extended resources are only supported as integer resources and cannot be overcommitted" in its doc https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#extended-resources, but the current ValidateResourceRequirements check actually allows that only request field is set for extended resources and the current extended resource doc also implies that this kind of resource spec is allowed: "Extended resources cannot be overcommitted, so request and limit must be equal if both are present in a container spec." In this PR, I am thinking to change ValidateResourceRequirements logic to return an error for such resource spec to make it clear that extended resource doesn't allow overcommit, but this may break some existing pod specs that are formatted this way. Would like to get some opinions on whether we should leave the current validation logic as is and rely on the ER doc to communicate the fact that overcommit is not supported on ER, or we should consider to change the validation logic in a future release but document that plan now.

ConnorDoyle · 2017-12-15T20:17:50Z

Copying hidden reply from above:

I think we should consider not making this change.

The semantics of resource requests are: a minimum amount provided to a container. There's no indication implied that bursting is allowed for arbitrary resources, although we allow it for cpu and memory and that is documented. Also, we do already disallow limit ≠ request in validation.

Could you outline the specific user benefit of the validation change? Then we can weigh it against the cost.

bsalamat · 2017-12-16T02:41:51Z

@jiayingz Scheduler uses "Requests" for scheduling and does not control overcommitment. It is up to the node/kubelet to allow a pod to use more resources, including ER, than requested. If kubelet does not allow overcommitment of ER, that will probably be enough for preventing pods from using more resources than requested.
I think the change in this PR generally makes sense, but as you mentioned, is not backward compatible and could break existing pods. So, we may want to rely on existing mechanisms that do not allow pods to consume more of ERs than requested.

vikaschoudhary16 · 2017-12-16T12:04:56Z

I am unable to visualize a issue that we may hit if we skip this validation check. In CPU and memory case, it is required to put upper cap, before hand, using Limits because if we dont, any container may consume more than desired/disproportionate amount of these resource. On the contrary, with ER, Is it possible for a pod to consume more than allocated resources? I guess, NO.
Therefore, IMO, it should be safe to skip this change if i have not missed some use case.

ConnorDoyle · 2017-12-16T18:15:08Z

Related issue: #57276

jiayingz · 2017-12-18T17:53:24Z

Thanks a lot for the comments! With #57276 fixed, we can probably leave the validation part as is. Will leave this PR open for a few days and close it if no objection is raised.

vishh · 2017-12-18T19:20:26Z

pkg/apis/core/validation/validation.go

 			if quantity.Cmp(limitQuantity) != 0 && !helper.IsOvercommitAllowed(resourceName) {
 				allErrs = append(allErrs, field.Invalid(reqPath, quantity.String(), fmt.Sprintf("must be equal to %s limit", resourceName)))
 			} else if quantity.Cmp(limitQuantity) > 0 {
 				allErrs = append(allErrs, field.Invalid(reqPath, quantity.String(), fmt.Sprintf("must be less than or equal to %s limit", resourceName)))
 			}
-		} else if resourceName == core.ResourceNvidiaGPU {
+		} else if !helper.IsOvercommitAllowed(resourceName) {


Can we some unit tests for this?

vishh · 2017-12-18T19:47:07Z

As @jiayingz mentioned, extended resources are not expected to allow for overcommit to begin with until we have better Resource APIs that allow for expressing overcommit capabilities offered at the node level. I can't think of a use case for overcommit at the cluster level.
We made an error in documentation and we should fix both documentation and code.
I suspect the user base for extended resources is relatively small atm and so I hope such a breaking change won't impact a lot of users. We can offer an opt-out flag if needed to ease with the transition (I doubt it will be necessary though).

vishh · 2017-12-18T19:48:00Z

And as far as device plugin goes, since it does not support overcommit, it should require request to equal limits if it considers requests for allocations.

vikaschoudhary16 · 2017-12-19T04:06:55Z

@vishh We tried to keep the API same and since in current behavior. Requests without Limits , though, is allowed, but IIUC, this is not the condition for overcommit. For overcommit, Limits must be greater than Requests and that is not allowed in current behavior. Therefore, in current code, overcommit is not allowed.
What you want is, please correct me if i am wrong:

To keep this check from Jiyang's PR. This will ensure that limits are present.
revert my change is manager.go to use Limits only, as it was before.
Create a PR to fix documentation, which today says that limits are optional.

Thats fine too. Its just that we were not sure initially about changing the API behavior. Your point about a very small user base of ER, makes sense though. Please confirm if above three points is the way forward.

vishh · 2017-12-19T04:14:41Z

absence of limit typically means give me everything available on the node.
so today's behavior is enabling overcommit according to my understanding.
+1 on the plan proposed.

ConnorDoyle · 2017-12-19T04:59:17Z

Apologies for jumping the gun on the other PR. I assumed that we wouldn’t tolerate an incompatible API change, but after all should have waited for discussion to settle before proceeding.

Given that @vishh prefers to make the API change, the new plan makes sense. The compelling case for me is a future scenario where we have a way to specify scheduling properties of each resource. This would make the overcommit-ibility of a resource distinguishable from the use site in container specs and less implicit.

bsalamat · 2017-12-19T21:41:24Z

After reading @vishh's comments and given that backward compatibility is not a major concern (given the small number of users), this change LGTM.

/lgtm

@vishh

Automatic merge from submit-queue (batch tested with PRs 57591, 57369). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. Revert back #57278 **What this PR does / why we need it**: This PR reverts back to behavior of scanning Limits. **Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*: Related # #57276 #57170 **Special notes for your reviewer**: **Release note**: ```release-note None ``` /sig node /cc @vishh @ConnorDoyle @jiayingz

jiayingz · 2017-12-27T19:13:46Z

/assign @liggitt

ConnorDoyle · 2018-01-03T18:50:18Z

Would love to get this landed. I ran into #57276 myself again (!!!) since the other patch was reverted.

jiayingz · 2018-01-03T18:59:24Z

/assign @thockin

set.

liggitt · 2018-01-03T20:49:15Z

/approve

bsalamat · 2018-01-05T23:10:19Z

/lgtm

k8s-ci-robot · 2018-01-05T23:10:59Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bsalamat, jiayingz, liggitt

Associated issue: #57276

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

~~pkg/apis/core/OWNERS~~ [liggitt]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

k8s-github-robot · 2018-01-05T23:52:09Z

/test all [submit-queue is verifying that this PR is safe to merge]

k8s-github-robot · 2018-01-06T00:01:33Z

Automatic merge from submit-queue (batch tested with PRs 57037, 57170). If you want to cherry-pick this change to another branch, please follow the instructions here.

k8s-github-robot assigned derekwaynecarr and errordeveloper Dec 14, 2017

k8s-ci-robot assigned ConnorDoyle and vishh Dec 14, 2017

ConnorDoyle reviewed Dec 14, 2017

View reviewed changes

jiayingz force-pushed the validation branch 2 times, most recently from 2da34de to d62b12e Compare December 15, 2017 18:48

k8s-ci-robot assigned bsalamat and davidopp Dec 15, 2017

ConnorDoyle mentioned this pull request Dec 16, 2017

Device manager only considers extended resource limits when allocating devices. #57276

Closed

vishh reviewed Dec 18, 2017

View reviewed changes

vikaschoudhary16 mentioned this pull request Dec 19, 2017

Revert back #57278 #57369

Merged

ConnorDoyle mentioned this pull request Dec 19, 2017

Fix device manager to scan resources.Requests #57278

Merged

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Dec 19, 2017

k8s-ci-robot assigned liggitt Dec 27, 2017

k8s-ci-robot assigned thockin Jan 3, 2018

jiayingz force-pushed the validation branch from d62b12e to 8350aa2 Compare January 3, 2018 19:32

k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jan 3, 2018

k8s-github-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 3, 2018

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Jan 3, 2018

Invalidate resource requirements on extended resources with only request

66c1c5e

set.

jiayingz force-pushed the validation branch from 8350aa2 to 66c1c5e Compare January 3, 2018 20:35

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 3, 2018

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 5, 2018

k8s-github-robot merged commit 4bdf282 into kubernetes:master Jan 6, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Invalidate resource requirements on extended resources with only request set #57170

Invalidate resource requirements on extended resources with only request set #57170

jiayingz commented Dec 14, 2017 •

edited

Loading

jiayingz commented Dec 14, 2017

jiayingz commented Dec 14, 2017

jiayingz commented Dec 14, 2017

ConnorDoyle Dec 14, 2017

vikaschoudhary16 Dec 15, 2017

jiayingz Dec 15, 2017

ConnorDoyle Dec 14, 2017 •

edited

Loading

jiayingz Dec 15, 2017

ConnorDoyle Dec 15, 2017

jiayingz commented Dec 15, 2017

ConnorDoyle commented Dec 15, 2017

bsalamat commented Dec 16, 2017

vikaschoudhary16 commented Dec 16, 2017

ConnorDoyle commented Dec 16, 2017

jiayingz commented Dec 18, 2017

vishh Dec 18, 2017

vishh commented Dec 18, 2017

vishh commented Dec 18, 2017

vikaschoudhary16 commented Dec 19, 2017

vishh commented Dec 19, 2017

ConnorDoyle commented Dec 19, 2017

bsalamat commented Dec 19, 2017

jiayingz commented Dec 27, 2017

ConnorDoyle commented Jan 3, 2018

jiayingz commented Jan 3, 2018

liggitt commented Jan 3, 2018

bsalamat commented Jan 5, 2018

k8s-ci-robot commented Jan 5, 2018

k8s-github-robot commented Jan 5, 2018

k8s-github-robot commented Jan 6, 2018

Invalidate resource requirements on extended resources with only request set #57170

Invalidate resource requirements on extended resources with only request set #57170

Conversation

jiayingz commented Dec 14, 2017 • edited Loading

jiayingz commented Dec 14, 2017

jiayingz commented Dec 14, 2017

jiayingz commented Dec 14, 2017

ConnorDoyle Dec 14, 2017

Choose a reason for hiding this comment

vikaschoudhary16 Dec 15, 2017

Choose a reason for hiding this comment

jiayingz Dec 15, 2017

Choose a reason for hiding this comment

ConnorDoyle Dec 14, 2017 • edited Loading

Choose a reason for hiding this comment

jiayingz Dec 15, 2017

Choose a reason for hiding this comment

ConnorDoyle Dec 15, 2017

Choose a reason for hiding this comment

jiayingz commented Dec 15, 2017

ConnorDoyle commented Dec 15, 2017

bsalamat commented Dec 16, 2017

vikaschoudhary16 commented Dec 16, 2017

ConnorDoyle commented Dec 16, 2017

jiayingz commented Dec 18, 2017

vishh Dec 18, 2017

Choose a reason for hiding this comment

vishh commented Dec 18, 2017

vishh commented Dec 18, 2017

vikaschoudhary16 commented Dec 19, 2017

vishh commented Dec 19, 2017

ConnorDoyle commented Dec 19, 2017

bsalamat commented Dec 19, 2017

jiayingz commented Dec 27, 2017

ConnorDoyle commented Jan 3, 2018

jiayingz commented Jan 3, 2018

liggitt commented Jan 3, 2018

bsalamat commented Jan 5, 2018

k8s-ci-robot commented Jan 5, 2018

k8s-github-robot commented Jan 5, 2018

k8s-github-robot commented Jan 6, 2018

jiayingz commented Dec 14, 2017 •

edited

Loading

ConnorDoyle Dec 14, 2017 •

edited

Loading