Pod Scheduling Readiness

[Huang-Wei]

Enhancement Description

• One-line enhancement description (can be used as a release note): Add a schedulingGates field to the Pod spec that marks a Pod's scheduling readiness.
• Kubernetes Enhancement Proposal: 3521-pod-scheduling-readiness/README.md
• Discussion Link: https://docs.google.com/document/d/1cPxF3wIbGRw_inv4CCG6z6qC4WjHAPR5GLT__dRBx7I/edit#
• Primary contact (assignee): @Huang-Wei
• Responsible SIGs: sig-scheduling
• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y): 1.26
  • Beta release target (x.y): 1.27
  • Stable release target (x.y): 1.30

Alpha

Preview Give feedback

1. KEP: Pod Scheduling Readiness #3522
   approved cncf-cla: yes kind/kep lgtm sig/scheduling size/XXL tide/merge-method-squash +7
   
2. [KEP-3521] Part 1: New Pod API .spec.schedulingGates kubernetes#113274
   api-review approved area/code-generation area/e2e-test-framework area/test cncf-cla: yes kind/api-change kind/feature lgtm needs-priority release-note sig/apps sig/scheduling sig/testing size/XL triage/accepted +16
   
3. [KEP-3521] Part 2: Core scheduling implementation kubernetes#113275
   api-review approved area/code-generation area/e2e-test-framework area/stable-metrics area/test cncf-cla: yes kind/api-change kind/feature lgtm needs-priority release-note sig/api-machinery sig/apps sig/instrumentation sig/scheduling sig/testing size/XL triage/accepted +19
   

   
4. [KEP-3521] Part 3: Bug fixes, integration & E2E Test kubernetes#113442
   approved area/code-generation area/e2e-test-framework area/stable-metrics area/test cncf-cla: yes kind/api-change kind/feature lgtm needs-priority release-note-none sig/api-machinery sig/apps sig/instrumentation sig/scheduling sig/testing size/L triage/accepted +18
   

   
5. Fix an accuracy issue of scheduler_pending_pods metric kubernetes#113946
   approved cncf-cla: yes kind/bug lgtm needs-priority needs-triage release-note-none sig/scheduling size/L +9
   
6. Doc for Alpha feature PodSchedulingReadiness website#37675
   approved cncf-cla: yes language/en lgtm sig/docs sig/scheduling size/L +7
   

Beta

Preview Give feedback

1. KEP-3521: Graduate PodSchedulingReadiness to beta #3739
   approved cncf-cla: yes kind/kep lgtm sig/scheduling size/L +6
   
2. Umbrella issue tracking Beta impl. of KEP 3521 (Pod Scheduling Readiness) kubernetes#113608
   10 of 10
   lifecycle/stale needs-triage sig/scheduling +3
   
3. Doc for Beta feature PodSchedulingReadiness website#39773
   approved cncf-cla: yes language/en lgtm sig/docs size/XS +6
   

Stable

Preview Give feedback

1. KEP 3521/3838: graduate PodSchedulingReadiness to stable #4455
   approved cncf-cla: yes kind/kep lgtm sig/scheduling size/M +6
   
2. Graduate PodSchedulingReadiness to stable kubernetes#123575
   approved area/code-generation area/test cncf-cla: yes kind/api-change kind/feature lgtm needs-priority needs-triage release-note sig/api-machinery sig/apps sig/node sig/scheduling sig/testing size/XL triage/accepted +17
   
3. promote feature PodSchedulingReadiness to stable website#45234
   approved cncf-cla: yes language/en lgtm sig/docs size/S +6
   

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

[Huang-Wei]

/sig scheduling

[ahg-g]

/label lead-opted-in

[ahg-g]

/milestone v1.26

[Atharva-Shinde]

Hey @kerthcet 👋, 1.26 Enhancements team here!

Just checking in as we approach Enhancements Freeze on 18:00 PDT on Thursday 6th October 2022.

This enhancement is targeting for stage alpha for 1.26 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP file using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable
• KEP has an updated detailed test plan section filled out
• KEP has up to date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements.

For this KEP, we would need to:

• Change the status of kep.yaml to implementable (I've also added a comment review)
• Include the new updated PR of this KEP in the Issue Description and get it merged before Enhancements Freeze to make this enhancement eligible for 1.26 release.

The status of this enhancement is marked as at risk. Please keep the issue description up-to-date with appropriate stages as well.
Thank you :)

[Huang-Wei]

Thanks @Atharva-Shinde.

• The KEP is updated to marked as implementable
• Waiting for @johnbelamaric or @wojtek-t to be the approver of PRR: KEP: Pod Scheduling Readiness #3522 (comment)

[Atharva-Shinde]

Hello @Huang-Wei 👋, just a quick check-in again, as we approach the 1.26 Enhancements freeze.

Please plan to get the PR #3522 merged before Enhancements freeze on 18:00 PDT on Thursday 6th October 2022 i.e tomorrow

For note, the current status of the enhancement is marked at-risk :)

[Huang-Wei]

Thanks for the reminder. It's 99% accomplished atm, just some final comments waiting for the approver to +1.

[rhockenbury]

With #3522 merged, we have this marked as tracked for v1.26.

[alculquicondor]

this label can be modified

Yes, it can, but you might be breaking a few first-party and third-party controllers that assume that this label matches the nodeName or at least that it is unique. The label is documented as well-known, so it should be treated with care https://kubernetes.io/docs/reference/labels-annotations-taints/#kubernetesiohostname

[fabiand]

A good example of how .spec.nodeName means "scheduling done" is by setting nodeName and an arbitrary resource, the resource will be ignored, and the pod fails with

…
status:
  phase: Failed
…
  message: 'Pod was rejected: Node didn''t have enough resource: cpu, requested: 400000000, used: 400038007, capacity: 159500'
  reason: OutOfcpu
…
  containerStatuses:
    - name: nginx
      state:
        terminated:
          exitCode: 137
          reason: ContainerStatusUnknown
          message: The container could not be located when the pod was terminated
…
      image: 'nginx:1.14.2'
      started: false

To some extent .spec.nodeName could be considered to be a scheduling gate as well - a special one: Only if it is set, then we can (are) schedule(d).
However, due to it's special meaning in the coordination with kubelet, we can not change it's seamtics.

I do not know the current state, but I do wonder - if we are not already doing this today - a Pod with scheduling gates and spec.nodeName should be rejected at admission time.

[vladikr]

A good example of how .spec.nodeName means "scheduling done" is by setting nodeName and an arbitrary resource, the resource will be ignored, and the pod fails with

…
status:
  phase: Failed
…
  message: 'Pod was rejected: Node didn''t have enough resource: cpu, requested: 400000000, used: 400038007, capacity: 159500'
  reason: OutOfcpu
…
  containerStatuses:
    - name: nginx
      state:
        terminated:
          exitCode: 137
          reason: ContainerStatusUnknown
          message: The container could not be located when the pod was terminated
…
      image: 'nginx:1.14.2'
      started: false

To some extent .spec.nodeName could be considered to be a scheduling gate as well - a special one: Only if it is set, then we can (are) schedule(d). However, due to it's special meaning in the coordination with kubelet, we can not change it's seamtics.

I do not know the current state, but I do wonder - if we are not already doing this today - a Pod with scheduling gates and spec.nodeName should be rejected at admission time.

@fabiand Yes, it will be rejected at the admission.
However, the Application-Aware Quota adds the scheduling gate to pods at the admission time as well.
Therefore, pods created with .spec.nodeName will get rejected in the namespaces where AAQ operates.
One example is with kubectl debug node which explicitly sets .spec.nodeName in the pod spec.

[fabiand]

I share that it's a general problem, but due to the special handling of .spec.nodeName I do not see how we can resolve the problem in

1. a backwards compatile manner
2. keeping the user spec

I do fear that - in your example - kubectl debug or oc debug should change and use affinity instead.

The core problem is that kubelet starts to react once nodeName is set.

Was it considered to change kubelet to only start acting once nodeName is set and schedulingGates is empty?

[alculquicondor]

Was it considered to change kubelet to only start acting once nodeName is set and schedulingGates is empty?

According to the version skew policy, the change would have to be in the kubelet for 3 versions before we can relax the validation in apiserver.

I guess that could be backwards compatible if we start in 1.31 and we allow scheduling_gates + nodeName in 1.34.
@liggitt wdyt?

One example is with kubectl debug node which explicitly sets .spec.nodeName in the pod spec.

IMO, that falls under the special case where it might make sense to skip scheduler or an external quota system. You probably wouldn't even want to set requests in a debug pod.

[fabiand]

probably wouldn't even want to set requests in a debug pod.

FWIW - I do wonder if debug pods should actually be guaranteed. I had a couple of cases where debug pods (as best effort) got killed quickly on busy nodes.

[liggitt]

I guess that could be backwards compatible if we start in 1.31 and we allow scheduling_gates + nodeName in 1.34.
@liggitt wdyt?

that seems like making the problem and confusion around use of spec.nodeName worse to me... I don't see a compelling reason to do that

[iholder101]

@alculquicondor

IMO, that falls under the special case where it might make sense to skip scheduler or an external quota system.

TBH, I still try to understand how skipping the scheduler is ever helpful (when you're not using a custom scheduler).
Can you please elaborate on how skipping the scheduler helps in this case? In other words, if we would change kubectl debug to use node affinity instead of nodeName, what would be the implications?

You probably wouldn't even want to set requests in a debug pod.

While this might be correct, the question to me is who makes the decision. Granting a user a knob to skip quota mechanisms feels to me like granting a linux user to bypass permission checks when writing to a file. In both cases the whole idea is to restrict the users and enforce them to comply to a certain policy. Handing the user the possibility to bypass such mechanisms seems entirely contradicting to me and de-facto it makes external quota mechanisms unpractical.

@liggitt

that seems like making the problem and confusion around use of spec.nodeName worse to me... I don't see a compelling reason to do that

Are you open to discussion on that?
IMHO this behavior is consistent with scheduling gates' intent to (as the KEP states) force creating pods in a "not-ready to schedule" state. I understand that technically the pod is not scheduled at all with nodeName set (although I think we all agree it's a surprising behavior that's kept for backward compatibility), however having kubelet to wait for the scheduling gates to be removed before running the pod with a some kind of "not ready" condition sounds very intuitive and consistent to me.

This way we can avoid breaking backward compatibility, support external quota mechanisms and extend scheduling-gates in a consistent matter, which IMHO makes the exceptional nodeName case to be less exceptional.

[liggitt]

having kubelet to wait for the scheduling gates to be removed before running the pod with a some kind of "not ready" condition sounds very intuitive and consistent to me.

Not to me... expecting pods which are already assigned to a node to run through scheduling gate removal phases (which couldn't do anything to affect the selected node) seems more confusing than the current behavior which simply forbids that combination. I don't think we should relax validation there and increase confusion.

[fabiand]

I (sadly) concur - If the nodeName is set, then it's a matter of fact that the scheduling has happened. This is the contract we can not break.

[fabiand]

Fun fact: I created a deployment with pod template that had nodeName defined AND a resource (cpu: 400k) which can not be met. The Deployment had an RC of 2. But due to this scheduling contradiction it ended up with 12k failed pods.

IOW: I wonder if this spec.nodeName is troublesome for other controllers in kube as well.

[Barakmor1]

I share that it's a general problem, but due to the special handling of .spec.nodeName I do not see how we can resolve the problem in

1. a backwards compatile manner
2. keeping the user spec

I do fear that - in your example - kubectl debug or oc debug should change and use affinity instead.

The reason kubectl debug or oc debug creates pods using .spec.nodeName instead of affinity is likely because it allows the debug pod to bypass taint constraints.

Although it might be achievable with tolerations as well.

[sreeram-venkitesh]

Hey folks 👋, 1.31 Enhancements Lead here.

Since this KEP graduated to GA in the v1.30 release, please update the status field in the kep.yaml file here to implemented. Once everything related to the KEP is completed and the status has been updated, we can close this issue.

/remove-label lead-opted-in

[Huang-Wei]

@sreeram-venkitesh the status field of 3521-pod-scheduling-readiness/kep.yaml has been updated to implementable. I think we can close this issue as completed now?

[sreeram-venkitesh]

@Huang-Wei Sorry, my mistake! It should be implemented since it has graduated to GA.

[Huang-Wei]

@sreeram-venkitesh feel free to close this issue.

[sreeram-venkitesh]

@Huang-Wei Sorry for the confusion, we still need to update the status to field to implemented in the kep.yaml file.

[Huang-Wei]

Sure, #4686 is gonna update it.

[sreeram-venkitesh]

Ah I missed it, thanks for linking it! 🙌🏼