scheduler: impose a backoff penalty on gated Pods

[sanposhiho]

What type of PR is this?

/kind feature

What this PR does / why we need it:

Starting the story from the concept of backoff in the scheduler; a backoff time in the scheduler is a penalty that we impose on Pods when they consume a scheduling cycle, but they didn't get scheduled and came back to the queue.

But, currently all gated Pods are always regarded as not backing off.
That is only correct for a vanilla scheduler because all Pods gated by SchedulingGates haven't experienced any scheduling and thus are not backing off for sure.
A custom PreEnqueue plugin might gate Pods after they experience some scheduling cycles; those mean -

1. Pods have experienced some scheduling cycles.
2. They get gated by a custom PreEnqueue plugin.
3. They get un-gated for some reasons.
4. 💥 Whoa! They're moved to activeQ without a backoff penalty.

Regardless of whether a Pod is gated or not, they are supposed to get a penalty if they wasted some scheduling cycles before.
It's the law in the scheduler; it's their obligation that they must meet before retrying a schedule again.

This PR changes isPodBackingoff() not to skip gated Pods so that we can prevent such Pods from exploiting loopholes, ignoring the law, and escaping a penalty.

Which issue(s) this PR fixes:

Fixes #125538

Special notes for your reviewer:

Does this PR introduce a user-facing change?

The scheduler retries gated Pods more appropriately, giving them a backoff penalty too.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

[k8s-ci-robot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sanposhiho

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• pkg/scheduler/OWNERS [sanposhiho]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[sanposhiho]

/hold

to go thru an approver.

[sanposhiho]

/cc @alculquicondor

[sanposhiho]

I found out I have to fix some tests. Just converted to WIP for now.

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: 9f6ecea95badeb34063e338f2136bb6a905b4cd0

[sanposhiho]

/assign @alculquicondor
for approval

[sanposhiho]

Looks like Aldo is on vacation.
/assign @kerthcet

[sanposhiho]

/cc @macsko @alculquicondor

Just fixed the conflict.

[macsko]

/lgtm

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: 6aa2aea7fee9c6b4a12019a9b4be80d5b8432268

[alculquicondor]

what is increasing attempts in this case?

Could we recreate podInfos for every case, instead?

[sanposhiho]

Some of test.operations (addPodUnschedulablePods) do.

[sanposhiho]

Could we recreate podInfos for every case, instead?

Given all test.operands referpInfos / pInfosWithDelay like the following, that'd require a big change in the test implementation, which I want to avoid (at least, in this PR)

			operands: [][]*framework.QueuedPodInfo{
				pInfos[:30], // Evern test case refers to the same pInfos. 
				pInfos[30:],
			},

[alculquicondor]

what's the name of the test for this?

[sanposhiho]

QueueHintFunction is called when Pod is gated by a plugin other than SchedulingGate test case in TestPriorityQueue_MoveAllToActiveOrBackoffQueueWithQueueingHint ensures that the Pod with zero attempt doesn't get backoff.

[alculquicondor]

what if the hints are disabled?

[sanposhiho]

The test isn't related to the feature gate.
It's just that when the feature gate is disabled we don't accept the queueing hint from the plugin, but still use the default queueing hint.

[sanposhiho]

@alculquicondor Updated based on your point.

[sanposhiho]

/retest

[alculquicondor]

/lgtm

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: 5189157ac5ac5fbfac5a470f15e8e97efd6353a2

[alculquicondor]

/hold cancel

[k8s-triage-robot]

The Kubernetes project has merge-blocking tests that are currently too flaky to consistently pass.

This bot retests PRs for certain kubernetes repos according to the following rules:

• The PR does have any do-not-merge/* labels
• The PR does not have the needs-ok-to-test label
• The PR is mergeable (does not have a needs-rebase label)
• The PR is approved (has cncf-cla: yes, lgtm, approved labels)
• The PR is failing tests required for merge

You can:

• Review the full test history for this PR
• Prevent this bot from retesting with /lgtm cancel or /hold
• Help make our tests less flaky by following our Flaky Tests Guide

/retest