Support assign more resources to Partial Admitted job when there are some idle resources

[KunWuLuan]

What would you like to be added:
If there is no pending workloads in ClusterQueue, a partial admitted workload can get more resource by a reclaiming schedule.

Why is this needed:
For example, two workloads are running in cluster. Both of them have 2 pods and each pod uses 1 GPU. When one of the workload completed, the other workload should have 4 GPUs to use.

Completion requirements:

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

[KunWuLuan]

If needed, I can work on this issue. 👋 👋 👋

[trasc]

There main issue , in my opinion , is that the changes in pod counts to the actual job is done in RunWithPodSetsInfo, pass this point, at least for batch/job to change the count we'll need to Stop and Restart the job.

Also just changing the counts in the Quota Reservation, without Eviction, can have implications to Admission Checks also.

[alculquicondor]

I'm supportive, but please prepare a KEP. Note some work in this direction #1038

The tricky part is preventing this race condition:

1. The scheduler creates a snapshot and is admitting a new workload
2. The new partial admission with resize feature increases the size of the job.
3. Resources are over-committed.

[alculquicondor]

cc @andrewsykim, as this relates to #77

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[tenzen-y]

/remove-lifecycle stale

[alculquicondor]

cc @andrewsykim @vicentefb

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[tenzen-y]

/remove-lifecycle stale

[tenzen-y]

/remove-lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[tenzen-y]

/lifecycle frozen

I guess that this depends on #77.

[mimowo]

I guess that this depends on #77.

It matches my understanding.