Handle changing Job parallelism

[alculquicondor]

What would you like to be added:

Handle the change in parallelism. Before admission, this is simple: we need to update the workload.

What if the workload is already admitted? If there is enough quota, we could directly update the workload, even though that might violate FIFO semantics. Perhaps we will need to use scheduling gates to hold pods.

Why is this needed:

Parallelism is a mutable field.

Completion requirements:

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

[alculquicondor]

We could have a short term solution: handle scale down properly and disallow scale up via webhook.

[ahg-g]

Note that in the job controller, if parallelism changes, the job gets suspended and the workload object gets deleted, and so forcing re-queueing the job.

kueue/pkg/controller/workload/job/job_controller.go

Line 523 in 7059151

if *job.Spec.Parallelism != wl.Spec.PodSets[0].Count {

[mwielgus]

/assign @mwielgus

[ahg-g]

This is already handled, what else do we want to do with this one?

[ahg-g]

I think we want to think of this in the context of #77, not as a separate issue.

[alculquicondor]

I had an offline discussion with @mwielgus and we agreed that it's probably not worth doing anything in this space until scheduling readiness gates are GA.

So yes, closing in favor of #77.

[alculquicondor]

/close

[k8s-ci-robot]

@alculquicondor: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.