Make backoffBaseSeconds default consistent with timeout in waitForPodsReady

[mwielgus]

What would you like to be cleaned:

Increase the default for waitForPodsReady to 5-10 minutes.

Why is this needed:

Currently Kueue retries to run a workload almost immediately after it failed to start it for 5 minutes. If the situation was not favorable for the workload for 5 minutes (there was no space in the cluster, cluster autoscaler failed to deliver nodes, the workload is crashing immediately after start etc) it is unlikely to improve just after 10 or 20 seconds.

cc: @tenzen-y @mimowo

[tenzen-y]

SGTM
@mwielgus I'm not familiar with ClusterAutoscaler, so could you tell me about how many minutes does cluster-autoscaler prepare a new Node generally take?

[mwielgus]

This depends on the cloud provider. Usually somewhere between 1-3 minutes. Within 5 minutes the nodes should be up and pods running unless there is limited availability of the requested resources.

[alculquicondor]

I would put it to 1 minute and only for v0.7, where it's configurable.

[mimowo]

I agree with both suggestions.

• 0.6: yes, it is not configurable so keeping a lower base (10s as currently) might be more universal
• 0.7+: I think 1min might be more practical for users and admins who want to observe the timeout in action, say after config change, as waiting 5min in that case might be tedious.

[mimowo]

/trasc
EDIT: mistake :)

[mimowo]

@trasc

[mbobrovskyi]

/assign