Limit the number of restarts under ExitCode restartPolicy

[goyalankit]

In the current implementation, the runPolicy.BackoffLimit is only applicable on OnFailure and Always restart policies. I was wondering if there's a reason it's not supported on the ExitCode restart policy. Often in the case of OOM, the job exits with an exit code of 137 which is rightfully a retriable error. However, it will keep restarting indefinitely since the policy is not covered by the BackOffLimit. The behavior could be to honor the backOffLimit if present, else it keeps retrying indefinitely?

I am happy to submit a PR if you think this is a reasonable change.

[gaocegege]

Hi @goyalankit . Thanks for the issue.

I think it makes sense, WDYT @kubeflow/wg-training-leads

[Jeffwan]

I think this is reasonable improvement. @goyalankit Feel free to cut a PR and assign to us to help review

[johnugeorge]

Thanks for this

[Jeffwan]

Let's keep it open to track cherry-pick

/reopen

[google-oss-robot]

@Jeffwan: Reopened this issue.

In response to this:

Let's keep it open to track cherry-pick

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.