Add new condition JobReconcileFinished to fix performance issue #1097
Conversation
|
Hi @zionwu. Thanks for your PR. I'm waiting for a kubeflow member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
@gaocegege @johnugeorge Can you help to review this PR? Thanks. |
| // JobReconcileFinished means this job reached JobSucceeded or JobFailed | ||
| // and its sub-resources have been cleaned according to CleanPodPolicy, | ||
| // No more reconciliation is needed for this job. | ||
| JobReconcileFinished JobConditionType = "ReconcileFinished" |
There was a problem hiding this comment.
Name ReconcileFinished here is confusing to me. BTW, does 12h or 24h reconcile period works for you?
There was a problem hiding this comment.
Do you have a suggestion for the name of the status?
I never try using 12h or 24h reconcile period because in my environment I implemented another fix for #956, which requires reconcile period to be short.
There was a problem hiding this comment.
Gotcha. Do you have any suggestion @johnugeorge
And I think we should add it into kubeflow/common if we decide to merge it.
|
Hey @zionwu, TravisBuddy Request Identifier: 60ff2940-fbb8-11e9-9ef7-03a460f81868 |
| @@ -429,6 +436,14 @@ func (tc *TFController) reconcileTFJobs(tfjob *tfv1.TFJob) error { | |||
| tfjob.Status.ReplicaStatuses[rtype].Active = 0 | |||
| } | |||
| } | |||
|
|
|||
| //If the job is terminated and its sub-resources are deleted, it no longer need reconciliation, | |||
There was a problem hiding this comment.
I don't think this is correct
Because even a job is failed or succeeded,
there is a cleanupTFJob and it might haven't pass the TTL, so the tfjob haven't been deleted and need to wait until future reconcile.
If we mark it as finish reconcile here, then there will be no reconcile in the future
the cleanupTFJob will never be triggered
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
Fix issue #965.
This change is