[v1alpha2] Invalid Job Status

[codeflitting]

I created a TFJob，TFJobFailed come before TFJobRunning

Here is the TFJobSpec

apiVersion: "kubeflow.org/v1alpha2"
kind: "TFJob"
metadata:
  name: "dist-mnist-for-e2e-test"
spec:
  tfReplicaSpecs:
    PS:
      replicas: 2
      restartPolicy: Never
      template:
        spec:
          containers:
            - name: tensorflow
              image: codeflitting/tf-dist-mnist-test:1.0
              command:
              - /bin/sh
              - -c
              - sleep
    Worker:
      replicas: 4
      restartPolicy: OnFailure
      template:
        spec:
          containers:
            - name: tensorflow
              image: codeflitting/tf-dist-mnist-test:1.0
              command:
              - /bin/sh
              - -c
              - sleep

JobStatus:

  status:
    conditions:
    - lastTransitionTime: 2018-07-04T02:53:58Z
      lastUpdateTime: 2018-07-04T02:53:58Z
      message: TFJob dist-mnist-for-e2e-test is created.
      reason: TFJobCreated
      status: "True"
      type: Created
    - lastTransitionTime: 2018-07-04T02:53:58Z
      lastUpdateTime: 2018-07-04T02:54:03Z
      message: TFJob dist-mnist-for-e2e-test is failed.
      reason: TFJobFailed
      status: "True"
      type: Failed
    - lastTransitionTime: 2018-07-04T02:53:58Z
      lastUpdateTime: 2018-07-04T02:54:03Z
      message: TFJob dist-mnist-for-e2e-test is running.
      reason: TFJobRunning
      status: "True"
      type: Running
    tfReplicaStatuses:
      Chief: {}
      PS: {}
      Worker: {}

Discovered by @yph152

[codeflitting]

@gaocegege What do you think the final status should be

[gaocegege]

I am not sure why the job is failed. Is there any worker failed?

[yph152]

I think adding the following can solve it.

func setCondition(status *tfv1alpha2.TFJobStatus, condition tfv1alpha2.TFJobCondition) {                                                                                                                          
      if isFailed(*status) {                                                                                                                                                                                      
            return                                                                                                                                                                                                
      }                                                                                                                                                                                                           
      currentCond := getCondition(*status, condition.Type)                                                                                                                                                        
                                                                                                                                                                                                                  
      // Do nothing if condition doesn't change                                                                                                                                                                   
      if currentCond != nil && currentCond.Status == condition.Status && currentCond.Reason == condition.Reason {                                                                                                 
            return                                                                                                                                                                                                
      }