[bug]: pytorchjob status conditions out-of-order

[codeflitting]

Created after Succeeded

[gaocegege]

Can you give us more details about it? I think conditions does not require order, we can only rely on the time field in them.

[codeflitting]

When the job just ended the conditions is

• Created -> Running -> Succeeded

After a while the conditions is

• Running -> Succeeded -> Created

when the last conditions is succeeded，tf-operator wont update the conditions， i think this is a pytorch-operator bug

[gaocegege]

Gotcha, then I think so. And we should check it in tf-operator, too.

[johnugeorge]

I haven't seen this before. I couldn't reproduce this issue.
In the screenshot, I see that lastUpdateTime of Create event(2018-10-24) is later than the other two. How did it happen?

[codeflitting]

@johnugeorge

spec:
  cleanPodPolicy: None
  pytorchReplicaSpecs:
    Master:
      replicas: 1
      restartPolicy: OnFailure
      template:
        metadata:
          annotations:
            helm.sh/namespace: test
            helm.sh/path: pytorchjob-1540278456
            helm.sh/release: pytorchjob-1540278456
          creationTimestamp: null
        spec:
          containers:
          - command:
            - /bin/bash
            - -c
            - python /mnist-pytorch/train.py
            image: pytorch:v0.4.1-py36
            name: pytorch
            ports:
            - containerPort: 23456
              name: pytorchjob-port
            resources:
              limits:
                nvidia.com/gpu: "0"
              requests:
                cpu: 500m
                memory: 512Mi

[johnugeorge]

I haven't seen this before. I couldn't reproduce this issue.
In the screenshot, I see that lastUpdateTime of Create event(2018-10-24) is later than the other two. How did it happen?

Did you do any manual change?

[codeflitting]

I haven't seen this before. I couldn't reproduce this issue.
In the screenshot, I see that lastUpdateTime of Create event(2018-10-24) is later than the other two. How did it happen?

Did you do any manual change?

no， just a normal job

[johnugeorge]

I tried out with repo examples and I couldn't reproduce it. Are you seeing this consistently? Is the specified image public? I can try out with your image

[codeflitting]

are you seeing this consistently?

I didn't care about it before. Not all pytorch job like this，just a fraction

[gaocegege]

I looked through the code. We only have one place to update Created condition:

pytorch-operator/pkg/controller.v2/pytorch/job.go

Line 84 in 46e8cb6

err = updatePyTorchJobConditions(job, v1alpha2.PyTorchJobCreated, pytorchJobCreatedReason, msg)

Maybe it is caused by multiple runs of the operator. For example, you run the operator and deal with the create event. Then the operator is crashed and restarted, then addjob is called again. The condition is updated again.

🤔

[johnugeorge]

I am confused at the timestamp of the events. eg: the lastTransitionTime and lastUpdatedTime of the Create event

I looked through the code. We only have one place to update Created condition:

pytorch-operator/pkg/controller.v2/pytorch/job.go
Line 84 in 46e8cb6
err = updatePyTorchJobConditions(job, v1alpha2.PyTorchJobCreated, pytorchJobCreatedReason, msg)
Maybe it is caused by multiple runs of the operator. For example, you run the operator and deal with the create event. Then the operator is crashed and restarted, then addjob is called again. The condition is updated again.

🤔

[codeflitting]

pytorch-operator-operator-v1-0-745458d6f9-5tcth 1/1 Running 6 10d

It did reboot a few times

if the operator has more than one replicas，it maybe reproduce this issue，right ？

I looked through the code. We only have one place to update Created condition:

pytorch-operator/pkg/controller.v2/pytorch/job.go

Line 84 in 46e8cb6

err = updatePyTorchJobConditions(job, v1alpha2.PyTorchJobCreated, pytorchJobCreatedReason, msg)
Maybe it is caused by multiple runs of the operator. For example, you run the operator and deal with the create event. Then the operator is crashed and restarted, then addjob is called again. The condition is updated again.

🤔

[gaocegege]

I think so

[johnugeorge]

I feel, this should be common across operators then

[gaocegege]

Yeah, definitely

[gaocegege]

I think the created condition should be created and never updated.

I will work on the issue.

[gaocegege]

/assign

[johnugeorge]

This has been already fixed by #114

[johnugeorge]

Closing the issue

[johnugeorge]

/close

[k8s-ci-robot]

@johnugeorge: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.