In TFjob, when the workers Completed, i want the ps Completed too, how can i do?

[nosmatch]

{
    "apiVersion": "kubeflow.org/v1alpha2",
    "kind": "TFJob",
    "metadata": {
        "name": "fengzhu-tf-v2-2"
    },
    "spec": {
        "tfReplicaSpecs": {
            "Master": {
                "replicas": 1,
                "restartPolicy": "OnFailure",
                "template": {
                    "spec": {
                        "containers": [
                            {
                                "command": [
                                    "python",
                                    "/distribute.py"
                                ],
                                "image": "registry.v2.wx.service.mogujie.org/public/tinytf_baseline:201806131831",
                                "name": "tensorflow",
                                "ports": [
                                    {
                                        "containerPort": 2222,
                                        "name": "tfjob-port"
                                    }
                                ],
                                "resources": {}
                            }
                        ]
                    }
                }
            },
            "PS": {
                "replicas": 1,
                "restartPolicy": "OnFailure",
                "template": {
                    "spec": {
                        "containers": [
                            {
                                "command": [
                                    "python",
                                    "/distribute.py"
                                ],
                                "image": "registry.v2.wx.service.mogujie.org/public/tinytf_baseline:201806131831",
                                "name": "tensorflow",
                                "ports": [
                                    {
                                        "containerPort": 2222,
                                        "name": "tfjob-port"
                                    }
                                ],
                                "resources": {}
                            }
                        ]
                    }
                }
            },
            "Worker": {
                "replicas": 1,
                "restartPolicy": "OnFailure",
                "template": {
                    "spec": {
                        "containers": [
                            {
                                "command": [
                                    "python",
                                    "/distribute.py"
                                ],
                                "image": "registry.v2.wx.service.mogujie.org/public/tinytf_baseline:201806131831",
                                "name": "tensorflow",
                                "ports": [
                                    {
                                        "containerPort": 2222,
                                        "name": "tfjob-port"
                                    }
                                ],
                                "resources": {}
                            }
                        ]
                    }
                }
            }
        }
    }
}

and for ps:
server = server = tf.train.Server(cluster, job_name=local_type, task_index=local_index)
...
if local_type == "ps":
print "local server is ps"
server.join()

[nosmatch]

@gaocegege

[gaocegege]

I do not think you could do it, because PS is designed to be long running in TF

[nosmatch]

but the ps belongs to one tfjob, and when i create a new tfjob we will create new ps, according my understand when a tfjob compled we will delete all resources the tfjob used, right?

[gaocegege]

We do not implement the logic to delete all pods and services after the TFJob is completed in v1alpha2. And even if we implement it, we do not set the ps to be completed, we just delete it.

[nosmatch]

which one will delete the ps server when TFjob completed? mster?

[nosmatch]

Thank you for your reply.

[gaocegege]

controller for v1alpha1 will do it.

[nosmatch]

what about v1alpha2？

[gaocegege]

We will investigate if there is another way to do the same thing without deletions. Ref #661

[gaocegege]

You could use features in #691 , to control the behaviour.