Add watch function for TFJob python Client API #1122

jinchihe · 2019-12-31T09:41:06Z

From feedback, user would like to watch the TFJob with key info, such as status, like kubeflow get tfjob.
The PR is going to add watch function (added one more option watch=Flase) for TFJob python Client get() and waiting_for_job() API, as below:

tfjob_client = TFJobClient()
tfjob_client.wait_for_job('mnist', namespace=namespace, watch=True)
NAME                           STATE                TIME
mnist                          Created              2019-12-31T09:20:07Z
mnist                          Running              2019-12-31T09:20:19Z
mnist                          Running              2019-12-31T09:20:19Z
mnist                          Succeeded            2019-12-31T09:22:04Z

The CI test of /hack/verify-codegen.sh failed due to time changed, the PR also update that.

This change is

review-notebook-app · 2019-12-31T09:41:13Z

Check out this pull request on

You'll be able to see Jupyter notebook diff and discuss changes. Powered by ReviewNB.

coveralls · 2019-12-31T09:44:51Z

Coverage remained the same at 96.512% when pulling 6030761 on jinchihe:add_watch_api into c588dae on kubeflow:master.

TravisBuddy · 2019-12-31T09:44:53Z

Hey @jinchihe,
Your changes look good to me!

View build log

TravisBuddy Request Identifier: 30c28950-2bb2-11ea-9b13-1f3f6380198b

TravisBuddy · 2020-01-02T03:33:03Z

Travis tests have failed

Hey @jinchihe,
Please read the following log in order to understand the failure reason.
It'll be awesome if you fix what's wrong and commit the changes.

1st Build

View build log

hack/verify-codegen.sh

Generating deepcopy funcs
Generating clientset for tensorflow:v1 at github.com/kubeflow/tf-operator/pkg/client/clientset
Generating listers for tensorflow:v1 at github.com/kubeflow/tf-operator/pkg/client/listers
Generating informers for tensorflow:v1 at github.com/kubeflow/tf-operator/pkg/client/informers
Generating defaulters for tensorflow/v1
Generating OpenAPI specification for tensorflow/v1
diffing hack/../pkg against freshly generated codegen
diff -Naupr hack/../pkg/apis/tensorflow/v1/openapi_generated.go hack/../_tmp/pkg/apis/tensorflow/v1/openapi_generated.go
--- hack/../pkg/apis/tensorflow/v1/openapi_generated.go	2020-01-02 03:30:49.563594066 +0000
+++ hack/../_tmp/pkg/apis/tensorflow/v1/openapi_generated.go	2020-01-02 03:30:11.000000000 +0000
@@ -1,6 +1,6 @@
 // +build !ignore_autogenerated
 
-// Copyright 2020 The Kubeflow Authors
+// Copyright 2019 The Kubeflow Authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
diff -Naupr hack/../pkg/apis/tensorflow/v1/zz_generated.deepcopy.go hack/../_tmp/pkg/apis/tensorflow/v1/zz_generated.deepcopy.go
--- hack/../pkg/apis/tensorflow/v1/zz_generated.deepcopy.go	2020-01-02 03:30:37.579594066 +0000
+++ hack/../_tmp/pkg/apis/tensorflow/v1/zz_generated.deepcopy.go	2020-01-02 03:30:11.000000000 +0000
@@ -1,6 +1,6 @@
 // +build !ignore_autogenerated
 
-// Copyright 2020 The Kubeflow Authors
+// Copyright 2019 The Kubeflow Authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
diff -Naupr hack/../pkg/apis/tensorflow/v1/zz_generated.defaults.go hack/../_tmp/pkg/apis/tensorflow/v1/zz_generated.defaults.go
--- hack/../pkg/apis/tensorflow/v1/zz_generated.defaults.go	2020-01-02 03:30:46.487594066 +0000
+++ hack/../_tmp/pkg/apis/tensorflow/v1/zz_generated.defaults.go	2020-01-02 03:30:11.000000000 +0000
@@ -1,6 +1,6 @@
 // +build !ignore_autogenerated
 
-// Copyright 2020 The Kubeflow Authors
+// Copyright 2019 The Kubeflow Authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
diff -Naupr hack/../pkg/client/clientset/versioned/clientset.go hack/../_tmp/pkg/client/clientset/versioned/clientset.go
--- hack/../pkg/client/clientset/versioned/clientset.go	2020-01-02 03:30:39.779594066 +0000
+++ hack/../_tmp/pkg/client/clientset/versioned/clientset.go	2020-01-02 03:30:11.000000000 +0000
@@ -1,4 +1,4 @@
-// Copyright 2020 The Kubeflow Authors
+// Copyright 2019 The Kubeflow Authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
diff -Naupr hack/../pkg/client/clientset/versioned/doc.go hack/../_tmp/pkg/client/clientset/versioned/doc.go
--- hack/../pkg/client/clientset/versioned/doc.go	2020-01-02 03:30:39.779594066 +0000
+++ hack/../_tmp/pkg/client/clientset/versioned/doc.go	2020-01-02 03:30:11.000000000 +0000
@@ -1,4 +1,4 @@
-// Copyright 2020 The Kubeflow Authors
+// Copyright 2019 The Kubeflow Authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
diff -Naupr hack/../pkg/client/clientset/versioned/fake/clientset_generated.go hack/../_tmp/pkg/client/clientset/versioned/fake/clientset_generated.go
--- hack/../pkg/client/clientset/versioned/fake/clientset_generated.go	2020-01-02 03:30:39.815594066 +0000
+++ hack/../_tmp/pkg/client/clientset/versioned/fake/clientset_generated.go	2020-01-02 03:30:11.000000000 +0000
@@ -1,4 +1,4 @@
-// Copyright 2020 The Kubeflow Authors
+// Copyright 2019 The Kubeflow Authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
diff -Naupr hack/../pkg/client/clientset/versioned/fake/doc.go hack/../_tmp/pkg/client/clientset/versioned/fake/doc.go
--- hack/../pkg/client/clientset/versioned/fake/doc.go	2020-01-02 03:30:39.779594066 +0000
+++ hack/../_tmp/pkg/client/clientset/versioned/fake/doc.go	2020-01-02 03:30:11.000000000 +0000
@@ -1,4 +1,4 @@
-// Copyright 2020 The Kubeflow Authors
+// Copyright 2019 The Kubeflow Authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
diff -Naupr hack/../pkg/client/clientset/versioned/fake/register.go hack/../_tmp/pkg/client/clientset/versioned/fake/register.go
--- hack/../pkg/client/clientset/versioned/fake/register.go	2020-01-02 03:30:39.815594066 +0000
+++ hack/../_tmp/pkg/client/clientset/versioned/fake/register.go	2020-01-02 03:30:11.000000000 +0000
@@ -1,4 +1,4 @@
-// Copyright 2020 The Kubeflow Authors
+// Copyright 2019 The Kubeflow Authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
diff -Naupr hack/../pkg/client/clientset/versioned/scheme/doc.go hack/../_tmp/pkg/client/clientset/versioned/scheme/doc.go
--- hack/../pkg/client/clientset/versioned/scheme/doc.go	2020-01-02 03:30:39.779594066 +0000
+++ hack/../_tmp/pkg/client/clientset/versioned/scheme/doc.go	2020-01-02 03:30:11.000000000 +0000
@@ -1,4 +1,4 @@
-// Copyright 2020 The Kubeflow Authors
+// Copyright 2019 The Kubeflow Authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
diff -Naupr hack/../pkg/client/clientset/versioned/scheme/register.go hack/../_tmp/pkg/client/clientset/versioned/scheme/register.go
--- hack/../pkg/client/clientset/versioned/scheme/register.go	2020-01-02 03:30:39.779594066 +0000
+++ hack/../_tmp/pkg/client/clientset/versioned/scheme/register.go	2020-01-02 03:30:11.000000000 +0000
@@ -1,4 +1,4 @@
-// Copyright 2020 The Kubeflow Authors
+// Copyright 2019 The Kubeflow Authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
diff -Naupr hack/../pkg/client/clientset/versioned/typed/tensorflow/v1/doc.go hack/../_tmp/pkg/client/clientset/versioned/typed/tensorflow/v1/doc.go
--- hack/../pkg/client/clientset/versioned/typed/tensorflow/v1/doc.go	2020-01-02 03:30:39.815594066 +0000
+++ hack/../_tmp/pkg/client/clientset/versioned/typed/tensorflow/v1/doc.go	2020-01-02 03:30:11.000000000 +0000
@@ -1,4 +1,4 @@
-// Copyright 2020 The Kubeflow Authors
+// Copyright 2019 The Kubeflow Authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
diff -Naupr hack/../pkg/client/clientset/versioned/typed/tensorflow/v1/fake/doc.go hack/../_tmp/pkg/client/clientset/versioned/typed/tensorflow/v1/fake/doc.go
--- hack/../pkg/client/clientset/versioned/typed/tensorflow/v1/fake/doc.go	2020-01-02 03:30:39.819594066 +0000
+++ hack/../_tmp/pkg/client/clientset/versioned/typed/tensorflow/v1/fake/doc.go	2020-01-02 03:30:11.000000000 +0000
@@ -1,4 +1,4 @@
-// Copyright 2020 The Kubeflow Authors
+// Copyright 2019 The Kubeflow Authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
diff -Naupr hack/../pkg/client/clientset/versioned/typed/tensorflow/v1/fake/fake_tensorflow_client.go hack/../_tmp/pkg/client/clientset/versioned/typed/tensorflow/v1/fake/fake_tensorflow_client.go
--- hack/../pkg/client/clientset/versioned/typed/tensorflow/v1/fake/fake_tensorflow_client.go	2020-01-02 03:30:39.823594066 +0000
+++ hack/../_tmp/pkg/client/clientset/versioned/typed/tensorflow/v1/fake/fake_tensorflow_client.go	2020-01-02 03:30:11.000000000 +0000
@@ -1,4 +1,4 @@
-// Copyright 2020 The Kubeflow Authors
+// Copyright 2019 The Kubeflow Authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
diff -Naupr hack/../pkg/client/clientset/versioned/typed/tensorflow/v1/fake/fake_tfjob.go hack/../_tmp/pkg/client/clientset/versioned/typed/tensorflow/v1/fake/fake_tfjob.go
--- hack/../pkg/client/clientset/versioned/typed/tensorflow/v1/fake/fake_tfjob.go	2020-01-02 03:30:39.823594066 +0000
+++ hack/../_tmp/pkg/client/clientset/versioned/typed/tensorflow/v1/fake/fake_tfjob.go	2020-01-02 03:30:11.000000000 +0000
@@ -1,4 +1,4 @@
-// Copyright 2020 The Kubeflow Authors
+// Copyright 2019 The Kubeflow Authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
diff -Naupr hack/../pkg/client/clientset/versioned/typed/tensorflow/v1/generated_expansion.go hack/../_tmp/pkg/client/clientset/versioned/typed/tensorflow/v1/generated_expansion.go
--- hack/../pkg/client/clientset/versioned/typed/tensorflow/v1/generated_expansion.go	2020-01-02 03:30:39.819594066 +0000
+++ hack/../_tmp/pkg/client/clientset/versioned/typed/tensorflow/v1/generated_expansion.go	2020-01-02 03:30:11.000000000 +0000
@@ -1,4 +1,4 @@
-// Copyright 2020 The Kubeflow Authors
+// Copyright 2019 The Kubeflow Authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
diff -Naupr hack/../pkg/client/clientset/versioned/typed/tensorflow/v1/tensorflow_client.go hack/../_tmp/pkg/client/clientset/versioned/typed/tensorflow/v1/tensorflow_client.go
--- hack/../pkg/client/clientset/versioned/typed/tensorflow/v1/tensorflow_client.go	2020-01-02 03:30:39.819594066 +0000
+++ hack/../_tmp/pkg/client/clientset/versioned/typed/tensorflow/v1/tensorflow_client.go	2020-01-02 03:30:11.000000000 +0000
@@ -1,4 +1,4 @@
-// Copyright 2020 The Kubeflow Authors
+// Copyright 2019 The Kubeflow Authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
diff -Naupr hack/../pkg/client/clientset/versioned/typed/tensorflow/v1/tfjob.go hack/../_tmp/pkg/client/clientset/versioned/typed/tensorflow/v1/tfjob.go
--- hack/../pkg/client/clientset/versioned/typed/tensorflow/v1/tfjob.go	2020-01-02 03:30:39.819594066 +0000
+++ hack/../_tmp/pkg/client/clientset/versioned/typed/tensorflow/v1/tfjob.go	2020-01-02 03:30:11.000000000 +0000
@@ -1,4 +1,4 @@
-// Copyright 2020 The Kubeflow Authors
+// Copyright 2019 The Kubeflow Authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
diff -Naupr hack/../pkg/client/informers/externalversions/factory.go hack/../_tmp/pkg/client/informers/externalversions/factory.go
--- hack/../pkg/client/informers/externalversions/factory.go	2020-01-02 03:30:44.263594066 +0000
+++ hack/../_tmp/pkg/client/informers/externalversions/factory.go	2020-01-02 03:30:11.000000000 +0000
@@ -1,4 +1,4 @@
-// Copyright 2020 The Kubeflow Authors
+// Copyright 2019 The Kubeflow Authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
diff -Naupr hack/../pkg/client/informers/externalversions/generic.go hack/../_tmp/pkg/client/informers/externalversions/generic.go
--- hack/../pkg/client/informers/externalversions/generic.go	2020-01-02 03:30:44.263594066 +0000
+++ hack/../_tmp/pkg/client/informers/externalversions/generic.go	2020-01-02 03:30:11.000000000 +0000
@@ -1,4 +1,4 @@
-// Copyright 2020 The Kubeflow Authors
+// Copyright 2019 The Kubeflow Authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
diff -Naupr hack/../pkg/client/informers/externalversions/internalinterfaces/factory_interfaces.go hack/../_tmp/pkg/client/informers/externalversions/internalinterfaces/factory_interfaces.go
--- hack/../pkg/client/informers/externalversions/internalinterfaces/factory_interfaces.go	2020-01-02 03:30:44.259594066 +0000
+++ hack/../_tmp/pkg/client/informers/externalversions/internalinterfaces/factory_interfaces.go	2020-01-02 03:30:11.000000000 +0000
@@ -1,4 +1,4 @@
-// Copyright 2020 The Kubeflow Authors
+// Copyright 2019 The Kubeflow Authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
diff -Naupr hack/../pkg/client/informers/externalversions/tensorflow/interface.go hack/../_tmp/pkg/client/informers/externalversions/tensorflow/interface.go
--- hack/../pkg/client/informers/externalversions/tensorflow/interface.go	2020-01-02 03:30:44.263594066 +0000
+++ hack/../_tmp/pkg/client/informers/externalversions/tensorflow/interface.go	2020-01-02 03:30:11.000000000 +0000
@@ -1,4 +1,4 @@
-// Copyright 2020 The Kubeflow Authors
+// Copyright 2019 The Kubeflow Authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
diff -Naupr hack/../pkg/client/informers/externalversions/tensorflow/v1/interface.go hack/../_tmp/pkg/client/informers/externalversions/tensorflow/v1/interface.go
--- hack/../pkg/client/informers/externalversions/tensorflow/v1/interface.go	2020-01-02 03:30:44.259594066 +0000
+++ hack/../_tmp/pkg/client/informers/externalversions/tensorflow/v1/interface.go	2020-01-02 03:30:11.000000000 +0000
@@ -1,4 +1,4 @@
-// Copyright 2020 The Kubeflow Authors
+// Copyright 2019 The Kubeflow Authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
diff -Naupr hack/../pkg/client/informers/externalversions/tensorflow/v1/tfjob.go hack/../_tmp/pkg/client/informers/externalversions/tensorflow/v1/tfjob.go
--- hack/../pkg/client/informers/externalversions/tensorflow/v1/tfjob.go	2020-01-02 03:30:44.259594066 +0000
+++ hack/../_tmp/pkg/client/informers/externalversions/tensorflow/v1/tfjob.go	2020-01-02 03:30:11.000000000 +0000
@@ -1,4 +1,4 @@
-// Copyright 2020 The Kubeflow Authors
+// Copyright 2019 The Kubeflow Authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
diff -Naupr hack/../pkg/client/listers/tensorflow/v1/expansion_generated.go hack/../_tmp/pkg/client/listers/tensorflow/v1/expansion_generated.go
--- hack/../pkg/client/listers/tensorflow/v1/expansion_generated.go	2020-01-02 03:30:42.035594066 +0000
+++ hack/../_tmp/pkg/client/listers/tensorflow/v1/expansion_generated.go	2020-01-02 03:30:11.000000000 +0000
@@ -1,4 +1,4 @@
-// Copyright 2020 The Kubeflow Authors
+// Copyright 2019 The Kubeflow Authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
diff -Naupr hack/../pkg/client/listers/tensorflow/v1/tfjob.go hack/../_tmp/pkg/client/listers/tensorflow/v1/tfjob.go
--- hack/../pkg/client/listers/tensorflow/v1/tfjob.go	2020-01-02 03:30:42.035594066 +0000
+++ hack/../_tmp/pkg/client/listers/tensorflow/v1/tfjob.go	2020-01-02 03:30:11.000000000 +0000
@@ -1,4 +1,4 @@
-// Copyright 2020 The Kubeflow Authors
+// Copyright 2019 The Kubeflow Authors
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
hack/../pkg is out of date. Please run hack/update-codegen.sh

goveralls -service=travis-ci -v -package ./pkg/... -ignore "pkg/client/*/*.go,pkg/client/*/*/*.go,pkg/client/*/*/*/*.go,pkg/client/*/*/*/*/*.go,pkg/client/*/*/*/*/*/*.go,pkg/client/*/*/*/*/*/*/*.go,pkg/util/*.go,pkg/util/*/*.go,pkg/apis/tensorflow/*/zz_generated.*.go,pkg/apis/tensorflow/*/*_generated.go,pkg/apis/common/*/zz_generated.*.go,pkg/apis/common/*/*_generated.go"

=== RUN   TestSetTypeNames
--- PASS: TestSetTypeNames (0.00s)
=== RUN   TestSetDefaultTFJob
--- PASS: TestSetDefaultTFJob (0.00s)
=== RUN   TestIsChieforMaster
--- PASS: TestIsChieforMaster (0.00s)
PASS
coverage: 27.7% of statements in github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1	0.032s	coverage: 27.7% of statements in github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
=== RUN   TestValidateV1TFJobSpec
time="2020-01-02T03:32:21Z" level=error msg="TFJobSpec is not valid: Image is undefined in the container of Worker"
time="2020-01-02T03:32:21Z" level=error msg="TFJobSpec is not valid: There is no container named tensorflow in Worker"
--- PASS: TestValidateV1TFJobSpec (0.00s)
PASS
coverage: 20.1% of statements in github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation	0.031s	coverage: 20.1% of statements in github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1	[no test files]
=== RUN   TestGenGeneralName
--- PASS: TestGenGeneralName (0.00s)
PASS
coverage: 0.5% of statements in github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/common/jobcontroller	0.015s	coverage: 0.5% of statements in github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured	[no test files]
=== RUN   TestCreatePods
--- PASS: TestCreatePods (0.01s)
=== RUN   TestCreateService
time="2020-01-02T03:32:44Z" level=info msg="Controller test-tfjob created service empty_service"
--- PASS: TestCreateService (0.00s)
=== RUN   TestCreateServicesWithControllerRef
time="2020-01-02T03:32:44Z" level=info msg="Controller test-tfjob created service empty_service"
--- PASS: TestCreateServicesWithControllerRef (0.00s)
=== RUN   TestClaimServices
--- PASS: TestClaimServices (0.00s)
PASS
coverage: 24.4% of statements in github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/control	0.045s	coverage: 24.4% of statements in github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
=== RUN   TestNormalPath
time="2020-01-02T03:32:49Z" level=info msg="Creating TFJob controller"
time="2020-01-02T03:32:49Z" level=info msg="Creating Job controller"
time="2020-01-02T03:32:49Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-01-02T03:32:49Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2020-01-02T03:32:49Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2020-01-02T03:32:49Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=0, failed=0" job=default.test-tfjob uid=
time="2020-01-02T03:32:49Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2020-01-02T03:32:49Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2020-01-02T03:32:49Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2020-01-02T03:32:49Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2020-01-02T03:32:49Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (2.853771ms)" job=default.test-tfjob
time="2020-01-02T03:32:49Z" level=info msg="Creating TFJob controller"
time="2020-01-02T03:32:49Z" level=info msg="Creating Job controller"
time="2020-01-02T03:32:49Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-01-02T03:32:49Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2020-01-02T03:32:49Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2020-01-02T03:32:49Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (457.776µs)" job=default.test-tfjob
time="2020-01-02T03:32:49Z" level=info msg="Creating TFJob controller"
time="2020-01-02T03:32:49Z" level=info msg="Creating Job controller"
time="2020-01-02T03:32:49Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-01-02T03:32:49Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2020-01-02T03:32:49Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=1, failed=0" job=default.test-tfjob uid=
time="2020-01-02T03:32:49Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2020-01-02T03:32:49Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2020-01-02T03:32:49Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2020-01-02T03:32:49Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2020-01-02T03:32:49Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (543.962µs)" job=default.test-tfjob
time="2020-01-02T03:32:49Z" level=info msg="Creating TFJob controller"
time="2020-01-02T03:32:49Z" level=info msg="Creating Job controller"
time="2020-01-02T03:32:49Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-01-02T03:32:49Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2020-01-02T03:32:49Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2020-01-02T03:32:49Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2020-01-02T03:32:49Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2020-01-02T03:32:49Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2020-01-02T03:32:49Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2020-01-02T03:32:49Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
E0102 03:32:49.513697   10398 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(0xc000584a50), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"PS":(*v1.ReplicaSpec)(0xc000356dc0), "Worker":(*v1.ReplicaSpec)(0xc000356b00)}}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"Worker":(*v1.ReplicaStatus)(0xc0003a4ac0)}, StartTime:(*v1.Time)(0xc000380ae0), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'TFJobSucceeded' 'TFJob test-tfjob successfully completed.'
time="2020-01-02T03:32:49Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2020-01-02T03:32:49Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (872.715µs)" job=default.test-tfjob
time="2020-01-02T03:32:49Z" level=info msg="Creating TFJob controller"
time="2020-01-02T03:32:49Z" level=info msg="Creating Job controller"
time="2020-01-02T03:32:49Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-01-02T03:32:49Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2020-01-02T03:32:49Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2020-01-02T03:32:49Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2020-01-02T03:32:49Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (253.167µs)" job=default.test-tfjob
time="2020-01-02T03:32:49Z" level=info msg="Creating TFJob controller"
time="2020-01-02T03:32:49Z" level=info msg="Creating Job controller"
time="2020-01-02T03:32:49Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-01-02T03:32:49Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2020-01-02T03:32:49Z" level=info msg="Need to create new pod: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2020-01-02T03:32:49Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2020-01-02T03:32:49Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2020-01-02T03:32:49Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2020-01-02T03:32:49Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2020-01-02T03:32:49Z" level=info msg="need to create new service: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2020-01-02T03:32:49Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2020-01-02T03:32:49Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2020-01-02T03:32:49Z" level=info msg="Need to create new pod: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2020-01-02T03:32:49Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2020-01-02T03:32:49Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2020-01-02T03:32:49Z" level=info msg="need to create new service: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2020-01-02T03:32:49Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2020-01-02T03:32:49Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (700.356µs)" job=default.test-tfjob
time="2020-01-02T03:32:49Z" level=info msg="Creating TFJob controller"
time="2020-01-02T03:32:49Z" level=info msg="Creating Job controller"
time="2020-01-02T03:32:49Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-01-02T03:32:49Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2020-01-02T03:32:49Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2020-01-02T03:32:49Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.599567ms)" job=default.test-tfjob
time="2020-01-02T03:32:49Z" level=info msg="Creating TFJob controller"
time="2020-01-02T03:32:49Z" level=info msg="Creating Job controller"
time="2020-01-02T03:32:49Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-01-02T03:32:49Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2020-01-02T03:32:49Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2020-01-02T03:32:49Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2020-01-02T03:32:49Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2020-01-02T03:32:49Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2020-01-02T03:32:49Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2020-01-02T03:32:49Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2020-01-02T03:32:49Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2020-01-02T03:32:49Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (582.785µs)" job=default.test-tfjob
--- PASS: TestNormalPath (0.02s)
=== RUN   TestRun
time="2020-01-02T03:32:49Z" level=info msg="Creating TFJob controller"
time="2020-01-02T03:32:49Z" level=info msg="Creating Job controller"
time="2020-01-02T03:32:49Z" level=info msg="Starting TFJob controller"
time="2020-01-02T03:32:49Z" level=info msg="Waiting for informer caches to sync"
time="2020-01-02T03:32:49Z" level=info msg="Starting 1 workers"
time="2020-01-02T03:32:49Z" level=info msg="Started workers"
time="2020-01-02T03:32:50Z" level=info msg="Shutting down workers"
--- PASS: TestRun (0.50s)
=== RUN   TestAddTFJob
time="2020-01-02T03:32:50Z" level=info msg="Creating TFJob controller"
time="2020-01-02T03:32:50Z" level=info msg="Creating Job controller"
time="2020-01-02T03:32:50Z" level=info msg="Starting TFJob controller"
time="2020-01-02T03:32:50Z" level=info msg="Waiting for informer caches to sync"
time="2020-01-02T03:32:50Z" level=info msg="TFJob test-tfjob is created." job=default.test-tfjob uid=
time="2020-01-02T03:32:50Z" level=info msg="Starting 1 workers"
time="2020-01-02T03:32:50Z" level=info msg="Started workers"
--- PASS: TestAddTFJob (0.10s)
time="2020-01-02T03:32:50Z" level=info msg="Shutting down workers"
=== RUN   TestCopyLabelsAndAnnotation
time="2020-01-02T03:32:50Z" level=info msg="Creating TFJob controller"
time="2020-01-02T03:32:50Z" level=info msg="Creating Job controller"
time="2020-01-02T03:32:50Z" level=info msg="Starting TFJob controller"
time="2020-01-02T03:32:50Z" level=info msg="Waiting for informer caches to sync"
time="2020-01-02T03:32:50Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-01-02T03:32:50Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2020-01-02T03:32:50Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2020-01-02T03:32:50Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2020-01-02T03:32:50Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (480.022µs)" job=default.test-tfjob
--- PASS: TestCopyLabelsAndAnnotation (0.00s)
=== RUN   TestDeletePodsAndServices
time="2020-01-02T03:32:50Z" level=info msg="Starting 1 workers"
time="2020-01-02T03:32:50Z" level=info msg="Started workers"
time="2020-01-02T03:32:50Z" level=info msg="Shutting down workers"
time="2020-01-02T03:32:50Z" level=info msg="Creating TFJob controller"
time="2020-01-02T03:32:50Z" level=info msg="Creating Job controller"
time="2020-01-02T03:32:50Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-01-02T03:32:50Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (372.06µs)" job=default.test-tfjob
time="2020-01-02T03:32:50Z" level=info msg="Creating TFJob controller"
time="2020-01-02T03:32:50Z" level=info msg="Creating Job controller"
time="2020-01-02T03:32:50Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-01-02T03:32:50Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (288.271µs)" job=default.test-tfjob
time="2020-01-02T03:32:50Z" level=info msg="Creating TFJob controller"
time="2020-01-02T03:32:50Z" level=info msg="Creating Job controller"
time="2020-01-02T03:32:50Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-01-02T03:32:50Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (347.031µs)" job=default.test-tfjob
time="2020-01-02T03:32:50Z" level=info msg="Creating TFJob controller"
time="2020-01-02T03:32:50Z" level=info msg="Creating Job controller"
time="2020-01-02T03:32:50Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-01-02T03:32:50Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (300.048µs)" job=default.test-tfjob
--- PASS: TestDeletePodsAndServices (0.00s)
=== RUN   TestCleanupTFJob
time="2020-01-02T03:32:50Z" level=info msg="Creating TFJob controller"
time="2020-01-02T03:32:50Z" level=info msg="Creating Job controller"
time="2020-01-02T03:32:50Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-01-02T03:32:50Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (212.892µs)" job=default.test-tfjob
time="2020-01-02T03:32:50Z" level=info msg="Creating TFJob controller"
time="2020-01-02T03:32:50Z" level=info msg="Creating Job controller"
time="2020-01-02T03:32:50Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-01-02T03:32:50Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (183.737µs)" job=default.test-tfjob
time="2020-01-02T03:32:50Z" level=info msg="Creating TFJob controller"
time="2020-01-02T03:32:50Z" level=info msg="Creating Job controller"
time="2020-01-02T03:32:52Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-01-02T03:32:52Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (375.368µs)" job=default.test-tfjob
--- PASS: TestCleanupTFJob (2.00s)
=== RUN   TestActiveDeadlineSeconds
time="2020-01-02T03:32:52Z" level=info msg="Creating TFJob controller"
time="2020-01-02T03:32:52Z" level=info msg="Creating Job controller"
time="2020-01-02T03:32:52Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-01-02T03:32:52Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2020-01-02T03:32:52Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2020-01-02T03:32:52Z" level=info msg="Creating TFJob controller"
time="2020-01-02T03:32:52Z" level=info msg="Creating Job controller"
time="2020-01-02T03:32:54Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
--- PASS: TestActiveDeadlineSeconds (2.01s)
=== RUN   TestBackoffForOnFailure
time="2020-01-02T03:32:54Z" level=info msg="Creating TFJob controller"
time="2020-01-02T03:32:54Z" level=info msg="Creating Job controller"
time="2020-01-02T03:32:54Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-01-02T03:32:54Z" level=warning msg="The restart policy of replica PS of the job test-tfjob is not OnFailure or Always. Not counted in backoff limit." job=default.test-tfjob uid=
time="2020-01-02T03:32:54Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (319.186µs)" job=default.test-tfjob
--- PASS: TestBackoffForOnFailure (0.00s)
=== RUN   TestAddPod
time="2020-01-02T03:32:54Z" level=info msg="Creating TFJob controller"
time="2020-01-02T03:32:54Z" level=info msg="Creating Job controller"
time="2020-01-02T03:32:54Z" level=info msg="Starting TFJob controller"
time="2020-01-02T03:32:54Z" level=info msg="Waiting for informer caches to sync"
time="2020-01-02T03:32:54Z" level=info msg="Starting 1 workers"
time="2020-01-02T03:32:54Z" level=info msg="Started workers"
--- PASS: TestAddPod (0.10s)
=== RUN   TestClusterSpec
time="2020-01-02T03:32:54Z" level=info msg="Shutting down workers"
--- PASS: TestClusterSpec (0.00s)
=== RUN   TestIsDistributed
--- PASS: TestIsDistributed (0.00s)
=== RUN   TestRestartPolicy
--- PASS: TestRestartPolicy (0.00s)
=== RUN   TestExitCode
time="2020-01-02T03:32:54Z" level=info msg="Creating TFJob controller"
time="2020-01-02T03:32:54Z" level=info msg="Creating Job controller"
time="2020-01-02T03:32:54Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-01-02T03:32:54Z" level=info msg="Ignoring inactive pod default/worker-0 in state Failed, deletion time <nil>"
time="2020-01-02T03:32:54Z" level=info msg="Pod: default.worker-0 exited with code 130" job=default.test-tfjob replica-type=worker uid=
E0102 03:32:54.243469   10398 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(0xc000378d60), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"Worker":(*v1.ReplicaSpec)(0xc0002bdb80)}}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"Worker":(*v1.ReplicaStatus)(0xc0000466f0)}, StartTime:(*v1.Time)(nil), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'ExitedWithCode' 'Pod: default.worker-0 exited with code 130'
time="2020-01-02T03:32:54Z" level=info msg="Need to restart the pod: default.worker-0" job=default.test-tfjob replica-type=worker uid=
time="2020-01-02T03:32:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=1" job=default.test-tfjob uid=
E0102 03:32:54.243556   10398 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(0xc000378d60), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"Worker":(*v1.ReplicaSpec)(0xc0002bdb80)}}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"Worker":(*v1.ReplicaStatus)(0xc0000466f0)}, StartTime:(*v1.Time)(0xc0003810a0), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Warning' 'TFJobRestarting' 'TFJob test-tfjob is restarting because 1 Worker replica(s) failed.'
time="2020-01-02T03:32:54Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2020-01-02T03:32:54Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (342.67µs)" job=default.test-tfjob
--- PASS: TestExitCode (0.00s)
=== RUN   TestAddService
time="2020-01-02T03:32:54Z" level=info msg="Creating TFJob controller"
time="2020-01-02T03:32:54Z" level=info msg="Creating Job controller"
time="2020-01-02T03:32:54Z" level=info msg="Starting TFJob controller"
time="2020-01-02T03:32:54Z" level=info msg="Waiting for informer caches to sync"
time="2020-01-02T03:32:54Z" level=info msg="Starting TFJob controller"
time="2020-01-02T03:32:54Z" level=info msg="Waiting for informer caches to sync"
time="2020-01-02T03:32:54Z" level=info msg="Starting 1 workers"
time="2020-01-02T03:32:54Z" level=info msg="Started workers"
time="2020-01-02T03:32:54Z" level=info msg="Shutting down workers"
time="2020-01-02T03:32:54Z" level=info msg="Starting 1 workers"
time="2020-01-02T03:32:54Z" level=info msg="Started workers"
--- PASS: TestAddService (0.10s)
=== RUN   TestFailed
time="2020-01-02T03:32:54Z" level=info msg="Shutting down workers"
time="2020-01-02T03:32:54Z" level=info msg="Creating TFJob controller"
time="2020-01-02T03:32:54Z" level=info msg="Creating Job controller"
time="2020-01-02T03:32:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=0, failed=1" job=default.test-tfjob uid=
E0102 03:32:54.345543   10398 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(0xc0005bdcf0), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"Worker":(*v1.ReplicaSpec)(0xc000554840)}}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"Worker":(*v1.ReplicaStatus)(0xc0003a4ae0)}, StartTime:(*v1.Time)(0xc000247120), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'TFJobFailed' 'TFJob test-tfjob has failed because 1 Worker replica(s) failed.'
--- PASS: TestFailed (0.00s)
=== RUN   TestStatus
time="2020-01-02T03:32:54Z" level=info msg="Creating TFJob controller"
time="2020-01-02T03:32:54Z" level=info msg="Creating Job controller"
time="2020-01-02T03:32:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2020-01-02T03:32:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2020-01-02T03:32:54Z" level=info msg="Creating TFJob controller"
time="2020-01-02T03:32:54Z" level=info msg="Creating Job controller"
time="2020-01-02T03:32:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2020-01-02T03:32:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2020-01-02T03:32:54Z" level=info msg="Creating TFJob controller"
time="2020-01-02T03:32:54Z" level=info msg="Creating Job controller"
time="2020-01-02T03:32:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2020-01-02T03:32:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2020-01-02T03:32:54Z" level=info msg="Creating TFJob controller"
time="2020-01-02T03:32:54Z" level=info msg="Creating Job controller"
time="2020-01-02T03:32:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2020-01-02T03:32:54Z" level=info msg="Creating TFJob controller"
time="2020-01-02T03:32:54Z" level=info msg="Creating Job controller"
time="2020-01-02T03:32:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2020-01-02T03:32:54Z" level=info msg="Creating TFJob controller"
time="2020-01-02T03:32:54Z" level=info msg="Creating Job controller"
time="2020-01-02T03:32:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2020-01-02T03:32:54Z" level=info msg="Creating TFJob controller"
time="2020-01-02T03:32:54Z" level=info msg="Creating Job controller"
time="2020-01-02T03:32:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2020-01-02T03:32:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2020-01-02T03:32:54Z" level=info msg="Creating TFJob controller"
time="2020-01-02T03:32:54Z" level=info msg="Creating Job controller"
time="2020-01-02T03:32:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=2, failed=2" job=default.test-tfjob uid=
time="2020-01-02T03:32:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2020-01-02T03:32:54Z" level=info msg="Creating TFJob controller"
time="2020-01-02T03:32:54Z" level=info msg="Creating Job controller"
time="2020-01-02T03:32:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=2, running=0, failed=2" job=default.test-tfjob uid=
time="2020-01-02T03:32:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2020-01-02T03:32:54Z" level=info msg="Creating TFJob controller"
time="2020-01-02T03:32:54Z" level=info msg="Creating Job controller"
time="2020-01-02T03:32:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=3, failed=0" job=default.test-tfjob uid=
time="2020-01-02T03:32:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2020-01-02T03:32:54Z" level=info msg="Creating TFJob controller"
time="2020-01-02T03:32:54Z" level=info msg="Creating Job controller"
time="2020-01-02T03:32:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2020-01-02T03:32:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2020-01-02T03:32:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2020-01-02T03:32:54Z" level=info msg="Creating TFJob controller"
time="2020-01-02T03:32:54Z" level=info msg="Creating Job controller"
time="2020-01-02T03:32:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2020-01-02T03:32:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2020-01-02T03:32:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2020-01-02T03:32:54Z" level=info msg="Creating TFJob controller"
time="2020-01-02T03:32:54Z" level=info msg="Creating Job controller"
time="2020-01-02T03:32:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2020-01-02T03:32:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2020-01-02T03:32:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=1, failed=1" job=default.test-tfjob uid=
time="2020-01-02T03:32:54Z" level=info msg="Creating TFJob controller"
time="2020-01-02T03:32:54Z" level=info msg="Creating Job controller"
time="2020-01-02T03:32:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2020-01-02T03:32:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2020-01-02T03:32:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2020-01-02T03:32:54Z" level=info msg="Creating TFJob controller"
time="2020-01-02T03:32:54Z" level=info msg="Creating Job controller"
time="2020-01-02T03:32:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2020-01-02T03:32:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2020-01-02T03:32:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2020-01-02T03:32:54Z" level=info msg="Creating TFJob controller"
time="2020-01-02T03:32:54Z" level=info msg="Creating Job controller"
time="2020-01-02T03:32:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2020-01-02T03:32:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2020-01-02T03:32:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
--- PASS: TestStatus (0.01s)
=== RUN   TestGenOwnerReference
--- PASS: TestGenOwnerReference (0.00s)
=== RUN   TestGenLabels
--- PASS: TestGenLabels (0.00s)
=== RUN   TestConvertTFJobToUnstructured
--- PASS: TestConvertTFJobToUnstructured (0.00s)
PASS
coverage: 53.1% of statements in github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow	4.891s	coverage: 53.1% of statements in github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
?   	github.com/kubeflow/tf-operator/pkg/logger	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/util	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/util/k8sutil	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/util/signals	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/util/train	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/version	[no test files]
ignoring pkg/apis/tensorflow/v1/openapi_generated.go
ignoring pkg/apis/tensorflow/v1/zz_generated.deepcopy.go
ignoring pkg/apis/tensorflow/v1/zz_generated.defaults.go
ignoring pkg/util/util.go
Job #2691.1
https://coveralls.io/jobs/57353271

TravisBuddy Request Identifier: 92e94830-2d10-11ea-b5b0-b361e22ae1a5

TravisBuddy · 2020-01-02T06:39:51Z

Hey @jinchihe,
Your changes look good to me!

View build log

TravisBuddy Request Identifier: ac563a70-2d2a-11ea-b5b0-b361e22ae1a5

gaocegege

/lgtm

jinchihe · 2020-01-02T06:59:49Z

Thanks a lot! @gaocegege
/cc @johnugeorge

johnugeorge · 2020-01-02T07:01:04Z

/approve

k8s-ci-robot · 2020-01-02T07:01:10Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: johnugeorge

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [johnugeorge]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

gaocegege · 2020-01-02T07:02:52Z

/lgtm

TravisBuddy · 2020-01-02T07:56:28Z

Hey @jinchihe,
Your changes look good to me!

View build log

TravisBuddy Request Identifier: 609e38c0-2d35-11ea-b5b0-b361e22ae1a5

TravisBuddy · 2020-01-02T08:16:10Z

Hey @jinchihe,
Your changes look good to me!

View build log

TravisBuddy Request Identifier: 20adeff0-2d38-11ea-b5b0-b361e22ae1a5

TravisBuddy · 2020-01-02T08:30:44Z

Hey @jinchihe,
Your changes look good to me!

View build log

TravisBuddy Request Identifier: 29bd3400-2d3a-11ea-b5b0-b361e22ae1a5

TravisBuddy · 2020-01-02T08:48:46Z

Hey @jinchihe,
Your changes look good to me!

View build log

TravisBuddy Request Identifier: aec20520-2d3c-11ea-a185-6fb6d15019ed

gaocegege · 2020-01-02T09:33:58Z

/lgtm

k8s-ci-robot added the do-not-merge/work-in-progress label Dec 31, 2019

k8s-ci-robot requested review from ChanYiLin and jimexist December 31, 2019 09:41

k8s-ci-robot added the size/L label Dec 31, 2019

jinchihe changed the title ~~WIP: Add watch function for TFJob python Client API~~ Add watch function for TFJob python Client API Jan 2, 2020

k8s-ci-robot removed the do-not-merge/work-in-progress label Jan 2, 2020

gaocegege reviewed Jan 2, 2020

View reviewed changes

k8s-ci-robot requested a review from johnugeorge January 2, 2020 06:59

k8s-ci-robot added the approved label Jan 2, 2020

k8s-ci-robot assigned gaocegege Jan 2, 2020

k8s-ci-robot added lgtm and removed lgtm labels Jan 2, 2020

Add watch function for TFJob python Client API

6030761

k8s-ci-robot added the lgtm label Jan 2, 2020

k8s-ci-robot merged commit 2228b72 into kubeflow:master Jan 2, 2020

jinchihe deleted the add_watch_api branch January 2, 2020 09:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add watch function for TFJob python Client API #1122

Add watch function for TFJob python Client API #1122

jinchihe commented Dec 31, 2019 •

edited

Loading

review-notebook-app bot commented Dec 31, 2019

coveralls commented Dec 31, 2019 •

edited

Loading

TravisBuddy commented Dec 31, 2019

TravisBuddy commented Jan 2, 2020

TravisBuddy commented Jan 2, 2020

gaocegege left a comment

jinchihe commented Jan 2, 2020

johnugeorge commented Jan 2, 2020

k8s-ci-robot commented Jan 2, 2020

gaocegege commented Jan 2, 2020

TravisBuddy commented Jan 2, 2020

TravisBuddy commented Jan 2, 2020

TravisBuddy commented Jan 2, 2020

TravisBuddy commented Jan 2, 2020

gaocegege commented Jan 2, 2020

Add watch function for TFJob python Client API #1122

Add watch function for TFJob python Client API #1122

Conversation

jinchihe commented Dec 31, 2019 • edited Loading

review-notebook-app bot commented Dec 31, 2019

coveralls commented Dec 31, 2019 • edited Loading

TravisBuddy commented Dec 31, 2019

TravisBuddy Request Identifier: 30c28950-2bb2-11ea-9b13-1f3f6380198b

TravisBuddy commented Jan 2, 2020

Travis tests have failed

1st Build

TravisBuddy Request Identifier: 92e94830-2d10-11ea-b5b0-b361e22ae1a5

TravisBuddy commented Jan 2, 2020

TravisBuddy Request Identifier: ac563a70-2d2a-11ea-b5b0-b361e22ae1a5

gaocegege left a comment

Choose a reason for hiding this comment

jinchihe commented Jan 2, 2020

johnugeorge commented Jan 2, 2020

k8s-ci-robot commented Jan 2, 2020

gaocegege commented Jan 2, 2020

TravisBuddy commented Jan 2, 2020

TravisBuddy Request Identifier: 609e38c0-2d35-11ea-b5b0-b361e22ae1a5

TravisBuddy commented Jan 2, 2020

TravisBuddy Request Identifier: 20adeff0-2d38-11ea-b5b0-b361e22ae1a5

TravisBuddy commented Jan 2, 2020

TravisBuddy Request Identifier: 29bd3400-2d3a-11ea-b5b0-b361e22ae1a5

TravisBuddy commented Jan 2, 2020

TravisBuddy Request Identifier: aec20520-2d3c-11ea-a185-6fb6d15019ed

gaocegege commented Jan 2, 2020

jinchihe commented Dec 31, 2019 •

edited

Loading

coveralls commented Dec 31, 2019 •

edited

Loading