Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add controller-name label for Pod and service #1067

Merged
merged 2 commits into from
Aug 16, 2019
Merged

Add controller-name label for Pod and service #1067

merged 2 commits into from
Aug 16, 2019

Conversation

hougangliu
Copy link
Member

@hougangliu hougangliu commented Aug 15, 2019 •
edited by jlewi
Loading

This change is Reviewable

@hougangliu
Copy link
Member Author

Fixes: #1066

@TravisBuddy
Copy link

Travis tests have failed

Hey @hougangliu,
Please read the following log in order to understand the failure reason.
It'll be awesome if you fix what's wrong and commit the changes.

1st Build

View build log

goveralls -service=travis-ci -v -package ./pkg/... -ignore "pkg/client/*/*.go,pkg/client/*/*/*.go,pkg/client/*/*/*/*.go,pkg/client/*/*/*/*/*.go,pkg/client/*/*/*/*/*/*.go,pkg/client/*/*/*/*/*/*/*.go,pkg/util/testutil/*.go,pkg/apis/tensorflow/*/zz_generated.*.go,pkg/apis/tensorflow/*/*_generated.go,pkg/apis/common/*/zz_generated.*.go,pkg/apis/common/*/*_generated.go" 
?   	github.com/kubeflow/tf-operator/pkg/apis/common/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2	[no test files]
=== RUN   TestSetTypeNames
--- PASS: TestSetTypeNames (0.00s)
=== RUN   TestSetDefaultTFJob
--- PASS: TestSetDefaultTFJob (0.00s)
=== RUN   TestIsChieforMaster
--- PASS: TestIsChieforMaster (0.00s)
PASS
coverage: 20.9% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1	0.030s	coverage: 20.9% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
=== RUN   TestSetTypeNames
--- PASS: TestSetTypeNames (0.00s)
=== RUN   TestSetDefaultTFJob
--- PASS: TestSetDefaultTFJob (0.00s)
=== RUN   TestIsChieforMaster
--- PASS: TestIsChieforMaster (0.00s)
PASS
coverage: 20.9% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2	0.033s	coverage: 20.9% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
=== RUN   TestValidateBetaTwoTFJobSpec
time="2019-08-15T07:22:55Z" level=error msg="TFJobSpec is not valid: Image is undefined in the container of Worker"
time="2019-08-15T07:22:55Z" level=error msg="TFJobSpec is not valid: There is no container named tensorflow in Worker"
--- PASS: TestValidateBetaTwoTFJobSpec (0.00s)
=== RUN   TestValidateV1TFJobSpec
time="2019-08-15T07:22:55Z" level=error msg="TFJobSpec is not valid: Image is undefined in the container of Worker"
time="2019-08-15T07:22:55Z" level=error msg="TFJobSpec is not valid: There is no container named tensorflow in Worker"
--- PASS: TestValidateV1TFJobSpec (0.00s)
PASS
coverage: 14.2% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation	0.030s	coverage: 14.2% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2	[no test files]
=== RUN   TestGenGeneralName
--- PASS: TestGenGeneralName (0.00s)
PASS
coverage: 0.4% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/common/jobcontroller	0.023s	coverage: 0.4% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured	[no test files]
=== RUN   TestCreatePods
--- PASS: TestCreatePods (0.01s)
=== RUN   TestCreateService
time="2019-08-15T07:23:13Z" level=info msg="Controller test-tfjob created service empty_service"
--- PASS: TestCreateService (0.00s)
=== RUN   TestCreateServicesWithControllerRef
time="2019-08-15T07:23:13Z" level=info msg="Controller test-tfjob created service empty_service"
--- PASS: TestCreateServicesWithControllerRef (0.00s)
=== RUN   TestClaimServices
--- PASS: TestClaimServices (0.00s)
PASS
coverage: 41.1% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/control	0.067s	coverage: 41.1% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
=== RUN   TestNormalPath
time="2019-08-15T07:23:21Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:23:21Z" level=info msg="Creating Job controller"
time="2019-08-15T07:23:21Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T07:23:21Z" level=info msg="Need to create new pod: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:23:21Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:23:21Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:23:21Z" level=info msg="need to create new service: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:23:21Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:23:21Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="Need to create new pod: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:23:21Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="need to create new service: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (3.577261ms)" job=default.test-tfjob
time="2019-08-15T07:23:21Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:23:21Z" level=info msg="Creating Job controller"
time="2019-08-15T07:23:21Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T07:23:21Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="Need to create new pod: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:23:21Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="need to create new service: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="Need to create new pod: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:23:21Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:23:21Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:23:21Z" level=info msg="need to create new service: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:23:21Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:23:21Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.863136ms)" job=default.test-tfjob
time="2019-08-15T07:23:21Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:23:21Z" level=info msg="Creating Job controller"
time="2019-08-15T07:23:21Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T07:23:21Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:23:21Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (294.729µs)" job=default.test-tfjob
time="2019-08-15T07:23:21Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:23:21Z" level=info msg="Creating Job controller"
time="2019-08-15T07:23:21Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T07:23:21Z" level=info msg="Need to create new pod: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:23:21Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:23:21Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:23:21Z" level=info msg="need to create new service: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:23:21Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:23:21Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="Need to create new pod: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:23:21Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="need to create new service: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (784.832µs)" job=default.test-tfjob
time="2019-08-15T07:23:21Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:23:21Z" level=info msg="Creating Job controller"
time="2019-08-15T07:23:21Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T07:23:21Z" level=info msg="Need to create new pod: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:23:21Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:23:21Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:23:21Z" level=info msg="need to create new service: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:23:21Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:23:21Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="Need to create new pod: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:23:21Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="need to create new service: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (710.325µs)" job=default.test-tfjob
time="2019-08-15T07:23:21Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:23:21Z" level=info msg="Creating Job controller"
time="2019-08-15T07:23:21Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T07:23:21Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="Need to create new pod: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:23:21Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="need to create new service: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="Need to create new pod: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:23:21Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:23:21Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:23:21Z" level=info msg="need to create new service: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:23:21Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:23:21Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.579686ms)" job=default.test-tfjob
time="2019-08-15T07:23:21Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:23:21Z" level=info msg="Creating Job controller"
time="2019-08-15T07:23:21Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T07:23:21Z" level=info msg="Need to create new pod: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:23:21Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:23:21Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:23:21Z" level=info msg="need to create new service: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:23:21Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:23:21Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="Need to create new pod: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:23:21Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="need to create new service: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.691131ms)" job=default.test-tfjob
time="2019-08-15T07:23:21Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:23:21Z" level=info msg="Creating Job controller"
time="2019-08-15T07:23:21Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T07:23:21Z" level=info msg="Need to create new pod: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:23:21Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:23:21Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:23:21Z" level=info msg="need to create new service: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:23:21Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:23:21Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="Need to create new pod: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:23:21Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="need to create new service: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.744868ms)" job=default.test-tfjob
--- FAIL: TestNormalPath (0.03s)
	controller_test.go:287: Distributed TFJob (4 workers, 2 PS) is created, 2 workers, 1 PS are pending, 1 worker is running: unexpected number of pod creates.  Expected 2, saw 6
	controller_test.go:290: Distributed TFJob (4 workers, 2 PS) is created, 2 workers, 1 PS are pending, 1 worker is running: unexpected number of service creates.  Expected 2, saw 6
	controller_test.go:297: Distributed TFJob (4 workers, 2 PS) is created, 2 workers, 1 PS are pending, 1 worker is running: unexpected number of ControllerRefs.  Expected 2, saw 6
	controller_test.go:320: Distributed TFJob (4 workers, 2 PS) is created, 2 workers, 1 PS are pending, 1 worker is running: unexpected number of active pods.  Expected 1, saw 0
	controller_test.go:359: Distributed TFJob (4 workers, 2 PS) is created, 2 workers, 1 PS are pending, 1 worker is running: expected condition "Running", got []v1.JobCondition(nil)
	controller_test.go:287: Distributed TFJob (4 workers, 2 PS) is succeeded: unexpected number of pod creates.  Expected 0, saw 6
	controller_test.go:290: Distributed TFJob (4 workers, 2 PS) is succeeded: unexpected number of service creates.  Expected 0, saw 6
	controller_test.go:297: Distributed TFJob (4 workers, 2 PS) is succeeded: unexpected number of ControllerRefs.  Expected 0, saw 6
	controller_test.go:325: Distributed TFJob (4 workers, 2 PS) is succeeded: unexpected number of succeeded pods.  Expected 4, saw 0
	controller_test.go:343: Distributed TFJob (4 workers, 2 PS) is succeeded: unexpected number of succeeded pods.  Expected 2, saw 0
	controller_test.go:359: Distributed TFJob (4 workers, 2 PS) is succeeded: expected condition "Succeeded", got []v1.JobCondition(nil)
	controller_test.go:287: Distributed TFJob (4 workers, 2 PS) is created and all replicas are pending: unexpected number of pod creates.  Expected 0, saw 6
	controller_test.go:290: Distributed TFJob (4 workers, 2 PS) is created and all replicas are pending: unexpected number of service creates.  Expected 0, saw 6
	controller_test.go:297: Distributed TFJob (4 workers, 2 PS) is created and all replicas are pending: unexpected number of ControllerRefs.  Expected 0, saw 6
	controller_test.go:287: Distributed TFJob (4 workers, 2 PS) is created, 2 workers, 1 PS are pending: unexpected number of pod creates.  Expected 3, saw 6
	controller_test.go:290: Distributed TFJob (4 workers, 2 PS) is created, 2 workers, 1 PS are pending: unexpected number of service creates.  Expected 3, saw 6
	controller_test.go:297: Distributed TFJob (4 workers, 2 PS) is created, 2 workers, 1 PS are pending: unexpected number of ControllerRefs.  Expected 3, saw 6
	controller_test.go:287: Distributed TFJob (4 workers, 2 PS) is created, 2 workers, 1 PS are pending, 1 worker is succeeded: unexpected number of pod creates.  Expected 2, saw 6
	controller_test.go:290: Distributed TFJob (4 workers, 2 PS) is created, 2 workers, 1 PS are pending, 1 worker is succeeded: unexpected number of service creates.  Expected 2, saw 6
	controller_test.go:297: Distributed TFJob (4 workers, 2 PS) is created, 2 workers, 1 PS are pending, 1 worker is succeeded: unexpected number of ControllerRefs.  Expected 2, saw 6
	controller_test.go:325: Distributed TFJob (4 workers, 2 PS) is created, 2 workers, 1 PS are pending, 1 worker is succeeded: unexpected number of succeeded pods.  Expected 1, saw 0
	controller_test.go:287: Distributed TFJob (4 workers, 2 PS) is created and all replicas are running: unexpected number of pod creates.  Expected 0, saw 6
	controller_test.go:290: Distributed TFJob (4 workers, 2 PS) is created and all replicas are running: unexpected number of service creates.  Expected 0, saw 6
	controller_test.go:297: Distributed TFJob (4 workers, 2 PS) is created and all replicas are running: unexpected number of ControllerRefs.  Expected 0, saw 6
	controller_test.go:320: Distributed TFJob (4 workers, 2 PS) is created and all replicas are running: unexpected number of active pods.  Expected 4, saw 0
	controller_test.go:338: Distributed TFJob (4 workers, 2 PS) is created and all replicas are running: unexpected number of active pods.  Expected 2, saw 0
	controller_test.go:359: Distributed TFJob (4 workers, 2 PS) is created and all replicas are running: expected condition "Running", got []v1.JobCondition(nil)
=== RUN   TestRun
time="2019-08-15T07:23:21Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:23:21Z" level=info msg="Creating Job controller"
time="2019-08-15T07:23:21Z" level=info msg="Starting TFJob controller"
time="2019-08-15T07:23:21Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-15T07:23:21Z" level=info msg="Starting 1 workers"
time="2019-08-15T07:23:21Z" level=info msg="Started workers"
time="2019-08-15T07:23:21Z" level=info msg="Shutting down workers"
--- PASS: TestRun (0.50s)
=== RUN   TestSyncPdb
time="2019-08-15T07:23:21Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:23:21Z" level=info msg="Creating Job controller"
--- FAIL: TestSyncPdb (0.00s)
	controller_test.go:472: Got {MinAvailable:1 Selector:&LabelSelector{MatchLabels:map[string]string{controller-name: tf-operator,job-name: test-sync-pdb,},MatchExpressions:[],} MaxUnavailable:<nil>}, want {MinAvailable:1 Selector:&LabelSelector{MatchLabels:map[string]string{job-name: test-sync-pdb,},MatchExpressions:[],} MaxUnavailable:<nil>}
=== RUN   TestAddTFJob
time="2019-08-15T07:23:21Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:23:21Z" level=info msg="Creating Job controller"
time="2019-08-15T07:23:21Z" level=info msg="TFJob test-tfjob is created." job=default.test-tfjob uid=
time="2019-08-15T07:23:21Z" level=info msg="Starting TFJob controller"
time="2019-08-15T07:23:21Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-15T07:23:21Z" level=info msg="Starting 1 workers"
time="2019-08-15T07:23:21Z" level=info msg="Started workers"
--- PASS: TestAddTFJob (0.10s)
=== RUN   TestCopyLabelsAndAnnotation
time="2019-08-15T07:23:21Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:23:21Z" level=info msg="Creating Job controller"
time="2019-08-15T07:23:21Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T07:23:21Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:23:21Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:21Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (432.31µs)" job=default.test-tfjob
--- PASS: TestCopyLabelsAndAnnotation (0.00s)
=== RUN   TestDeletePodsAndServices
time="2019-08-15T07:23:21Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:23:21Z" level=info msg="Creating Job controller"
time="2019-08-15T07:23:21Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T07:23:21Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (351.565µs)" job=default.test-tfjob
time="2019-08-15T07:23:21Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:23:21Z" level=info msg="Creating Job controller"
time="2019-08-15T07:23:21Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T07:23:21Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (403.604µs)" job=default.test-tfjob
time="2019-08-15T07:23:21Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:23:21Z" level=info msg="Creating Job controller"
time="2019-08-15T07:23:21Z" level=info msg="Shutting down workers"
time="2019-08-15T07:23:21Z" level=info msg="Starting TFJob controller"
time="2019-08-15T07:23:21Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-15T07:23:21Z" level=info msg="Starting 1 workers"
time="2019-08-15T07:23:21Z" level=info msg="Started workers"
time="2019-08-15T07:23:21Z" level=info msg="Shutting down workers"
time="2019-08-15T07:23:21Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T07:23:21Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (904.3µs)" job=default.test-tfjob
time="2019-08-15T07:23:21Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:23:21Z" level=info msg="Creating Job controller"
time="2019-08-15T07:23:21Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T07:23:21Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (319.866µs)" job=default.test-tfjob
--- FAIL: TestDeletePodsAndServices (0.01s)
	job_test.go:367: 4 workers and 2 ps is running, policy is all: unexpected number of pod deletes.  Expected 6, saw 0
	job_test.go:370: 4 workers and 2 ps is running, policy is all: unexpected number of service deletes.  Expected 6, saw 0
	job_test.go:367: 4 workers and 2 ps is running, policy is running: unexpected number of pod deletes.  Expected 6, saw 0
	job_test.go:370: 4 workers and 2 ps is running, policy is running: unexpected number of service deletes.  Expected 6, saw 0
=== RUN   TestCleanupTFJob
time="2019-08-15T07:23:21Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:23:21Z" level=info msg="Creating Job controller"
time="2019-08-15T07:23:21Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T07:23:21Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (241.656µs)" job=default.test-tfjob
time="2019-08-15T07:23:21Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:23:21Z" level=info msg="Creating Job controller"
time="2019-08-15T07:23:21Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T07:23:21Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (233.247µs)" job=default.test-tfjob
time="2019-08-15T07:23:21Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:23:21Z" level=info msg="Creating Job controller"
time="2019-08-15T07:23:23Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T07:23:23Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (601.874µs)" job=default.test-tfjob
--- PASS: TestCleanupTFJob (2.00s)
=== RUN   TestActiveDeadlineSeconds
time="2019-08-15T07:23:23Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:23:23Z" level=info msg="Creating Job controller"
time="2019-08-15T07:23:23Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T07:23:23Z" level=info msg="Need to create new pod: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:23:23Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:23:23Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:23:23Z" level=info msg="need to create new service: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:23:23Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:23:23Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:23Z" level=info msg="Need to create new pod: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:23Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:23Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:23Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:23:23Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:23Z" level=info msg="need to create new service: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:23Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:23Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:23Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:23:23Z" level=info msg="Creating Job controller"
time="2019-08-15T07:23:25Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
--- FAIL: TestActiveDeadlineSeconds (2.01s)
	job_test.go:685: 4 workers and 2 ps is running, ActiveDeadlineSeconds is 2: unexpected number of pod deletes.  Expected 6, saw 0
	job_test.go:688: 4 workers and 2 ps is running, ActiveDeadlineSeconds is 2: unexpected number of service deletes.  Expected 6, saw 0
=== RUN   TestBackoffForOnFailure
time="2019-08-15T07:23:25Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:23:25Z" level=info msg="Creating Job controller"
time="2019-08-15T07:23:25Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T07:23:25Z" level=warning msg="The restart policy of replica PS of the job test-tfjob is not OnFailure or Always. Not counted in backoff limit." job=default.test-tfjob uid=
time="2019-08-15T07:23:25Z" level=info msg="Need to create new pod: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:23:25Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:23:25Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:23:25Z" level=info msg="need to create new service: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:23:25Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:23:25Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:25Z" level=info msg="Need to create new pod: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:25Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:25Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:25Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:23:25Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:25Z" level=info msg="need to create new service: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:25Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:25Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:25Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.324366ms)" job=default.test-tfjob
--- FAIL: TestBackoffForOnFailure (0.00s)
	job_test.go:806: 4 workers each having 1 restartCount and 2 ps is running, backoffLimit 4 : unexpected number of pod deletes.  Expected 6, saw 0
	job_test.go:809: 4 workers each having 1 restartCount and 2 ps is running, backoffLimit 4 : unexpected number of service deletes.  Expected 6, saw 0
=== RUN   TestAddPod
time="2019-08-15T07:23:25Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:23:25Z" level=info msg="Creating Job controller"
time="2019-08-15T07:23:25Z" level=info msg="Starting TFJob controller"
time="2019-08-15T07:23:25Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-15T07:23:25Z" level=info msg="Starting 1 workers"
time="2019-08-15T07:23:25Z" level=info msg="Started workers"
--- PASS: TestAddPod (0.10s)
=== RUN   TestClusterSpec
time="2019-08-15T07:23:25Z" level=info msg="Shutting down workers"
--- PASS: TestClusterSpec (0.00s)
=== RUN   TestRestartPolicy
--- PASS: TestRestartPolicy (0.00s)
=== RUN   TestExitCode
time="2019-08-15T07:23:25Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:23:25Z" level=info msg="Creating Job controller"
time="2019-08-15T07:23:25Z" level=info msg="Starting TFJob controller"
time="2019-08-15T07:23:25Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T07:23:25Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-15T07:23:25Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:25Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:23:25Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:23:25Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.153763ms)" job=default.test-tfjob
--- FAIL: TestExitCode (0.00s)
time="2019-08-15T07:23:25Z" level=info msg="Starting 1 workers"
	pod_test.go:297: Failed to delete pod worker-0
time="2019-08-15T07:23:25Z" level=info msg="Started workers"
time="2019-08-15T07:23:25Z" level=info msg="Shutting down workers"
=== RUN   TestAddService
time="2019-08-15T07:23:25Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:23:25Z" level=info msg="Creating Job controller"
time="2019-08-15T07:23:25Z" level=info msg="Starting TFJob controller"
time="2019-08-15T07:23:25Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-15T07:23:25Z" level=info msg="Starting 1 workers"
time="2019-08-15T07:23:25Z" level=info msg="Started workers"
--- PASS: TestAddService (0.10s)
time="2019-08-15T07:23:25Z" level=info msg="Shutting down workers"
=== RUN   TestFailed
time="2019-08-15T07:23:25Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:23:25Z" level=info msg="Creating Job controller"
time="2019-08-15T07:23:25Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=0, failed=1" job=default.test-tfjob uid=
E0815 07:23:25.900997    9653 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(nil), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"Worker":(*v1.ReplicaSpec)(0xc420a10b00)}}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"Worker":(*v1.ReplicaStatus)(0xc4204281f0)}, StartTime:(*v1.Time)(0xc420425000), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'TFJobFailed' 'TFJob test-tfjob has failed because 1 Worker replica(s) failed.'
--- PASS: TestFailed (0.00s)
=== RUN   TestStatus
time="2019-08-15T07:23:25Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:23:25Z" level=info msg="Creating Job controller"
time="2019-08-15T07:23:25Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:23:25Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:23:25Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:23:25Z" level=info msg="Creating Job controller"
time="2019-08-15T07:23:25Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:23:25Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:23:25Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:23:25Z" level=info msg="Creating Job controller"
time="2019-08-15T07:23:25Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-15T07:23:25Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:23:25Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:23:25Z" level=info msg="Creating Job controller"
time="2019-08-15T07:23:25Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-15T07:23:25Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:23:25Z" level=info msg="Creating Job controller"
time="2019-08-15T07:23:25Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:23:25Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:23:25Z" level=info msg="Creating Job controller"
time="2019-08-15T07:23:25Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:23:25Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:23:25Z" level=info msg="Creating Job controller"
time="2019-08-15T07:23:25Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:23:25Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:23:25Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:23:25Z" level=info msg="Creating Job controller"
time="2019-08-15T07:23:25Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=2, failed=2" job=default.test-tfjob uid=
time="2019-08-15T07:23:25Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:23:25Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:23:25Z" level=info msg="Creating Job controller"
time="2019-08-15T07:23:25Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=2, running=0, failed=2" job=default.test-tfjob uid=
time="2019-08-15T07:23:25Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:23:25Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:23:25Z" level=info msg="Creating Job controller"
time="2019-08-15T07:23:25Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=3, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:23:25Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:23:25Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:23:25Z" level=info msg="Creating Job controller"
time="2019-08-15T07:23:25Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:23:25Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2019-08-15T07:23:25Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:23:25Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:23:25Z" level=info msg="Creating Job controller"
time="2019-08-15T07:23:25Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:23:25Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:23:25Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:23:25Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:23:25Z" level=info msg="Creating Job controller"
time="2019-08-15T07:23:25Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:23:25Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:23:25Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=1, failed=1" job=default.test-tfjob uid=
time="2019-08-15T07:23:25Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:23:25Z" level=info msg="Creating Job controller"
time="2019-08-15T07:23:25Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-15T07:23:25Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:23:25Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:23:25Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:23:25Z" level=info msg="Creating Job controller"
time="2019-08-15T07:23:25Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:23:25Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2019-08-15T07:23:25Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:23:25Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:23:25Z" level=info msg="Creating Job controller"
time="2019-08-15T07:23:25Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-15T07:23:25Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2019-08-15T07:23:25Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
--- PASS: TestStatus (0.02s)
=== RUN   TestGenOwnerReference
--- PASS: TestGenOwnerReference (0.00s)
=== RUN   TestGenLabels
--- PASS: TestGenLabels (0.00s)
=== RUN   TestConvertTFJobToUnstructured
--- PASS: TestConvertTFJobToUnstructured (0.00s)
FAIL
coverage: 41.0% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
FAIL	github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow	4.938s
exit status 1: warning: no packages being tested depend on matches for pattern github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake
warning: no packages being tested depend on matches for pattern github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake
warning: no packages being tested depend on matches for pattern github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake
warning: no packages being tested depend on matches for pattern github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil
warning: no packages being tested depend on matches for pattern github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil
warning: no packages being tested depend on matches for pattern github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured
warning: no packages being tested depend on matches for pattern github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow
warning: no packages being tested depend on matches for pattern github.com/kubeflow/tf-operator/pkg/util
warning: no packages being tested depend on matches for pattern github.com/kubeflow/tf-operator/pkg/util/signals
warning: no packages being tested depend on matches for pattern github.com/kubeflow/tf-operator/pkg/version
TravisBuddy Request Identifier: 9279cc40-bf2d-11e9-b293-f57b21f1c1f9

gaocegege
gaocegege reviewed Aug 15, 2019
View reviewed changes
Copy link
Member

@gaocegege gaocegege left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@gaocegege
Copy link
Member

Please fix the test cases

https://travis-ci.org/kubeflow/tf-operator/builds/572182491?utm_source=github_status&utm_medium=notification

@TravisBuddy
Copy link

Travis tests have failed

Hey @hougangliu,
Please read the following log in order to understand the failure reason.
It'll be awesome if you fix what's wrong and commit the changes.

1st Build

View build log

goveralls -service=travis-ci -v -package ./pkg/... -ignore "pkg/client/*/*.go,pkg/client/*/*/*.go,pkg/client/*/*/*/*.go,pkg/client/*/*/*/*/*.go,pkg/client/*/*/*/*/*/*.go,pkg/client/*/*/*/*/*/*/*.go,pkg/util/testutil/*.go,pkg/apis/tensorflow/*/zz_generated.*.go,pkg/apis/tensorflow/*/*_generated.go,pkg/apis/common/*/zz_generated.*.go,pkg/apis/common/*/*_generated.go" 
?   	github.com/kubeflow/tf-operator/pkg/apis/common/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2	[no test files]
=== RUN   TestSetTypeNames
--- PASS: TestSetTypeNames (0.00s)
=== RUN   TestSetDefaultTFJob
--- PASS: TestSetDefaultTFJob (0.00s)
=== RUN   TestIsChieforMaster
--- PASS: TestIsChieforMaster (0.00s)
PASS
coverage: 20.9% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1	0.033s	coverage: 20.9% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
=== RUN   TestSetTypeNames
--- PASS: TestSetTypeNames (0.00s)
=== RUN   TestSetDefaultTFJob
--- PASS: TestSetDefaultTFJob (0.00s)
=== RUN   TestIsChieforMaster
--- PASS: TestIsChieforMaster (0.00s)
PASS
coverage: 20.9% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2	0.032s	coverage: 20.9% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
=== RUN   TestValidateBetaTwoTFJobSpec
time="2019-08-15T07:39:28Z" level=error msg="TFJobSpec is not valid: Image is undefined in the container of Worker"
time="2019-08-15T07:39:28Z" level=error msg="TFJobSpec is not valid: There is no container named tensorflow in Worker"
--- PASS: TestValidateBetaTwoTFJobSpec (0.00s)
=== RUN   TestValidateV1TFJobSpec
time="2019-08-15T07:39:28Z" level=error msg="TFJobSpec is not valid: Image is undefined in the container of Worker"
time="2019-08-15T07:39:28Z" level=error msg="TFJobSpec is not valid: There is no container named tensorflow in Worker"
--- PASS: TestValidateV1TFJobSpec (0.00s)
PASS
coverage: 14.2% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation	0.031s	coverage: 14.2% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2	[no test files]
=== RUN   TestGenGeneralName
--- PASS: TestGenGeneralName (0.00s)
PASS
coverage: 0.4% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/common/jobcontroller	0.022s	coverage: 0.4% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured	[no test files]
=== RUN   TestCreatePods
--- PASS: TestCreatePods (0.01s)
=== RUN   TestCreateService
time="2019-08-15T07:39:47Z" level=info msg="Controller test-tfjob created service empty_service"
--- PASS: TestCreateService (0.00s)
=== RUN   TestCreateServicesWithControllerRef
time="2019-08-15T07:39:47Z" level=info msg="Controller test-tfjob created service empty_service"
--- PASS: TestCreateServicesWithControllerRef (0.00s)
=== RUN   TestClaimServices
--- PASS: TestClaimServices (0.00s)
PASS
coverage: 41.1% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/control	0.063s	coverage: 41.1% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
=== RUN   TestNormalPath
time="2019-08-15T07:39:54Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:39:54Z" level=info msg="Creating Job controller"
time="2019-08-15T07:39:54Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T07:39:54Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="Need to create new pod: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:39:54Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="need to create new service: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="Need to create new pod: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:39:54Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:39:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:39:54Z" level=info msg="need to create new service: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:39:54Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:39:54Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (2.927047ms)" job=default.test-tfjob
time="2019-08-15T07:39:54Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:39:54Z" level=info msg="Creating Job controller"
time="2019-08-15T07:39:54Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T07:39:54Z" level=info msg="Need to create new pod: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:39:54Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:39:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:39:54Z" level=info msg="need to create new service: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:39:54Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:39:54Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="Need to create new pod: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:39:54Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="need to create new service: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (2.159411ms)" job=default.test-tfjob
time="2019-08-15T07:39:54Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:39:54Z" level=info msg="Creating Job controller"
time="2019-08-15T07:39:54Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T07:39:54Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:39:54Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (654.693µs)" job=default.test-tfjob
time="2019-08-15T07:39:54Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:39:54Z" level=info msg="Creating Job controller"
time="2019-08-15T07:39:54Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T07:39:54Z" level=info msg="Need to create new pod: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:39:54Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:39:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:39:54Z" level=info msg="need to create new service: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:39:54Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:39:54Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="Need to create new pod: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:39:54Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="need to create new service: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.73876ms)" job=default.test-tfjob
time="2019-08-15T07:39:54Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:39:54Z" level=info msg="Creating Job controller"
time="2019-08-15T07:39:54Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T07:39:54Z" level=info msg="Need to create new pod: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:39:54Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:39:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:39:54Z" level=info msg="need to create new service: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:39:54Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:39:54Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="Need to create new pod: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:39:54Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="need to create new service: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.529112ms)" job=default.test-tfjob
time="2019-08-15T07:39:54Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:39:54Z" level=info msg="Creating Job controller"
time="2019-08-15T07:39:54Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T07:39:54Z" level=info msg="Need to create new pod: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:39:54Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:39:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:39:54Z" level=info msg="need to create new service: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:39:54Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:39:54Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="Need to create new pod: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:39:54Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="need to create new service: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (3.295746ms)" job=default.test-tfjob
time="2019-08-15T07:39:54Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:39:54Z" level=info msg="Creating Job controller"
time="2019-08-15T07:39:54Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T07:39:54Z" level=info msg="Need to create new pod: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:39:54Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:39:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:39:54Z" level=info msg="need to create new service: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:39:54Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:39:54Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="Need to create new pod: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:39:54Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="need to create new service: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.688803ms)" job=default.test-tfjob
time="2019-08-15T07:39:54Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:39:54Z" level=info msg="Creating Job controller"
time="2019-08-15T07:39:54Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T07:39:54Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="Need to create new pod: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:39:54Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="need to create new service: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:54Z" level=info msg="Need to create new pod: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:39:54Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:39:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:39:54Z" level=info msg="need to create new service: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:39:54Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:39:54Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (842.455µs)" job=default.test-tfjob
--- FAIL: TestNormalPath (0.04s)
	controller_test.go:287: Distributed TFJob (4 workers, 2 PS) is created, 2 workers, 1 PS are pending, 1 worker is running: unexpected number of pod creates.  Expected 2, saw 6
	controller_test.go:290: Distributed TFJob (4 workers, 2 PS) is created, 2 workers, 1 PS are pending, 1 worker is running: unexpected number of service creates.  Expected 2, saw 6
	controller_test.go:297: Distributed TFJob (4 workers, 2 PS) is created, 2 workers, 1 PS are pending, 1 worker is running: unexpected number of ControllerRefs.  Expected 2, saw 6
	controller_test.go:320: Distributed TFJob (4 workers, 2 PS) is created, 2 workers, 1 PS are pending, 1 worker is running: unexpected number of active pods.  Expected 1, saw 0
	controller_test.go:359: Distributed TFJob (4 workers, 2 PS) is created, 2 workers, 1 PS are pending, 1 worker is running: expected condition "Running", got []v1.JobCondition(nil)
	controller_test.go:287: Distributed TFJob (4 workers, 2 PS) is succeeded: unexpected number of pod creates.  Expected 0, saw 6
	controller_test.go:290: Distributed TFJob (4 workers, 2 PS) is succeeded: unexpected number of service creates.  Expected 0, saw 6
	controller_test.go:297: Distributed TFJob (4 workers, 2 PS) is succeeded: unexpected number of ControllerRefs.  Expected 0, saw 6
	controller_test.go:325: Distributed TFJob (4 workers, 2 PS) is succeeded: unexpected number of succeeded pods.  Expected 4, saw 0
	controller_test.go:343: Distributed TFJob (4 workers, 2 PS) is succeeded: unexpected number of succeeded pods.  Expected 2, saw 0
	controller_test.go:359: Distributed TFJob (4 workers, 2 PS) is succeeded: expected condition "Succeeded", got []v1.JobCondition(nil)
	controller_test.go:287: Distributed TFJob (4 workers, 2 PS) is created and all replicas are pending: unexpected number of pod creates.  Expected 0, saw 6
	controller_test.go:290: Distributed TFJob (4 workers, 2 PS) is created and all replicas are pending: unexpected number of service creates.  Expected 0, saw 6
	controller_test.go:297: Distributed TFJob (4 workers, 2 PS) is created and all replicas are pending: unexpected number of ControllerRefs.  Expected 0, saw 6
	controller_test.go:287: Distributed TFJob (4 workers, 2 PS) is created, 2 workers, 1 PS are pending: unexpected number of pod creates.  Expected 3, saw 6
	controller_test.go:290: Distributed TFJob (4 workers, 2 PS) is created, 2 workers, 1 PS are pending: unexpected number of service creates.  Expected 3, saw 6
	controller_test.go:297: Distributed TFJob (4 workers, 2 PS) is created, 2 workers, 1 PS are pending: unexpected number of ControllerRefs.  Expected 3, saw 6
	controller_test.go:287: Distributed TFJob (4 workers, 2 PS) is created, 2 workers, 1 PS are pending, 1 worker is succeeded: unexpected number of pod creates.  Expected 2, saw 6
	controller_test.go:290: Distributed TFJob (4 workers, 2 PS) is created, 2 workers, 1 PS are pending, 1 worker is succeeded: unexpected number of service creates.  Expected 2, saw 6
	controller_test.go:297: Distributed TFJob (4 workers, 2 PS) is created, 2 workers, 1 PS are pending, 1 worker is succeeded: unexpected number of ControllerRefs.  Expected 2, saw 6
	controller_test.go:325: Distributed TFJob (4 workers, 2 PS) is created, 2 workers, 1 PS are pending, 1 worker is succeeded: unexpected number of succeeded pods.  Expected 1, saw 0
	controller_test.go:287: Distributed TFJob (4 workers, 2 PS) is created and all replicas are running: unexpected number of pod creates.  Expected 0, saw 6
	controller_test.go:290: Distributed TFJob (4 workers, 2 PS) is created and all replicas are running: unexpected number of service creates.  Expected 0, saw 6
	controller_test.go:297: Distributed TFJob (4 workers, 2 PS) is created and all replicas are running: unexpected number of ControllerRefs.  Expected 0, saw 6
	controller_test.go:320: Distributed TFJob (4 workers, 2 PS) is created and all replicas are running: unexpected number of active pods.  Expected 4, saw 0
	controller_test.go:338: Distributed TFJob (4 workers, 2 PS) is created and all replicas are running: unexpected number of active pods.  Expected 2, saw 0
	controller_test.go:359: Distributed TFJob (4 workers, 2 PS) is created and all replicas are running: expected condition "Running", got []v1.JobCondition(nil)
=== RUN   TestRun
time="2019-08-15T07:39:54Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:39:54Z" level=info msg="Creating Job controller"
time="2019-08-15T07:39:54Z" level=info msg="Starting TFJob controller"
time="2019-08-15T07:39:54Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-15T07:39:54Z" level=info msg="Starting 1 workers"
time="2019-08-15T07:39:54Z" level=info msg="Started workers"
time="2019-08-15T07:39:54Z" level=info msg="Shutting down workers"
--- PASS: TestRun (0.50s)
=== RUN   TestSyncPdb
time="2019-08-15T07:39:54Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:39:54Z" level=info msg="Creating Job controller"
--- FAIL: TestSyncPdb (0.00s)
	controller_test.go:472: Got {MinAvailable:1 Selector:&LabelSelector{MatchLabels:map[string]string{controller-name: tf-operator,job-name: test-sync-pdb,},MatchExpressions:[],} MaxUnavailable:<nil>}, want {MinAvailable:1 Selector:&LabelSelector{MatchLabels:map[string]string{job-name: test-sync-pdb,},MatchExpressions:[],} MaxUnavailable:<nil>}
=== RUN   TestAddTFJob
time="2019-08-15T07:39:54Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:39:54Z" level=info msg="Creating Job controller"
time="2019-08-15T07:39:54Z" level=info msg="Starting TFJob controller"
time="2019-08-15T07:39:54Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-15T07:39:54Z" level=info msg="TFJob test-tfjob is created." job=default.test-tfjob uid=
time="2019-08-15T07:39:55Z" level=info msg="Starting 1 workers"
time="2019-08-15T07:39:55Z" level=info msg="Started workers"
--- PASS: TestAddTFJob (0.10s)
=== RUN   TestCopyLabelsAndAnnotation
time="2019-08-15T07:39:55Z" level=info msg="Shutting down workers"
time="2019-08-15T07:39:55Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:39:55Z" level=info msg="Creating Job controller"
time="2019-08-15T07:39:55Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T07:39:55Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:55Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:39:55Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:55Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (622.032µs)" job=default.test-tfjob
--- PASS: TestCopyLabelsAndAnnotation (0.00s)
=== RUN   TestDeletePodsAndServices
time="2019-08-15T07:39:55Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:39:55Z" level=info msg="Creating Job controller"
time="2019-08-15T07:39:55Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T07:39:55Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (362.901µs)" job=default.test-tfjob
time="2019-08-15T07:39:55Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:39:55Z" level=info msg="Creating Job controller"
time="2019-08-15T07:39:55Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T07:39:55Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (493.714µs)" job=default.test-tfjob
time="2019-08-15T07:39:55Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:39:55Z" level=info msg="Creating Job controller"
time="2019-08-15T07:39:55Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T07:39:55Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (543.315µs)" job=default.test-tfjob
time="2019-08-15T07:39:55Z" level=info msg="Starting TFJob controller"
time="2019-08-15T07:39:55Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-15T07:39:55Z" level=info msg="Starting 1 workers"
time="2019-08-15T07:39:55Z" level=info msg="Started workers"
time="2019-08-15T07:39:55Z" level=info msg="Shutting down workers"
time="2019-08-15T07:39:55Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:39:55Z" level=info msg="Creating Job controller"
time="2019-08-15T07:39:55Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T07:39:55Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (275.617µs)" job=default.test-tfjob
--- FAIL: TestDeletePodsAndServices (0.01s)
	job_test.go:367: 4 workers and 2 ps is running, policy is all: unexpected number of pod deletes.  Expected 6, saw 0
	job_test.go:370: 4 workers and 2 ps is running, policy is all: unexpected number of service deletes.  Expected 6, saw 0
	job_test.go:367: 4 workers and 2 ps is running, policy is running: unexpected number of pod deletes.  Expected 6, saw 0
	job_test.go:370: 4 workers and 2 ps is running, policy is running: unexpected number of service deletes.  Expected 6, saw 0
=== RUN   TestCleanupTFJob
time="2019-08-15T07:39:55Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:39:55Z" level=info msg="Creating Job controller"
time="2019-08-15T07:39:55Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T07:39:55Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (527.274µs)" job=default.test-tfjob
time="2019-08-15T07:39:55Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:39:55Z" level=info msg="Creating Job controller"
time="2019-08-15T07:39:55Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T07:39:55Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (447.235µs)" job=default.test-tfjob
time="2019-08-15T07:39:55Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:39:55Z" level=info msg="Creating Job controller"
time="2019-08-15T07:39:57Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T07:39:57Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (683.743µs)" job=default.test-tfjob
--- PASS: TestCleanupTFJob (2.01s)
=== RUN   TestActiveDeadlineSeconds
time="2019-08-15T07:39:57Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:39:57Z" level=info msg="Creating Job controller"
time="2019-08-15T07:39:57Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T07:39:57Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:57Z" level=info msg="Need to create new pod: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:57Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:57Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:57Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:39:57Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:57Z" level=info msg="need to create new service: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:57Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:57Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:57Z" level=info msg="Need to create new pod: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:39:57Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:39:57Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:39:57Z" level=info msg="need to create new service: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:39:57Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:39:57Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:39:57Z" level=info msg="Creating Job controller"
time="2019-08-15T07:39:59Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
--- FAIL: TestActiveDeadlineSeconds (2.00s)
	job_test.go:685: 4 workers and 2 ps is running, ActiveDeadlineSeconds is 2: unexpected number of pod deletes.  Expected 6, saw 0
	job_test.go:688: 4 workers and 2 ps is running, ActiveDeadlineSeconds is 2: unexpected number of service deletes.  Expected 6, saw 0
=== RUN   TestBackoffForOnFailure
time="2019-08-15T07:39:59Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:39:59Z" level=info msg="Creating Job controller"
time="2019-08-15T07:39:59Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T07:39:59Z" level=warning msg="The restart policy of replica PS of the job test-tfjob is not OnFailure or Always. Not counted in backoff limit." job=default.test-tfjob uid=
time="2019-08-15T07:39:59Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:59Z" level=info msg="Need to create new pod: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:59Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:59Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:59Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:39:59Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:59Z" level=info msg="need to create new service: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:59Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:59Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T07:39:59Z" level=info msg="Need to create new pod: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:39:59Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:39:59Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:39:59Z" level=info msg="need to create new service: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:39:59Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T07:39:59Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.091403ms)" job=default.test-tfjob
--- FAIL: TestBackoffForOnFailure (0.00s)
	job_test.go:806: 4 workers each having 1 restartCount and 2 ps is running, backoffLimit 4 : unexpected number of pod deletes.  Expected 6, saw 0
	job_test.go:809: 4 workers each having 1 restartCount and 2 ps is running, backoffLimit 4 : unexpected number of service deletes.  Expected 6, saw 0
=== RUN   TestAddPod
time="2019-08-15T07:39:59Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:39:59Z" level=info msg="Creating Job controller"
time="2019-08-15T07:39:59Z" level=info msg="Starting TFJob controller"
time="2019-08-15T07:39:59Z" level=info msg="Waiting for informer caches to sync"
nfo msg="Started workers"
time="2019-08-15T07:39:59Z" level=info msg="Shutting down workers"
time="2019-08-15T07:39:59Z" level=info msg="Starting TFJob controller"
time="2019-08-15T07:39:59Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-15T07:39:59Z" level=info msg="Starting 1 workers"
time="2019-08-15T07:39:59Z" level=info msg="Started workers"
--- PASS: TestAddService (0.10s)
=== RUN   TestFailed
time="2019-08-15T07:39:59Z" level=info msg="Shutting down workers"
time="2019-08-15T07:39:59Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:39:59Z" level=info msg="Creating Job controller"
time="2019-08-15T07:39:59Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=0, failed=1" job=default.test-tfjob uid=
E0815 07:39:59.312469    9530 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(nil), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"Worker":(*v1.ReplicaSpec)(0xc42092f600)}}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"Worker":(*v1.ReplicaStatus)(0xc420376310)}, StartTime:(*v1.Time)(0xc42000c960), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'TFJobFailed' 'TFJob test-tfjob has failed because 1 Worker replica(s) failed.'
--- PASS: TestFailed (0.00s)
=== RUN   TestStatus
time="2019-08-15T07:39:59Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:39:59Z" level=info msg="Creating Job controller"
time="2019-08-15T07:39:59Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:39:59Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:39:59Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:39:59Z" level=info msg="Creating Job controller"
time="2019-08-15T07:39:59Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:39:59Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:39:59Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:39:59Z" level=info msg="Creating Job controller"
time="2019-08-15T07:39:59Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-15T07:39:59Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:39:59Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:39:59Z" level=info msg="Creating Job controller"
time="2019-08-15T07:39:59Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-15T07:39:59Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:39:59Z" level=info msg="Creating Job controller"
time="2019-08-15T07:39:59Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:39:59Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:39:59Z" level=info msg="Creating Job controller"
time="2019-08-15T07:39:59Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:39:59Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:39:59Z" level=info msg="Creating Job controller"
time="2019-08-15T07:39:59Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:39:59Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:39:59Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:39:59Z" level=info msg="Creating Job controller"
time="2019-08-15T07:39:59Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=2, failed=2" job=default.test-tfjob uid=
time="2019-08-15T07:39:59Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:39:59Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:39:59Z" level=info msg="Creating Job controller"
time="2019-08-15T07:39:59Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=2, running=0, failed=2" job=default.test-tfjob uid=
time="2019-08-15T07:39:59Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:39:59Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:39:59Z" level=info msg="Creating Job controller"
time="2019-08-15T07:39:59Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=3, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:39:59Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:39:59Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:39:59Z" level=info msg="Creating Job controller"
time="2019-08-15T07:39:59Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:39:59Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2019-08-15T07:39:59Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:39:59Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:39:59Z" level=info msg="Creating Job controller"
time="2019-08-15T07:39:59Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:39:59Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:39:59Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:39:59Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:39:59Z" level=info msg="Creating Job controller"
time="2019-08-15T07:39:59Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:39:59Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:39:59Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=1, failed=1" job=default.test-tfjob uid=
time="2019-08-15T07:39:59Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:39:59Z" level=info msg="Creating Job controller"
time="2019-08-15T07:39:59Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-15T07:39:59Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:39:59Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:39:59Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:39:59Z" level=info msg="Creating Job controller"
time="2019-08-15T07:39:59Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:39:59Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2019-08-15T07:39:59Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T07:39:59Z" level=info msg="Creating TFJob controller"
time="2019-08-15T07:39:59Z" level=info msg="Creating Job controller"
time="2019-08-15T07:39:59Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-15T07:39:59Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2019-08-15T07:39:59Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
--- PASS: TestStatus (0.02s)
=== RUN   TestGenOwnerReference
--- PASS: TestGenOwnerReference (0.00s)
=== RUN   TestGenLabels
--- PASS: TestGenLabels (0.00s)
=== RUN   TestConvertTFJobToUnstructured
--- PASS: TestConvertTFJobToUnstructured (0.00s)
FAIL
coverage: 41.0% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
FAIL	github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow	4.946s
exit status 1: warning: no packages being tested depend on matches for pattern github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake
warning: no packages being tested depend on matches for pattern github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake
warning: no packages being tested depend on matches for pattern github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake
warning: no packages being tested depend on matches for pattern github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil
warning: no packages being tested depend on matches for pattern github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil
warning: no packages being tested depend on matches for pattern github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured
warning: no packages being tested depend on matches for pattern github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow
warning: no packages being tested depend on matches for pattern github.com/kubeflow/tf-operator/pkg/util
warning: no packages being tested depend on matches for pattern github.com/kubeflow/tf-operator/pkg/util/signals
warning: no packages being tested depend on matches for pattern github.com/kubeflow/tf-operator/pkg/version
TravisBuddy Request Identifier: e2b01730-bf2f-11e9-b293-f57b21f1c1f9

@TravisBuddy
Copy link

Travis tests have failed

Hey @hougangliu,
Please read the following log in order to understand the failure reason.
It'll be awesome if you fix what's wrong and commit the changes.

1st Build

View build log

gometalinter --config=linter_config.json --vendor ./... 
pkg/controller.v1/tensorflow/pod_test.go:1::warning: file is not goimported (goimports)
pkg/controller.v1beta2/tensorflow/pod_test.go:1::warning: file is not goimported (goimports)
goveralls -service=travis-ci -v -package ./pkg/... -ignore "pkg/client/*/*.go,pkg/client/*/*/*.go,pkg/client/*/*/*/*.go,pkg/client/*/*/*/*/*.go,pkg/client/*/*/*/*/*/*.go,pkg/client/*/*/*/*/*/*/*.go,pkg/util/testutil/*.go,pkg/apis/tensorflow/*/zz_generated.*.go,pkg/apis/tensorflow/*/*_generated.go,pkg/apis/common/*/zz_generated.*.go,pkg/apis/common/*/*_generated.go" 
?   	github.com/kubeflow/tf-operator/pkg/apis/common/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2	[no test files]
=== RUN   TestSetTypeNames
--- PASS: TestSetTypeNames (0.00s)
=== RUN   TestSetDefaultTFJob
--- PASS: TestSetDefaultTFJob (0.00s)
=== RUN   TestIsChieforMaster
--- PASS: TestIsChieforMaster (0.00s)
PASS
coverage: 20.9% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1	0.031s	coverage: 20.9% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
=== RUN   TestSetTypeNames
--- PASS: TestSetTypeNames (0.00s)
=== RUN   TestSetDefaultTFJob
--- PASS: TestSetDefaultTFJob (0.00s)
=== RUN   TestIsChieforMaster
--- PASS: TestIsChieforMaster (0.00s)
PASS
coverage: 20.9% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2	0.030s	coverage: 20.9% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
=== RUN   TestValidateBetaTwoTFJobSpec
time="2019-08-15T15:08:20Z" level=error msg="TFJobSpec is not valid: Image is undefined in the container of Worker"
time="2019-08-15T15:08:20Z" level=error msg="TFJobSpec is not valid: There is no container named tensorflow in Worker"
--- PASS: TestValidateBetaTwoTFJobSpec (0.00s)
=== RUN   TestValidateV1TFJobSpec
time="2019-08-15T15:08:20Z" level=error msg="TFJobSpec is not valid: Image is undefined in the container of Worker"
time="2019-08-15T15:08:20Z" level=error msg="TFJobSpec is not valid: There is no container named tensorflow in Worker"
--- PASS: TestValidateV1TFJobSpec (0.00s)
PASS
coverage: 14.2% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation	0.030s	coverage: 14.2% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2	[no test files]
=== RUN   TestGenGeneralName
--- PASS: TestGenGeneralName (0.00s)
PASS
coverage: 0.4% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/common/jobcontroller	0.023s	coverage: 0.4% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured	[no test files]
=== RUN   TestCreatePods
--- PASS: TestCreatePods (0.01s)
=== RUN   TestCreateService
time="2019-08-15T15:08:38Z" level=info msg="Controller test-tfjob created service empty_service"
--- PASS: TestCreateService (0.00s)
=== RUN   TestCreateServicesWithControllerRef
time="2019-08-15T15:08:38Z" level=info msg="Controller test-tfjob created service empty_service"
--- PASS: TestCreateServicesWithControllerRef (0.00s)
=== RUN   TestClaimServices
--- PASS: TestClaimServices (0.00s)
PASS
coverage: 41.1% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/control	0.061s	coverage: 41.1% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
=== RUN   TestNormalPath
time="2019-08-15T15:08:46Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:46Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:46Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:08:46Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:08:46Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:46Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:08:46Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.854172ms)" job=default.test-tfjob
time="2019-08-15T15:08:46Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:46Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:46Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:08:46Z" level=info msg="Need to create new pod: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T15:08:46Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T15:08:46Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:46Z" level=info msg="need to create new service: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T15:08:46Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T15:08:46Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:08:46Z" level=info msg="Need to create new pod: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:08:46Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:08:46Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:08:46Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:46Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:08:46Z" level=info msg="need to create new service: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:08:46Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:08:46Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:08:46Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (678.931µs)" job=default.test-tfjob
time="2019-08-15T15:08:46Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:46Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:46Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:08:46Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:46Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:46Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (427.916µs)" job=default.test-tfjob
time="2019-08-15T15:08:46Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:46Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:46Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:08:46Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T15:08:46Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:46Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T15:08:46Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:08:46Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:08:46Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:46Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:08:46Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:08:46Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (543.337µs)" job=default.test-tfjob
time="2019-08-15T15:08:46Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:46Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:46Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:08:46Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:46Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T15:08:46Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:46Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T15:08:46Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:08:46Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:46Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:08:46Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (563.942µs)" job=default.test-tfjob
time="2019-08-15T15:08:46Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:46Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:46Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:08:46Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:46Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:46Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (527.926µs)" job=default.test-tfjob
time="2019-08-15T15:08:46Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:46Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:46Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:08:46Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T15:08:46Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:46Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T15:08:46Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:08:46Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:46Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:08:46Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (837.231µs)" job=default.test-tfjob
time="2019-08-15T15:08:46Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:46Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:46Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:08:46Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:46Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:46Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:46Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:46Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:46Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:46Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:46Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
E0815 15:08:46.234314    9538 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(0xc420624fa0), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"PS":(*v1.ReplicaSpec)(0xc42010e2c0), "Worker":(*v1.ReplicaSpec)(0xc42010e840)}}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"PS":(*v1.ReplicaStatus)(0xc4207abd60), "Worker":(*v1.ReplicaStatus)(0xc4207abd90)}, StartTime:(*v1.Time)(0xc42050cc80), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'TFJobSucceeded' 'TFJob test-tfjob successfully completed.'
time="2019-08-15T15:08:46Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (893.67µs)" job=default.test-tfjob
--- PASS: TestNormalPath (0.03s)
=== RUN   TestRun
time="2019-08-15T15:08:46Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:46Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:46Z" level=info msg="Starting TFJob controller"
time="2019-08-15T15:08:46Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-15T15:08:46Z" level=info msg="Starting 1 workers"
time="2019-08-15T15:08:46Z" level=info msg="Started workers"
time="2019-08-15T15:08:46Z" level=info msg="Shutting down workers"
--- PASS: TestRun (0.50s)
=== RUN   TestSyncPdb
time="2019-08-15T15:08:46Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:46Z" level=info msg="Creating Job controller"
--- PASS: TestSyncPdb (0.00s)
=== RUN   TestAddTFJob
time="2019-08-15T15:08:46Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:46Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:46Z" level=info msg="TFJob test-tfjob is created." job=default.test-tfjob uid=
time="2019-08-15T15:08:46Z" level=info msg="Starting TFJob controller"
time="2019-08-15T15:08:46Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-15T15:08:46Z" level=info msg="Starting 1 workers"
time="2019-08-15T15:08:46Z" level=info msg="Started workers"
--- PASS: TestAddTFJob (0.10s)
=== RUN   TestCopyLabelsAndAnnotation
time="2019-08-15T15:08:46Z" level=info msg="Shutting down workers"
time="2019-08-15T15:08:46Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:46Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:46Z" level=info msg="Starting TFJob controller"
time="2019-08-15T15:08:46Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-15T15:08:46Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:08:46Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:08:46Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:46Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:08:46Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (634.252µs)" job=default.test-tfjob
--- PASS: TestCopyLabelsAndAnnotation (0.00s)
=== RUN   TestDeletePodsAndServices
time="2019-08-15T15:08:46Z" level=info msg="Starting 1 workers"
time="2019-08-15T15:08:46Z" level=info msg="Started workers"
time="2019-08-15T15:08:46Z" level=info msg="Shutting down workers"
time="2019-08-15T15:08:46Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:46Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:46Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:08:46Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (331.398µs)" job=default.test-tfjob
time="2019-08-15T15:08:46Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:46Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:46Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:08:46Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (290.53µs)" job=default.test-tfjob
time="2019-08-15T15:08:46Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:46Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:46Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:08:46Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:46Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:46Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:46Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:46Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:46Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:46Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (527.044µs)" job=default.test-tfjob
time="2019-08-15T15:08:46Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:46Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:46Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:08:46Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:46Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:46Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:46Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:46Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:46Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:46Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (3.388787ms)" job=default.test-tfjob
--- PASS: TestDeletePodsAndServices (0.02s)
=== RUN   TestCleanupTFJob
time="2019-08-15T15:08:46Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:46Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:46Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:08:46Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (305.744µs)" job=default.test-tfjob
time="2019-08-15T15:08:46Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:46Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:46Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:08:46Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (294.637µs)" job=default.test-tfjob
time="2019-08-15T15:08:46Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:46Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:48Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:08:48Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:48Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:48Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:48Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:48Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:48Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:48Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (569.829µs)" job=default.test-tfjob
--- PASS: TestCleanupTFJob (2.01s)
=== RUN   TestActiveDeadlineSeconds
time="2019-08-15T15:08:48Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:48Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:48Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:08:48Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:48Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:48Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:48Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:50Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
--- PASS: TestActiveDeadlineSeconds (2.00s)
=== RUN   TestBackoffForOnFailure
time="2019-08-15T15:08:50Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:50Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:50Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:08:50Z" level=warning msg="The restart policy of replica PS of the job test-tfjob is not OnFailure or Always. Not counted in backoff limit." job=default.test-tfjob uid=
time="2019-08-15T15:08:50Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (404.606µs)" job=default.test-tfjob
--- PASS: TestBackoffForOnFailure (0.00s)
=== RUN   TestAddPod
time="2019-08-15T15:08:50Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:50Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:50Z" level=info msg="Starting TFJob controller"
time="2019-08-15T15:08:50Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-15T15:08:50Z" level=info msg="Starting 1 workers"
time="2019-08-15T15:08:50Z" level=info msg="Started workers"
--- PASS: TestAddPod (0.10s)
=== RUN   TestClusterSpec
time="2019-08-15T15:08:50Z" level=info msg="Shutting down workers"
--- PASS: TestClusterSpec (0.00s)
=== RUN   TestRestartPolicy
--- PASS: TestRestartPolicy (0.00s)
=== RUN   TestExitCode
time="2019-08-15T15:08:50Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:50Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:50Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:08:50Z" level=info msg="Ignoring inactive pod default/worker-0 in state Failed, deletion time <nil>"
time="2019-08-15T15:08:50Z" level=info msg="Pod: default.worker-0 exited with code 130" job=default.test-tfjob replica-type=worker uid=
E0815 15:08:50.972358    9538 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(0xc4206671f0), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"Worker":(*v1.ReplicaSpec)(0xc420388840)}}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"Worker":(*v1.ReplicaStatus)(0xc4207256e0)}, StartTime:(*v1.Time)(nil), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'ExitedWithCode' 'Pod: default.worker-0 exited with code 130'
time="2019-08-15T15:08:50Z" level=info msg="Starting TFJob controller"
time="2019-08-15T15:08:50Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-15T15:08:50Z" level=info msg="Need to restart the pod: default.worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:08:50Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=1" job=default.test-tfjob uid=
E0815 15:08:50.973074    9538 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(0xc4206671f0), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"Worker":(*v1.ReplicaSpec)(0xc420388840)}}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"Worker":(*v1.ReplicaStatus)(0xc4207256e0)}, StartTime:(*v1.Time)(0xc42050d980), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Warning' 'TFJobRestarting' 'TFJob test-tfjob is restarting because 1 Worker replica(s) failed.'
time="2019-08-15T15:08:50Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:08:50Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.336623ms)" job=default.test-tfjob
--- PASS: TestExitCode (0.00s)
=== RUN   TestAddService
time="2019-08-15T15:08:50Z" level=info msg="Starting 1 workers"
time="2019-08-15T15:08:50Z" level=info msg="Started workers"
time="2019-08-15T15:08:50Z" level=info msg="Shutting down workers"
time="2019-08-15T15:08:50Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:50Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:50Z" level=info msg="Starting TFJob controller"
time="2019-08-15T15:08:50Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-15T15:08:51Z" level=info msg="Starting 1 workers"
time="2019-08-15T15:08:51Z" level=info msg="Started workers"
--- PASS: TestAddService (0.10s)
=== RUN   TestFailed
time="2019-08-15T15:08:51Z" level=info msg="Shutting down workers"
time="2019-08-15T15:08:51Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:51Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:51Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=0, failed=1" job=default.test-tfjob uid=
E0815 15:08:51.076958    9538 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(nil), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"Worker":(*v1.ReplicaSpec)(0xc4206ab600)}}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"Worker":(*v1.ReplicaStatus)(0xc4209f4180)}, StartTime:(*v1.Time)(0xc4209e9e80), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'TFJobFailed' 'TFJob test-tfjob has failed because 1 Worker replica(s) failed.'
--- PASS: TestFailed (0.00s)
=== RUN   TestStatus
time="2019-08-15T15:08:51Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:51Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:51Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:51Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:51Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:51Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:51Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:51Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:51Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:51Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:51Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-15T15:08:51Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:51Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:51Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:51Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-15T15:08:51Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:51Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:51Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:51Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:51Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:51Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:51Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:51Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:51Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:51Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:51Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:51Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:51Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=2, failed=2" job=default.test-tfjob uid=
time="2019-08-15T15:08:51Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:51Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:51Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:51Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=2, running=0, failed=2" job=default.test-tfjob uid=
time="2019-08-15T15:08:51Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:51Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:51Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:51Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=3, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:51Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:51Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:51Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:51Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:51Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2019-08-15T15:08:51Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:51Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:51Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:51Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:51Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:51Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:51Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:51Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:51Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:51Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:51Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=1, failed=1" job=default.test-tfjob uid=
time="2019-08-15T15:08:51Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:51Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:51Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-15T15:08:51Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:51Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:51Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:51Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:51Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:51Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2019-08-15T15:08:51Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:51Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:51Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:51Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-15T15:08:51Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2019-08-15T15:08:51Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
--- PASS: TestStatus (0.02s)
=== RUN   TestGenOwnerReference
--- PASS: TestGenOwnerReference (0.00s)
=== RUN   TestGenLabels
--- PASS: TestGenLabels (0.00s)
=== RUN   TestConvertTFJobToUnstructured
--- PASS: TestConvertTFJobToUnstructured (0.00s)
PASS
coverage: 44.0% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow	4.932s	coverage: 44.0% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
=== RUN   TestNormalPath
time="2019-08-15T15:08:57Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:57Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:57Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:08:57Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:57Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:57Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:57Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:57Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:57Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:57Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
E0815 15:08:57.260348    9652 event.go:259] Could not construct reference to: '&v1beta2.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta2.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1beta2.CleanPodPolicy)(0xc4203e00f0), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1beta2.TFReplicaType]*v1beta2.ReplicaSpec{"PS":(*v1beta2.ReplicaSpec)(0xc42032eb00), "Worker":(*v1beta2.ReplicaSpec)(0xc42032edc0)}}, Status:v1beta2.JobStatus{Conditions:[]v1beta2.JobCondition(nil), ReplicaStatuses:map[v1beta2.ReplicaType]*v1beta2.ReplicaStatus{"Worker":(*v1beta2.ReplicaStatus)(0xc420108a10)}, StartTime:(*v1.Time)(0xc420390280), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'TFJobSucceeded' 'TFJob test-tfjob successfully completed.'
time="2019-08-15T15:08:57Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:57Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (3.925852ms)" job=default.test-tfjob
time="2019-08-15T15:08:57Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:57Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:57Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:08:57Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:08:57Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:57Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:08:57Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.0655ms)" job=default.test-tfjob
time="2019-08-15T15:08:57Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:57Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:57Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:08:57Z" level=info msg="Need to create new pod: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T15:08:57Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T15:08:57Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:57Z" level=info msg="need to create new service: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T15:08:57Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T15:08:57Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:08:57Z" level=info msg="Need to create new pod: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:08:57Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:08:57Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:08:57Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:57Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:08:57Z" level=info msg="need to create new service: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:08:57Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:08:57Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:08:57Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (926.553µs)" job=default.test-tfjob
time="2019-08-15T15:08:57Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:57Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:57Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:08:57Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:57Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:57Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (511.345µs)" job=default.test-tfjob
time="2019-08-15T15:08:57Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:57Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:57Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:08:57Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:08:57Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:08:57Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:57Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:08:57Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:08:57Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T15:08:57Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:57Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T15:08:57Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (935.175µs)" job=default.test-tfjob
time="2019-08-15T15:08:57Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:57Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:57Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:08:57Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:57Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T15:08:57Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:57Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T15:08:57Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:08:57Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:57Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:08:57Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (902.463µs)" job=default.test-tfjob
time="2019-08-15T15:08:57Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:57Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:57Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:08:57Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:57Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:57Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (528.606µs)" job=default.test-tfjob
time="2019-08-15T15:08:57Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:57Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:57Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:08:57Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:08:57Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:57Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:08:57Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T15:08:57Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:57Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T15:08:57Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.49458ms)" job=default.test-tfjob
--- PASS: TestNormalPath (0.04s)
=== RUN   TestRun
time="2019-08-15T15:08:57Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:57Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:57Z" level=info msg="Starting TFJob controller"
time="2019-08-15T15:08:57Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-15T15:08:57Z" level=info msg="Starting 1 workers"
time="2019-08-15T15:08:57Z" level=info msg="Started workers"
time="2019-08-15T15:08:57Z" level=info msg="Shutting down workers"
--- PASS: TestRun (0.50s)
=== RUN   TestSyncPdb
time="2019-08-15T15:08:57Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:57Z" level=info msg="Creating Job controller"
--- PASS: TestSyncPdb (0.00s)
=== RUN   TestAddTFJob
time="2019-08-15T15:08:57Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:57Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:57Z" level=info msg="Starting TFJob controller"
time="2019-08-15T15:08:57Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-15T15:08:57Z" level=info msg="TFJob test-tfjob is created." job=default.test-tfjob uid=
time="2019-08-15T15:08:57Z" level=info msg="Starting 1 workers"
time="2019-08-15T15:08:57Z" level=info msg="Started workers"
--- PASS: TestAddTFJob (0.10s)
=== RUN   TestCopyLabelsAndAnnotation
time="2019-08-15T15:08:57Z" level=info msg="Shutting down workers"
time="2019-08-15T15:08:57Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:57Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:57Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:08:57Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:08:57Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:57Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:08:57Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (346.548µs)" job=default.test-tfjob
--- PASS: TestCopyLabelsAndAnnotation (0.00s)
=== RUN   TestDeletePodsAndServices
time="2019-08-15T15:08:57Z" level=info msg="Starting TFJob controller"
time="2019-08-15T15:08:57Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-15T15:08:57Z" level=info msg="Starting 1 workers"
time="2019-08-15T15:08:57Z" level=info msg="Started workers"
time="2019-08-15T15:08:57Z" level=info msg="Shutting down workers"
time="2019-08-15T15:08:57Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:57Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:57Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:08:57Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:57Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:57Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (959.84µs)" job=default.test-tfjob
time="2019-08-15T15:08:57Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:57Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:57Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:08:57Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:57Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:57Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (723.539µs)" job=default.test-tfjob
time="2019-08-15T15:08:57Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:57Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:57Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:08:57Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:57Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:57Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:57Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:57Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:57Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:57Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:57Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:57Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (664.554µs)" job=default.test-tfjob
time="2019-08-15T15:08:57Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:57Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:57Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:08:57Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:57Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:57Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:57Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:57Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:57Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:57Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:57Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:57Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (4.352456ms)" job=default.test-tfjob
--- PASS: TestDeletePodsAndServices (0.01s)
=== RUN   TestCleanupTFJob
time="2019-08-15T15:08:57Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:57Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:57Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:08:57Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:57Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:57Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (459.546µs)" job=default.test-tfjob
time="2019-08-15T15:08:57Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:57Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:57Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:08:57Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:57Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:57Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (393.25µs)" job=default.test-tfjob
time="2019-08-15T15:08:57Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:57Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:59Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:08:59Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:59Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:59Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:59Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:59Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:59Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:08:59Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:59Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:59Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.153245ms)" job=default.test-tfjob
--- PASS: TestCleanupTFJob (2.01s)
=== RUN   TestActiveDeadlineSeconds
time="2019-08-15T15:08:59Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:59Z" level=info msg="Creating Job controller"
time="2019-08-15T15:08:59Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:08:59Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:59Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:08:59Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:08:59Z" level=info msg="Creating Job controller"
time="2019-08-15T15:09:01Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
--- PASS: TestActiveDeadlineSeconds (2.00s)
=== RUN   TestBackoffForOnFailure
time="2019-08-15T15:09:01Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:09:01Z" level=info msg="Creating Job controller"
time="2019-08-15T15:09:01Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:09:01Z" level=warning msg="The restart policy of replica PS of the job test-tfjob is not OnFailure or Always. Not counted in backoff limit." job=default.test-tfjob uid=
time="2019-08-15T15:09:01Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (409.037µs)" job=default.test-tfjob
--- PASS: TestBackoffForOnFailure (0.00s)
=== RUN   TestAddPod
time="2019-08-15T15:09:01Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:09:01Z" level=info msg="Creating Job controller"
time="2019-08-15T15:09:01Z" level=info msg="Starting TFJob controller"
time="2019-08-15T15:09:01Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-15T15:09:02Z" level=info msg="Starting 1 workers"
time="2019-08-15T15:09:02Z" level=info msg="Started workers"
--- PASS: TestAddPod (0.10s)
=== RUN   TestClusterSpec
time="2019-08-15T15:09:02Z" level=info msg="Shutting down workers"
--- PASS: TestClusterSpec (0.00s)
=== RUN   TestRestartPolicy
--- PASS: TestRestartPolicy (0.00s)
=== RUN   TestExitCode
time="2019-08-15T15:09:02Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:09:02Z" level=info msg="Creating Job controller"
time="2019-08-15T15:09:02Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:09:02Z" level=info msg="Ignoring inactive pod default/worker-0 in state Failed, deletion time <nil>"
time="2019-08-15T15:09:02Z" level=info msg="Pod: default.worker-0 exited with code 130" job=default.test-tfjob replica-type=worker uid=
E0815 15:09:02.014677    9652 event.go:259] Could not construct reference to: '&v1beta2.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta2.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1beta2.CleanPodPolicy)(0xc4207f5040), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1beta2.TFReplicaType]*v1beta2.ReplicaSpec{"Worker":(*v1beta2.ReplicaSpec)(0xc42087db80)}}, Status:v1beta2.JobStatus{Conditions:[]v1beta2.JobCondition(nil), ReplicaStatuses:map[v1beta2.ReplicaType]*v1beta2.ReplicaStatus{"Worker":(*v1beta2.ReplicaStatus)(0xc420646c80)}, StartTime:(*v1.Time)(nil), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'ExitedWithCode' 'Pod: default.worker-0 exited with code 130'
time="2019-08-15T15:09:02Z" level=info msg="Need to restart the pod: default.worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:09:02Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-15T15:09:02Z" level=info msg="Starting TFJob controller"
time="2019-08-15T15:09:02Z" level=info msg="Waiting for informer caches to sync"
E0815 15:09:02.014990    9652 event.go:259] Could not construct reference to: '&v1beta2.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta2.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1beta2.CleanPodPolicy)(0xc4207f5040), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1beta2.TFReplicaType]*v1beta2.ReplicaSpec{"Worker":(*v1beta2.ReplicaSpec)(0xc42087db80)}}, Status:v1beta2.JobStatus{Conditions:[]v1beta2.JobCondition(nil), ReplicaStatuses:map[v1beta2.ReplicaType]*v1beta2.ReplicaStatus{"Worker":(*v1beta2.ReplicaStatus)(0xc420646c80)}, StartTime:(*v1.Time)(0xc420844ba0), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Warning' 'TFJobRestarting' 'TFJob test-tfjob is restarting because 1 Worker replica(s) failed.'
time="2019-08-15T15:09:02Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:09:02Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (988.327µs)" job=default.test-tfjob
--- PASS: TestExitCode (0.00s)
=== RUN   TestAddService
time="2019-08-15T15:09:02Z" level=info msg="Starting 1 workers"
time="2019-08-15T15:09:02Z" level=info msg="Started workers"
time="2019-08-15T15:09:02Z" level=info msg="Shutting down workers"
time="2019-08-15T15:09:02Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:09:02Z" level=info msg="Creating Job controller"
time="2019-08-15T15:09:02Z" level=info msg="Starting TFJob controller"
time="2019-08-15T15:09:02Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-15T15:09:02Z" level=info msg="Starting 1 workers"
time="2019-08-15T15:09:02Z" level=info msg="Started workers"
--- PASS: TestAddService (0.10s)
=== RUN   TestFailed
time="2019-08-15T15:09:02Z" level=info msg="Shutting down workers"
time="2019-08-15T15:09:02Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:09:02Z" level=info msg="Creating Job controller"
time="2019-08-15T15:09:02Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=0, failed=1" job=default.test-tfjob uid=
E0815 15:09:02.118894    9652 event.go:259] Could not construct reference to: '&v1beta2.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta2.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1beta2.CleanPodPolicy)(nil), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1beta2.TFReplicaType]*v1beta2.ReplicaSpec{"Worker":(*v1beta2.ReplicaSpec)(0xc4207f22c0)}}, Status:v1beta2.JobStatus{Conditions:[]v1beta2.JobCondition(nil), ReplicaStatuses:map[v1beta2.ReplicaType]*v1beta2.ReplicaStatus{"Worker":(*v1beta2.ReplicaStatus)(0xc4208de530)}, StartTime:(*v1.Time)(0xc4208e2740), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'TFJobFailed' 'TFJob test-tfjob has failed because 1 Worker replica(s) failed.'
--- PASS: TestFailed (0.00s)
=== RUN   TestStatus
time="2019-08-15T15:09:02Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:09:02Z" level=info msg="Creating Job controller"
time="2019-08-15T15:09:02Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:09:02Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:09:02Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:09:02Z" level=info msg="Creating Job controller"
time="2019-08-15T15:09:02Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:09:02Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:09:02Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:09:02Z" level=info msg="Creating Job controller"
time="2019-08-15T15:09:02Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-15T15:09:02Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:09:02Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:09:02Z" level=info msg="Creating Job controller"
time="2019-08-15T15:09:02Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-15T15:09:02Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:09:02Z" level=info msg="Creating Job controller"
time="2019-08-15T15:09:02Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:09:02Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:09:02Z" level=info msg="Creating Job controller"
time="2019-08-15T15:09:02Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:09:02Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:09:02Z" level=info msg="Creating Job controller"
time="2019-08-15T15:09:02Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:09:02Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:09:02Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:09:02Z" level=info msg="Creating Job controller"
time="2019-08-15T15:09:02Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=2, failed=2" job=default.test-tfjob uid=
time="2019-08-15T15:09:02Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:09:02Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:09:02Z" level=info msg="Creating Job controller"
time="2019-08-15T15:09:02Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=2, running=0, failed=2" job=default.test-tfjob uid=
time="2019-08-15T15:09:02Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:09:02Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:09:02Z" level=info msg="Creating Job controller"
time="2019-08-15T15:09:02Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=3, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:09:02Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:09:02Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:09:02Z" level=info msg="Creating Job controller"
time="2019-08-15T15:09:02Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:09:02Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2019-08-15T15:09:02Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:09:02Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:09:02Z" level=info msg="Creating Job controller"
time="2019-08-15T15:09:02Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:09:02Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:09:02Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:09:02Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:09:02Z" level=info msg="Creating Job controller"
time="2019-08-15T15:09:02Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:09:02Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:09:02Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=1, failed=1" job=default.test-tfjob uid=
time="2019-08-15T15:09:02Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:09:02Z" level=info msg="Creating Job controller"
time="2019-08-15T15:09:02Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-15T15:09:02Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:09:02Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:09:02Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:09:02Z" level=info msg="Creating Job controller"
time="2019-08-15T15:09:02Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:09:02Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2019-08-15T15:09:02Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:09:02Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:09:02Z" level=info msg="Creating Job controller"
time="2019-08-15T15:09:02Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-15T15:09:02Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2019-08-15T15:09:02Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
--- PASS: TestStatus (0.02s)
=== RUN   TestGenOwnerReference
--- PASS: TestGenOwnerReference (0.00s)
=== RUN   TestGenLabels
--- PASS: TestGenLabels (0.00s)
=== RUN   TestConvertTFJobToUnstructured
--- PASS: TestConvertTFJobToUnstructured (0.00s)
PASS
coverage: 43.9% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow	4.944s	coverage: 43.9% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
?   	github.com/kubeflow/tf-operator/pkg/logger	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/util	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/util/k8sutil	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/util/signals	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/util/train	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/version	[no test files]
ignoring pkg/apis/common/v1/openapi_generated.go
ignoring pkg/apis/common/v1/zz_generated.deepcopy.go
ignoring pkg/apis/common/v1/zz_generated.defaults.go
ignoring pkg/apis/tensorflow/v1/openapi_generated.go
ignoring pkg/apis/tensorflow/v1/zz_generated.deepcopy.go
ignoring pkg/apis/tensorflow/v1/zz_generated.defaults.go
Job #2537.1
https://coveralls.io/jobs/52063956
TravisBuddy Request Identifier: a02bbec0-bf6e-11e9-b293-f57b21f1c1f9

@coveralls
Copy link

coveralls commented Aug 15, 2019 •
edited
Loading

Coverage Status

Coverage remained the same at 76.744% when pulling f1e52ed on hougangliu:genLabels into b7de6e4 on kubeflow:master.

@TravisBuddy
Copy link

Travis tests have failed

Hey @hougangliu,
Please read the following log in order to understand the failure reason.
It'll be awesome if you fix what's wrong and commit the changes.

1st Build

View build log

gometalinter --config=linter_config.json --vendor ./... 
pkg/controller.v1/tensorflow/pod_test.go:1::warning: file is not goimported (goimports)
pkg/controller.v1beta2/tensorflow/pod_test.go:1::warning: file is not goimported (goimports)
goveralls -service=travis-ci -v -package ./pkg/... -ignore "pkg/client/*/*.go,pkg/client/*/*/*.go,pkg/client/*/*/*/*.go,pkg/client/*/*/*/*/*.go,pkg/client/*/*/*/*/*/*.go,pkg/client/*/*/*/*/*/*/*.go,pkg/util/testutil/*.go,pkg/apis/tensorflow/*/zz_generated.*.go,pkg/apis/tensorflow/*/*_generated.go,pkg/apis/common/*/zz_generated.*.go,pkg/apis/common/*/*_generated.go" 
?   	github.com/kubeflow/tf-operator/pkg/apis/common/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2	[no test files]
=== RUN   TestSetTypeNames
--- PASS: TestSetTypeNames (0.00s)
=== RUN   TestSetDefaultTFJob
--- PASS: TestSetDefaultTFJob (0.00s)
=== RUN   TestIsChieforMaster
--- PASS: TestIsChieforMaster (0.00s)
PASS
coverage: 20.9% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1	0.035s	coverage: 20.9% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
=== RUN   TestSetTypeNames
--- PASS: TestSetTypeNames (0.00s)
=== RUN   TestSetDefaultTFJob
--- PASS: TestSetDefaultTFJob (0.00s)
=== RUN   TestIsChieforMaster
--- PASS: TestIsChieforMaster (0.00s)
PASS
coverage: 20.9% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2	0.037s	coverage: 20.9% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
=== RUN   TestValidateBetaTwoTFJobSpec
time="2019-08-15T15:32:00Z" level=error msg="TFJobSpec is not valid: Image is undefined in the container of Worker"
time="2019-08-15T15:32:00Z" level=error msg="TFJobSpec is not valid: There is no container named tensorflow in Worker"
--- PASS: TestValidateBetaTwoTFJobSpec (0.00s)
=== RUN   TestValidateV1TFJobSpec
time="2019-08-15T15:32:00Z" level=error msg="TFJobSpec is not valid: Image is undefined in the container of Worker"
time="2019-08-15T15:32:00Z" level=error msg="TFJobSpec is not valid: There is no container named tensorflow in Worker"
--- PASS: TestValidateV1TFJobSpec (0.00s)
PASS
coverage: 14.2% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation	0.032s	coverage: 14.2% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2	[no test files]
=== RUN   TestGenGeneralName
--- PASS: TestGenGeneralName (0.00s)
PASS
coverage: 0.4% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/common/jobcontroller	0.024s	coverage: 0.4% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured	[no test files]
=== RUN   TestCreatePods
--- PASS: TestCreatePods (0.01s)
=== RUN   TestCreateService
time="2019-08-15T15:32:20Z" level=info msg="Controller test-tfjob created service empty_service"
--- PASS: TestCreateService (0.00s)
=== RUN   TestCreateServicesWithControllerRef
time="2019-08-15T15:32:20Z" level=info msg="Controller test-tfjob created service empty_service"
--- PASS: TestCreateServicesWithControllerRef (0.00s)
=== RUN   TestClaimServices
--- PASS: TestClaimServices (0.00s)
PASS
coverage: 41.1% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/control	0.065s	coverage: 41.1% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
=== RUN   TestNormalPath
time="2019-08-15T15:32:28Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:28Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:28Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:32:28Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:32:28Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:28Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:32:28Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (2.236534ms)" job=default.test-tfjob
time="2019-08-15T15:32:28Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:28Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:28Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:32:28Z" level=info msg="Need to create new pod: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T15:32:28Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T15:32:28Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:28Z" level=info msg="need to create new service: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T15:32:28Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T15:32:28Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:32:28Z" level=info msg="Need to create new pod: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:32:28Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:32:28Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:32:28Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:28Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:32:28Z" level=info msg="need to create new service: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:32:28Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:32:28Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:32:28Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (767.105µs)" job=default.test-tfjob
time="2019-08-15T15:32:28Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:28Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:28Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:32:28Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:28Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:28Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (470.388µs)" job=default.test-tfjob
time="2019-08-15T15:32:28Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:28Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:28Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:32:28Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:32:28Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:32:28Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:28Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:32:28Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:32:28Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T15:32:28Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:28Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T15:32:28Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (865.104µs)" job=default.test-tfjob
time="2019-08-15T15:32:28Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:28Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:28Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:32:28Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:28Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T15:32:28Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:28Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T15:32:28Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:32:28Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:28Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:32:28Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (785.386µs)" job=default.test-tfjob
time="2019-08-15T15:32:28Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:28Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:28Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:32:28Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:28Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:28Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (2.728262ms)" job=default.test-tfjob
time="2019-08-15T15:32:28Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:28Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:28Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:32:28Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:32:28Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:28Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:32:28Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T15:32:28Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:28Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T15:32:28Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.097421ms)" job=default.test-tfjob
time="2019-08-15T15:32:28Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:28Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:28Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:32:28Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:28Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:28Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:28Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:28Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:28Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:28Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
E0815 15:32:28.728656    9657 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(0xc42081b990), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"PS":(*v1.ReplicaSpec)(0xc420751080), "Worker":(*v1.ReplicaSpec)(0xc420751340)}}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"Worker":(*v1.ReplicaStatus)(0xc420429020)}, StartTime:(*v1.Time)(0xc42081d3c0), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'TFJobSucceeded' 'TFJob test-tfjob successfully completed.'
time="2019-08-15T15:32:28Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:28Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.557579ms)" job=default.test-tfjob
--- PASS: TestNormalPath (0.04s)
=== RUN   TestRun
time="2019-08-15T15:32:28Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:28Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:28Z" level=info msg="Starting TFJob controller"
time="2019-08-15T15:32:28Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-15T15:32:28Z" level=info msg="Starting 1 workers"
time="2019-08-15T15:32:28Z" level=info msg="Started workers"
time="2019-08-15T15:32:29Z" level=info msg="Shutting down workers"
--- PASS: TestRun (0.50s)
=== RUN   TestSyncPdb
time="2019-08-15T15:32:29Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:29Z" level=info msg="Creating Job controller"
--- PASS: TestSyncPdb (0.00s)
=== RUN   TestAddTFJob
time="2019-08-15T15:32:29Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:29Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:29Z" level=info msg="Starting TFJob controller"
time="2019-08-15T15:32:29Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-15T15:32:29Z" level=info msg="TFJob test-tfjob is created." job=default.test-tfjob uid=
time="2019-08-15T15:32:29Z" level=info msg="Starting 1 workers"
time="2019-08-15T15:32:29Z" level=info msg="Started workers"
--- PASS: TestAddTFJob (0.10s)
=== RUN   TestCopyLabelsAndAnnotation
time="2019-08-15T15:32:29Z" level=info msg="Shutting down workers"
time="2019-08-15T15:32:29Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:29Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:29Z" level=info msg="Starting TFJob controller"
time="2019-08-15T15:32:29Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-15T15:32:29Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:32:29Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:32:29Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:29Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:32:29Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (605.935µs)" job=default.test-tfjob
--- PASS: TestCopyLabelsAndAnnotation (0.00s)
=== RUN   TestDeletePodsAndServices
time="2019-08-15T15:32:29Z" level=info msg="Starting 1 workers"
time="2019-08-15T15:32:29Z" level=info msg="Started workers"
time="2019-08-15T15:32:29Z" level=info msg="Shutting down workers"
time="2019-08-15T15:32:29Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:29Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:29Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:32:29Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (468.714µs)" job=default.test-tfjob
time="2019-08-15T15:32:29Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:29Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:29Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:32:29Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (350.279µs)" job=default.test-tfjob
time="2019-08-15T15:32:29Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:29Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:29Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:32:29Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:29Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:29Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:29Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:29Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:29Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:29Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (468.272µs)" job=default.test-tfjob
time="2019-08-15T15:32:29Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:29Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:29Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:32:29Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:29Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:29Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:29Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:29Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:29Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:29Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (396.625µs)" job=default.test-tfjob
--- PASS: TestDeletePodsAndServices (0.01s)
=== RUN   TestCleanupTFJob
time="2019-08-15T15:32:29Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:29Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:29Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:32:29Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (635.479µs)" job=default.test-tfjob
time="2019-08-15T15:32:29Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:29Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:29Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:32:29Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (683.247µs)" job=default.test-tfjob
time="2019-08-15T15:32:29Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:29Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:31Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:32:31Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:31Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:31Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:31Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:31Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:31Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:31Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (887.643µs)" job=default.test-tfjob
--- PASS: TestCleanupTFJob (2.01s)
=== RUN   TestActiveDeadlineSeconds
time="2019-08-15T15:32:31Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:31Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:31Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:32:31Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:31Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:31Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:31Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:33Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
--- PASS: TestActiveDeadlineSeconds (2.01s)
=== RUN   TestBackoffForOnFailure
time="2019-08-15T15:32:33Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:33Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:33Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:32:33Z" level=warning msg="The restart policy of replica PS of the job test-tfjob is not OnFailure or Always. Not counted in backoff limit." job=default.test-tfjob uid=
time="2019-08-15T15:32:33Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (672.156µs)" job=default.test-tfjob
--- PASS: TestBackoffForOnFailure (0.00s)
=== RUN   TestAddPod
time="2019-08-15T15:32:33Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:33Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:33Z" level=info msg="Starting TFJob controller"
time="2019-08-15T15:32:33Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-15T15:32:33Z" level=info msg="Starting 1 workers"
time="2019-08-15T15:32:33Z" level=info msg="Started workers"
--- PASS: TestAddPod (0.10s)
=== RUN   TestClusterSpec
--- PASS: TestClusterSpec (0.00s)
=== RUN   TestRestartPolicy
--- PASS: TestRestartPolicy (0.00s)
=== RUN   TestExitCode
time="2019-08-15T15:32:33Z" level=info msg="Shutting down workers"
time="2019-08-15T15:32:33Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:33Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:33Z" level=info msg="Starting TFJob controller"
time="2019-08-15T15:32:33Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-15T15:32:33Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:32:33Z" level=info msg="Ignoring inactive pod default/worker-0 in state Failed, deletion time <nil>"
time="2019-08-15T15:32:33Z" level=info msg="Pod: default.worker-0 exited with code 130" job=default.test-tfjob replica-type=worker uid=
E0815 15:32:33.470116    9657 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(0xc4200bb680), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"Worker":(*v1.ReplicaSpec)(0xc4203f7340)}}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"Worker":(*v1.ReplicaStatus)(0xc4205e4920)}, StartTime:(*v1.Time)(nil), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'ExitedWithCode' 'Pod: default.worker-0 exited with code 130'
time="2019-08-15T15:32:33Z" level=info msg="Need to restart the pod: default.worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:32:33Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=1" job=default.test-tfjob uid=
E0815 15:32:33.470307    9657 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(0xc4200bb680), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"Worker":(*v1.ReplicaSpec)(0xc4203f7340)}}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"Worker":(*v1.ReplicaStatus)(0xc4205e4920)}, StartTime:(*v1.Time)(0xc4201cfbc0), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Warning' 'TFJobRestarting' 'TFJob test-tfjob is restarting because 1 Worker replica(s) failed.'
time="2019-08-15T15:32:33Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:32:33Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.40954ms)" job=default.test-tfjob
--- PASS: TestExitCode (0.00s)
=== RUN   TestAddService
time="2019-08-15T15:32:33Z" level=info msg="Starting 1 workers"
time="2019-08-15T15:32:33Z" level=info msg="Started workers"
time="2019-08-15T15:32:33Z" level=info msg="Shutting down workers"
time="2019-08-15T15:32:33Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:33Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:33Z" level=info msg="Starting TFJob controller"
time="2019-08-15T15:32:33Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-15T15:32:33Z" level=info msg="Starting 1 workers"
time="2019-08-15T15:32:33Z" level=info msg="Started workers"
--- PASS: TestAddService (0.10s)
time="2019-08-15T15:32:33Z" level=info msg="Shutting down workers"
=== RUN   TestFailed
time="2019-08-15T15:32:33Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:33Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:33Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=0, failed=1" job=default.test-tfjob uid=
E0815 15:32:33.574944    9657 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(nil), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"Worker":(*v1.ReplicaSpec)(0xc4204da580)}}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"Worker":(*v1.ReplicaStatus)(0xc4200447e0)}, StartTime:(*v1.Time)(0xc420417a20), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'TFJobFailed' 'TFJob test-tfjob has failed because 1 Worker replica(s) failed.'
--- PASS: TestFailed (0.00s)
=== RUN   TestStatus
time="2019-08-15T15:32:33Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:33Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:33Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:33Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:33Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:33Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:33Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:33Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:33Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:33Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:33Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-15T15:32:33Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:33Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:33Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:33Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-15T15:32:33Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:33Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:33Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:33Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:33Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:33Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:33Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:33Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:33Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:33Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:33Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:33Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:33Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=2, failed=2" job=default.test-tfjob uid=
time="2019-08-15T15:32:33Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:33Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:33Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:33Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=2, running=0, failed=2" job=default.test-tfjob uid=
time="2019-08-15T15:32:33Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:33Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:33Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:33Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=3, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:33Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:33Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:33Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:33Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:33Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2019-08-15T15:32:33Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:33Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:33Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:33Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:33Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:33Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:33Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:33Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:33Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:33Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:33Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=1, failed=1" job=default.test-tfjob uid=
time="2019-08-15T15:32:33Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:33Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:33Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-15T15:32:33Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:33Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:33Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:33Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:33Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:33Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2019-08-15T15:32:33Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:33Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:33Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:33Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-15T15:32:33Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2019-08-15T15:32:33Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
--- PASS: TestStatus (0.02s)
=== RUN   TestGenOwnerReference
--- PASS: TestGenOwnerReference (0.00s)
=== RUN   TestGenLabels
--- PASS: TestGenLabels (0.00s)
=== RUN   TestConvertTFJobToUnstructured
--- PASS: TestConvertTFJobToUnstructured (0.00s)
PASS
coverage: 44.0% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow	4.961s	coverage: 44.0% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
=== RUN   TestNormalPath
time="2019-08-15T15:32:40Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:40Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:40Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:32:40Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:40Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T15:32:40Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:40Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T15:32:40Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:32:40Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:40Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:32:40Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (2.169828ms)" job=default.test-tfjob
time="2019-08-15T15:32:40Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:40Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:40Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:32:40Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:40Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:40Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (588.324µs)" job=default.test-tfjob
time="2019-08-15T15:32:40Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:40Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:40Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:32:40Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:32:40Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:40Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:32:40Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T15:32:40Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:40Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T15:32:40Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (719.629µs)" job=default.test-tfjob
time="2019-08-15T15:32:40Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:40Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:40Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:32:40Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:40Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:40Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:40Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:40Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:40Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:40Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:40Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
E0815 15:32:40.586335    9766 event.go:259] Could not construct reference to: '&v1beta2.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta2.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1beta2.CleanPodPolicy)(0xc420342640), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1beta2.TFReplicaType]*v1beta2.ReplicaSpec{"PS":(*v1beta2.ReplicaSpec)(0xc420657080), "Worker":(*v1beta2.ReplicaSpec)(0xc420657340)}}, Status:v1beta2.JobStatus{Conditions:[]v1beta2.JobCondition(nil), ReplicaStatuses:map[v1beta2.ReplicaType]*v1beta2.ReplicaStatus{"PS":(*v1beta2.ReplicaStatus)(0xc4201d5f60), "Worker":(*v1beta2.ReplicaStatus)(0xc4201d5f90)}, StartTime:(*v1.Time)(0xc4204ec880), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'TFJobSucceeded' 'TFJob test-tfjob successfully completed.'
time="2019-08-15T15:32:40Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.418368ms)" job=default.test-tfjob
time="2019-08-15T15:32:40Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:40Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:40Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:32:40Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:32:40Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:40Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:32:40Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (515.126µs)" job=default.test-tfjob
time="2019-08-15T15:32:40Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:40Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:40Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:32:40Z" level=info msg="Need to create new pod: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T15:32:40Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T15:32:40Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:40Z" level=info msg="need to create new service: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T15:32:40Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T15:32:40Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:32:40Z" level=info msg="Need to create new pod: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:32:40Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:32:40Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:32:40Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:40Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:32:40Z" level=info msg="need to create new service: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:32:40Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:32:40Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:32:40Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (2.22012ms)" job=default.test-tfjob
time="2019-08-15T15:32:40Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:40Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:40Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:32:40Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:40Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:40Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (987.643µs)" job=default.test-tfjob
time="2019-08-15T15:32:40Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:40Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:40Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:32:40Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T15:32:40Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:40Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-15T15:32:40Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:32:40Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:32:40Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:40Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:32:40Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:32:40Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.612598ms)" job=default.test-tfjob
--- PASS: TestNormalPath (0.03s)
=== RUN   TestRun
time="2019-08-15T15:32:40Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:40Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:40Z" level=info msg="Starting TFJob controller"
time="2019-08-15T15:32:40Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-15T15:32:40Z" level=info msg="Starting 1 workers"
time="2019-08-15T15:32:40Z" level=info msg="Started workers"
time="2019-08-15T15:32:41Z" level=info msg="Shutting down workers"
--- PASS: TestRun (0.50s)
=== RUN   TestSyncPdb
time="2019-08-15T15:32:41Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:41Z" level=info msg="Creating Job controller"
--- PASS: TestSyncPdb (0.00s)
=== RUN   TestAddTFJob
time="2019-08-15T15:32:41Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:41Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:41Z" level=info msg="Starting TFJob controller"
time="2019-08-15T15:32:41Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-15T15:32:41Z" level=info msg="TFJob test-tfjob is created." job=default.test-tfjob uid=
time="2019-08-15T15:32:41Z" level=info msg="Starting 1 workers"
time="2019-08-15T15:32:41Z" level=info msg="Started workers"
--- PASS: TestAddTFJob (0.10s)
=== RUN   TestCopyLabelsAndAnnotation
time="2019-08-15T15:32:41Z" level=info msg="Shutting down workers"
time="2019-08-15T15:32:41Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:41Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:41Z" level=info msg="Starting TFJob controller"
time="2019-08-15T15:32:41Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-15T15:32:41Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:32:41Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:32:41Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:41Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:32:41Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (374.093µs)" job=default.test-tfjob
--- PASS: TestCopyLabelsAndAnnotation (0.00s)
=== RUN   TestDeletePodsAndServices
time="2019-08-15T15:32:41Z" level=info msg="Starting 1 workers"
time="2019-08-15T15:32:41Z" level=info msg="Started workers"
time="2019-08-15T15:32:41Z" level=info msg="Shutting down workers"
time="2019-08-15T15:32:41Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:41Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:41Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:32:41Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:41Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:41Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (618.389µs)" job=default.test-tfjob
time="2019-08-15T15:32:41Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:41Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:41Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:32:41Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:41Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:41Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (507.557µs)" job=default.test-tfjob
time="2019-08-15T15:32:41Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:41Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:41Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:32:41Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:41Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:41Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:41Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:41Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:41Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:41Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:41Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:41Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (617.787µs)" job=default.test-tfjob
time="2019-08-15T15:32:41Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:41Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:41Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:32:41Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:41Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:41Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:41Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:41Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:41Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:41Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:41Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:41Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (621.199µs)" job=default.test-tfjob
--- PASS: TestDeletePodsAndServices (0.01s)
=== RUN   TestCleanupTFJob
time="2019-08-15T15:32:41Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:41Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:41Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:32:41Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:41Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:41Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (487.858µs)" job=default.test-tfjob
time="2019-08-15T15:32:41Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:41Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:41Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:32:41Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:41Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:41Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (583.978µs)" job=default.test-tfjob
time="2019-08-15T15:32:41Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:41Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:43Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:32:43Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:43Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:43Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:43Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:43Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:43Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-15T15:32:43Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:43Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:43Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (871.39µs)" job=default.test-tfjob
--- PASS: TestCleanupTFJob (2.01s)
=== RUN   TestActiveDeadlineSeconds
time="2019-08-15T15:32:43Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:43Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:43Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:32:43Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:43Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:43Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:43Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:45Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
--- PASS: TestActiveDeadlineSeconds (2.00s)
=== RUN   TestBackoffForOnFailure
time="2019-08-15T15:32:45Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:45Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:45Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:32:45Z" level=warning msg="The restart policy of replica PS of the job test-tfjob is not OnFailure or Always. Not counted in backoff limit." job=default.test-tfjob uid=
time="2019-08-15T15:32:45Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (268.117µs)" job=default.test-tfjob
--- PASS: TestBackoffForOnFailure (0.00s)
=== RUN   TestAddPod
time="2019-08-15T15:32:45Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:45Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:45Z" level=info msg="Starting TFJob controller"
time="2019-08-15T15:32:45Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-15T15:32:45Z" level=info msg="Starting 1 workers"
time="2019-08-15T15:32:45Z" level=info msg="Started workers"
--- PASS: TestAddPod (0.10s)
=== RUN   TestClusterSpec
time="2019-08-15T15:32:45Z" level=info msg="Shutting down workers"
--- PASS: TestClusterSpec (0.00s)
=== RUN   TestRestartPolicy
--- PASS: TestRestartPolicy (0.00s)
=== RUN   TestExitCode
time="2019-08-15T15:32:45Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:45Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:45Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-15T15:32:45Z" level=info msg="Ignoring inactive pod default/worker-0 in state Failed, deletion time <nil>"
time="2019-08-15T15:32:45Z" level=info msg="Pod: default.worker-0 exited with code 130" job=default.test-tfjob replica-type=worker uid=
E0815 15:32:45.336198    9766 event.go:259] Could not construct reference to: '&v1beta2.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta2.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1beta2.CleanPodPolicy)(0xc4204de0a0), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1beta2.TFReplicaType]*v1beta2.ReplicaSpec{"Worker":(*v1beta2.ReplicaSpec)(0xc420524840)}}, Status:v1beta2.JobStatus{Conditions:[]v1beta2.JobCondition(nil), ReplicaStatuses:map[v1beta2.ReplicaType]*v1beta2.ReplicaStatus{"Worker":(*v1beta2.ReplicaStatus)(0xc420382940)}, StartTime:(*v1.Time)(nil), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'ExitedWithCode' 'Pod: default.worker-0 exited with code 130'
time="2019-08-15T15:32:45Z" level=info msg="Need to restart the pod: default.worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:32:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=1" job=default.test-tfjob uid=
E0815 15:32:45.336325    9766 event.go:259] Could not construct reference to: '&v1beta2.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta2.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1beta2.CleanPodPolicy)(0xc4204de0a0), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1beta2.TFReplicaType]*v1beta2.ReplicaSpec{"Worker":(*v1beta2.ReplicaSpec)(0xc420524840)}}, Status:v1beta2.JobStatus{Conditions:[]v1beta2.JobCondition(nil), ReplicaStatuses:map[v1beta2.ReplicaType]*v1beta2.ReplicaStatus{"Worker":(*v1beta2.ReplicaStatus)(0xc420382940)}, StartTime:(*v1.Time)(0xc42052a560), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Warning' 'TFJobRestarting' 'TFJob test-tfjob is restarting because 1 Worker replica(s) failed.'
time="2019-08-15T15:32:45Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-15T15:32:45Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (661.174µs)" job=default.test-tfjob
--- PASS: TestExitCode (0.00s)
=== RUN   TestAddService
time="2019-08-15T15:32:45Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:45Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:45Z" level=info msg="Starting TFJob controller"
time="2019-08-15T15:32:45Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-15T15:32:45Z" level=info msg="Starting TFJob controller"
time="2019-08-15T15:32:45Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-15T15:32:45Z" level=info msg="Starting 1 workers"
time="2019-08-15T15:32:45Z" level=info msg="Started workers"
time="2019-08-15T15:32:45Z" level=info msg="Shutting down workers"
time="2019-08-15T15:32:45Z" level=info msg="Starting 1 workers"
time="2019-08-15T15:32:45Z" level=info msg="Started workers"
--- PASS: TestAddService (0.10s)
=== RUN   TestFailed
time="2019-08-15T15:32:45Z" level=info msg="Shutting down workers"
time="2019-08-15T15:32:45Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:45Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=0, failed=1" job=default.test-tfjob uid=
E0815 15:32:45.440423    9766 event.go:259] Could not construct reference to: '&v1beta2.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta2.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1beta2.CleanPodPolicy)(nil), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1beta2.TFReplicaType]*v1beta2.ReplicaSpec{"Worker":(*v1beta2.ReplicaSpec)(0xc420525340)}}, Status:v1beta2.JobStatus{Conditions:[]v1beta2.JobCondition(nil), ReplicaStatuses:map[v1beta2.ReplicaType]*v1beta2.ReplicaStatus{"Worker":(*v1beta2.ReplicaStatus)(0xc420680b90)}, StartTime:(*v1.Time)(0xc42087e9a0), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'TFJobFailed' 'TFJob test-tfjob has failed because 1 Worker replica(s) failed.'
--- PASS: TestFailed (0.00s)
=== RUN   TestStatus
time="2019-08-15T15:32:45Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:45Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:45Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:45Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:45Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:45Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-15T15:32:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:45Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:45Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-15T15:32:45Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:45Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:45Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:45Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:45Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:45Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:45Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:45Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=2, failed=2" job=default.test-tfjob uid=
time="2019-08-15T15:32:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:45Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:45Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=2, running=0, failed=2" job=default.test-tfjob uid=
time="2019-08-15T15:32:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:45Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:45Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=3, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:45Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:45Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2019-08-15T15:32:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:45Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:45Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:45Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:45Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=1, failed=1" job=default.test-tfjob uid=
time="2019-08-15T15:32:45Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:45Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-15T15:32:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:45Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:45Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2019-08-15T15:32:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-15T15:32:45Z" level=info msg="Creating TFJob controller"
time="2019-08-15T15:32:45Z" level=info msg="Creating Job controller"
time="2019-08-15T15:32:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-15T15:32:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2019-08-15T15:32:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
--- PASS: TestStatus (0.02s)
=== RUN   TestGenOwnerReference
--- PASS: TestGenOwnerReference (0.00s)
=== RUN   TestGenLabels
--- PASS: TestGenLabels (0.00s)
=== RUN   TestConvertTFJobToUnstructured
--- PASS: TestConvertTFJobToUnstructured (0.00s)
PASS
coverage: 43.9% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow	4.948s	coverage: 43.9% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
?   	github.com/kubeflow/tf-operator/pkg/logger	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/util	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/util/k8sutil	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/util/signals	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/util/train	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/version	[no test files]
ignoring pkg/apis/common/v1/openapi_generated.go
ignoring pkg/apis/common/v1/zz_generated.deepcopy.go
ignoring pkg/apis/common/v1/zz_generated.defaults.go
ignoring pkg/apis/tensorflow/v1/openapi_generated.go
ignoring pkg/apis/tensorflow/v1/zz_generated.deepcopy.go
ignoring pkg/apis/tensorflow/v1/zz_generated.defaults.go
Job #2537.1
https://coveralls.io/jobs/52065212
TravisBuddy Request Identifier: f0911380-bf71-11e9-b293-f57b21f1c1f9

gaocegege
gaocegege reviewed Aug 16, 2019
View reviewed changes
Copy link
Member

@gaocegege gaocegege left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/approve

@k8s-ci-robot
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gaocegege

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot merged commit 3cc84f9 into kubeflow:master Aug 16, 2019
This was referenced Aug 30, 2019
Closed
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gaocegege gaocegege gaocegege left review comments

@jimexist jimexist Awaiting requested review from jimexist

@swiftdiaries swiftdiaries Awaiting requested review from swiftdiaries

Assignees

@gaocegege gaocegege

Labels
approved lgtm size/L
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@hougangliu @TravisBuddy @gaocegege @coveralls @k8s-ci-robot