unable to create a tfjob in the UI; namespace not set

[kkasravi]

I'm seeing the following error

time="2018-06-24T19:40:49Z" level=info msg="EnvKubeflowNamespace not set, use default namespace" filename="app/server.go:65"
time="2018-06-24T19:40:49Z" level=info msg="[API Version: v1alpha2 Version: v0.1.0-alpha Git SHA: Not provided. Go Version: go1.10.1 Go OS/Arch: linux/amd64]" filename="app/server.go:70"
W0624 19:40:49.182867       1 client_config.go:529] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0624 19:40:49.187303       1 leaderelection.go:174] attempting to acquire leader lease...
time="2018-06-24T19:40:49Z" level=info msg="This pod's tfjob does not exists" filename="controller.v2/controller_pod.go:309"
time="2018-06-24T19:40:49Z" level=info msg="This pod's tfjob does not exists" filename="controller.v2/controller_pod.go:309"
time="2018-06-24T19:40:49Z" level=info msg="This pod's tfjob does not exists" filename="controller.v2/controller_pod.go:309"
time="2018-06-24T19:40:49Z" level=info msg="This pod's tfjob does not exists" filename="controller.v2/controller_pod.go:309"
time="2018-06-24T19:40:49Z" level=info msg="This pod's tfjob does not exists" filename="controller.v2/controller_pod.go:309"
time="2018-06-24T19:40:49Z" level=info msg="This pod's tfjob does not exists" filenme="controller.v2/controller_pod.go:309"
time="2018-06-24T19:40:49Z" level=info msg="This pod's tfjob does not exists" filename="controller.v2/controller_pod.go:309"
time="2018-06-24T19:40:49Z" level=info msg="This pod's tfjob does not exists" filename="controller.v2/controller_pod.go:309"
time="2018-06-24T19:40:49Z" level=info msg="This pod's tfjob does not exists" filename="controller.v2/controller_pod.go:309"
time="2018-06-24T19:40:49Z" level=info msg="This pod's tfjob does not exists" filename="controller.v2/controller_pod.go:309"
time="2018-06-24T19:40:49Z" level=info msg="This pod's tfjob does not exists" filename="controller.v2/controller_pod.go:309"
time="2018-06-24T19:40:49Z" level=info msg="This pod's tfjob does not exists" filename="controller.v2/controller_pod.go:309"
time="2018-06-24T19:40:49Z" level=info msg="This pod's tfjob does not exists" filename="controller.v2/controller_pod.go:309"
time="2018-06-24T19:40:49Z" level=info msg="This pod's tfjob does not exists" filename="controller.v2/controller_pod.go:309"
time="2018-06-24T19:40:49Z" level=info msg="This pod's tfjob does not exists" filename="controller.v2/controller_pod.go:309"
time="2018-06-24T19:40:49Z" level=info msg="This pod's tfjob does not exists" filename="controller.v2/controller_pod.go:309"
time="2018-06-24T19:40:49Z" level=info msg="This pod's tfjob does not exists" filename="controller.v2/controller_pod.go:309"
time="2018-06-24T19:40:49Z" level=info msg="This pod's tfjob does not exists" filename="controller.v2/controller_pod.go:309"
time="2018-06-24T19:40:49Z" level=info msg="This pod's tfjob does not exists" filename="controller.v2/controller_pod.go:309"
time="2018-06-24T19:40:49Z" level=info msg="This pod's tfjob does not exists" filename="controller.v2/controller_pod.go:309"
time="2018-06-24T19:40:49Z" level=info msg="This pod's tfjob does not exists" filename="controller.v2/controller_pod.go:309"
time="2018-06-24T19:40:49Z" level=info msg="This pod's tfjob does not exists" filename="controller.v2/controller_pod.go:309"
time="2018-06-24T19:40:49Z" level=info msg="This pod's tfjob does not exists" filename="controller.v2/controller_pod.go:309"
time="2018-06-24T19:40:49Z" level=info msg="This pod's tfjob does not exists" filename="controller.v2/controller_pod.go:309"
time="2018-06-24T19:40:49Z" level=info msg="This pod's tfjob does not exists" filename="controller.v2/controller_pod.go:309"
time="2018-06-24T19:40:49Z" level=info msg="This pod's tfjob does not exists" filename="controller.v2/controller_pod.go:309"
time="2018-06-24T19:40:49Z" level=info msg="This pod's tfjob does not exists" filename="controller.v2/controller_pod.go:309"
E0624 19:41:29.473920       1 event.go:260] Could not construct reference to: '&v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"tf-operator", GenerateName:"", Namespace:"default", SelfLink:"/api/v1/namespaces/default/endpoints/tf-operator", UID:"7ccfef35-76f4-11e8-879b-42010a8a01ab", ResourceVersion:"326085", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63665362107, loc:(*time.Location)(0x1aded60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string{"control-plane.alpha.kubernetes.io/leader":"{\"holderIdentity\":\"tf-job-operator-v1alpha2-5bfdc6b65d-sdqg4\",\"leaseDurationSeconds\":15,\"acquireTime\":\"2018-06-24T19:41:29Z\",\"renewTime\":\"2018-06-24T19:41:29Z\",\"leaderTransitions\":5}"}, OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Subsets:[]v1.EndpointSubset(nil)}' due to: 'no kind is registered for the type v1.Endpoints'. Will not report event: 'Normal' 'LeaderElection' 'tf-job-operator-v1alpha2-5bfdc6b65d-sdqg4 became leader'
I0624 19:41:29.474470       1 leaderelection.go:184] successfully acquired lease default/tf-operator
time="2018-06-24T19:41:29Z" level=info msg="Starting TFJob controller" filename="controller.v2/controller.go:251"
time="2018-06-24T19:41:29Z" level=info msg="Waiting for informer caches to sync" filename="controller.v2/controller.go:254"
time="2018-06-24T19:41:29Z" level=info msg="Starting 1 workers" filename="controller.v2/controller.go:267"
time="2018-06-24T19:41:29Z" level=info msg="Started workers" filename="controller.v2/controller.go:273"

it looks like the namespace isn't being sent and there might be errors related to whether the worker or chief is specified.

[kkasravi]

/assign @swiftdiaries

[kkasravi]

/cc @jlewi

[kkasravi]

Blocked by #696
PR for 696 is available #1084

[jlewi]

I believe this was fixed by #1099.

I just tried it and I'm not hitting this particular issue.