no matches for tensorflow.org/, Kind=TfJob

[DjangoPeng]

I have set a configMap for bare metal environment, and got a problem when I created a TF job. I'm sure that I followed the instructions of README.md file step by step. I attached the error log as below:

error: unable to recognize "tf_job.yaml": no matches for tensorflow.org/, Kind=TfJob

[jlewi]

This error indicates that the Custom Resource for TfJob could not be created.

Please take a look at this comment for some initial steps to try to identify the particular cause.

There's a high likelihood that you are hitting the same issue with the ConfigMap not being found as in #149.

[DjangoPeng]

Thanks for replying. Let me have a look at the comment and refs.

[DjangoPeng]

@jlewi I checked my case according to your comment in #149 . The TF Job CRD is not created actually. But, the deployment and pod of TF Job are running. I took a look at the logs of TF Job pod, and found that it may be a permission problem of namespace.

E1127 03:01:44.508964       1 election.go:226] error retrieving resource lock default/tf-operator: endpoints "tf-operator" is forbidden: User "system:serviceaccount:default:default" cannot get endpoints in the namespace "default"

[DjangoPeng]

I forgot to set rbac.install=true when I run the helm install instruction. After setting that, the tf-operator is running well. But, I failed to docker pull gcr.io/tf-on-k8s-dogfood/tf_sample:dc944ff. Could you have a look? @jlewi

Error response from daemon: Get https://gcr.io/v1/_ping: dial tcp 64.233.189.82:443: i/o timeout

[jlewi]

I confirmed the docker image exists. I also confirmed the bucket is public. Could this be an issue with gcr.io being blocked by the firewall?

In the meantime you should be able to build the Docker image from the repo using https://github.com/tensorflow/k8s/blob/master/examples/tf_sample/build_and_push.py

[DjangoPeng]

Ok, I'm going to build the Docker image. Thanks!

[DjangoPeng]

@jlewi I noticed that we directly call subprocess to build and push Docker image in build_and_push.py file. I think replace it with Docker SDK would be better and more portable. What do you think?

[jimexist]

gcr.io is blocked, one needs VPN to access.

[DjangoPeng]

Anyway, the issue of no matches for tensorflow.org/, Kind=TfJob has been figured out. Close this issue. Others who facing same problem can trace #174 , and I will create a PR to add troubleshooting guide this week.

[wydwww]

Hi @jlewi Is there any solution for the same issue when tf-job-operator is deployed by Kubeflow instead of helm?
New issue is kubeflow/kubeflow#109
Thanks

[alien-xt]

I have same problem, how can i fix?

describe:
"
the provided version "kubeflow.org/v1alpha2" has no relevant versions: group kubeflow.org has not been registered
no matches for kubeflow.org/, Kind=TFJob
"