Generate TFJob Python SDK #1103

jinchihe · 2019-12-04T09:49:18Z

Generate TFJob Python SDK

fixes: #167

The PR is includes:

Generated TFJob Python SDK from the way: Go type -> openapi -> swagger file -> TFJob SDK.
Manually update the SDK name to kubeflow-tfjob and from kubeflow import tfjob.
Manually added some Client API such as create, get, update, delete ...
Release frist version kubeflow-tfjob SDK to Pypi: https://pypi.org/project/kubeflow-tfjob/
Drafted some docs and updated Readme file.
Created a sample for the SDK usage.
Added some CI tests for the SDK.
Fixed/work around some testing problems, such as check ClusterRoleBunding and update Daemonset API for GPU driver installing testing#538.
Enhanced openapi gen tag in Update openapi tag for SDK generating common#49

INFO:root:Subprocess output:
INFO:root:Error from server (AlreadyExists): clusterrolebindings.rbac.authorization.k8s.io "default-admin" already exists
INFO:root:Creating /mnt/test-data-volume/kubeflow-tf-operator-presubmit-v1-1103-c43c06e-3472-d38f/output/artifacts/junit_setupcluster.xml
Traceback (most recent call last):
 File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
   "__main__", fname, loader, pkg_name)
 File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
   exec code in run_globals
 File "/mnt/test-data-volume/kubeflow-tf-operator-presubmit-v1-1103-c43c06e-3472-d38f/src/kubeflow/tf-operator/py/kubeflow/tf_operator/de
ploy.py", line 358, in <module>
   main()
 File "/mnt/test-data-volume/kubeflow-tf-operator-presubmit-v1-1103-c43c06e-3472-d38f/src/kubeflow/tf-operator/py/kubeflow/tf_operator/de
ploy.py", line 354, in main
   args.func(args)
 File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 49, in wrapped_f
   return Retrying(*dargs, **dkw).call(f, *args, **kw)
 File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 212, in call
   raise attempt.get()
 File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 247, in get
   six.reraise(self.value[0], self.value[1], self.value[2])
 File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 200, in call
   attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
 File "/mnt/test-data-volume/kubeflow-tf-operator-presubmit-v1-1103-c43c06e-3472-d38f/src/kubeflow/tf-operator/py/kubeflow/tf_operator/de
ploy.py", line 161, in setup_cluster
   "--clusterrole=cluster-admin", "--user=" + account
 File "/mnt/test-data-volume/kubeflow-tf-operator-presubmit-v1-1103-c43c06e-3472-d38f/src/kubeflow/testing/py/kubeflow/testing/util.py",
line 88, in run
   " ".join(command), process.returncode), "\n".join(output))
subprocess.CalledProcessError: Command 'cmd: kubectl create clusterrolebinding default-admin --clusterrole=cluster-admin --user=kubeflow-
[email protected] exited with code 1' returned non-zero exit status 1

TravisBuddy · 2019-12-05T05:40:49Z

Hey @jinchihe,
Your changes look good to me!

View build log

TravisBuddy Request Identifier: c9a2c9f0-1721-11ea-b5b2-a33d2660b61c

TravisBuddy · 2019-12-05T08:39:36Z

Hey @jinchihe,
Your changes look good to me!

View build log

TravisBuddy Request Identifier: c2deba20-173a-11ea-b5b2-a33d2660b61c

TravisBuddy · 2019-12-05T09:27:51Z

Hey @jinchihe,
Your changes look good to me!

View build log

TravisBuddy Request Identifier: 80dbc120-1741-11ea-b5b2-a33d2660b61c

gaocegege · 2019-12-05T09:34:24Z

Why failed to set up cluster, seems nothing with the code change.

INFO:root:Subprocess output:
INFO:root:Error from server (AlreadyExists): clusterrolebindings.rbac.authorization.k8s.io "default-admin" already exists
INFO:root:Creating /mnt/test-data-volume/kubeflow-tf-operator-presubmit-v1-1103-c43c06e-3472-d38f/output/artifacts/junit_setupcluster.xml
Traceback (most recent call last):
 File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
   "__main__", fname, loader, pkg_name)
 File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
   exec code in run_globals
 File "/mnt/test-data-volume/kubeflow-tf-operator-presubmit-v1-1103-c43c06e-3472-d38f/src/kubeflow/tf-operator/py/kubeflow/tf_operator/de
ploy.py", line 358, in <module>
   main()
 File "/mnt/test-data-volume/kubeflow-tf-operator-presubmit-v1-1103-c43c06e-3472-d38f/src/kubeflow/tf-operator/py/kubeflow/tf_operator/de
ploy.py", line 354, in main
   args.func(args)
 File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 49, in wrapped_f
   return Retrying(*dargs, **dkw).call(f, *args, **kw)
 File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 212, in call
   raise attempt.get()
 File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 247, in get
   six.reraise(self.value[0], self.value[1], self.value[2])
 File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 200, in call
   attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
 File "/mnt/test-data-volume/kubeflow-tf-operator-presubmit-v1-1103-c43c06e-3472-d38f/src/kubeflow/tf-operator/py/kubeflow/tf_operator/de
ploy.py", line 161, in setup_cluster
   "--clusterrole=cluster-admin", "--user=" + account
 File "/mnt/test-data-volume/kubeflow-tf-operator-presubmit-v1-1103-c43c06e-3472-d38f/src/kubeflow/testing/py/kubeflow/testing/util.py",
line 88, in run
   " ".join(command), process.returncode), "\n".join(output))
subprocess.CalledProcessError: Command 'cmd: kubectl create clusterrolebinding default-admin --clusterrole=cluster-admin --user=kubeflow-
[email protected] exited with code 1' returned non-zero exit status 1

Did you solve it?

jinchihe · 2019-12-05T09:37:14Z

@gaocegege Yes, I commented out related code to create clusterrolebinding. That works fine now.

TravisBuddy · 2019-12-05T09:37:26Z

Hey @jinchihe,
Your changes look good to me!

View build log

TravisBuddy Request Identifier: d6c470e0-1742-11ea-b5b2-a33d2660b61c

TravisBuddy · 2019-12-05T15:13:57Z

Hey @jinchihe,
Your changes look good to me!

View build log

TravisBuddy Request Identifier: da6bc2a0-1771-11ea-b5b2-a33d2660b61c

jinchihe · 2019-12-06T01:37:27Z

/retest

jinchihe · 2019-12-06T02:21:13Z

Why another problem during set up cluster? :-( Seems also not related with the code change :-(

Traceback (most recent call last):
 File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
   "__main__", fname, loader, pkg_name)
 File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
   exec code in run_globals
 File "/mnt/test-data-volume/kubeflow-tf-operator-presubmit-v1-1103-1b41797-7040-a3f4/src/kubeflow/tf-operator/py/kubeflow/tf_operator/de
ploy.py", line 359, in <module>
   main()
 File "/mnt/test-data-volume/kubeflow-tf-operator-presubmit-v1-1103-1b41797-7040-a3f4/src/kubeflow/tf-operator/py/kubeflow/tf_operator/de
ploy.py", line 355, in main
   args.func(args)
 File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 49, in wrapped_f
   return Retrying(*dargs, **dkw).call(f, *args, **kw)
 File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 212, in call
   raise attempt.get()
 File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 247, in get
   six.reraise(self.value[0], self.value[1], self.value[2])
 File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 200, in call
   attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
 File "/mnt/test-data-volume/kubeflow-tf-operator-presubmit-v1-1103-1b41797-7040-a3f4/src/kubeflow/tf-operator/py/kubeflow/tf_operator/de
ploy.py", line 168, in setup_cluster
   util.setup_cluster(api_client)
...
W
 File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 200, in call
   attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
 File "/mnt/test-data-volume/kubeflow-tf-operator-presubmit-v1-1103-1b41797-7040-a3f4/src/kubeflow/tf-operator/py/kubeflow/tf_operator/de
ploy.py", line 168, in setup_cluster
   util.setup_cluster(api_client)
 File "/mnt/test-data-volume/kubeflow-tf-operator-presubmit-v1-1103-1b41797-7040-a3f4/src/kubeflow/testing/py/kubeflow/testing/util.py",
line 692, in setup_cluster
   install_gpu_drivers(api_client)
 File "/mnt/test-data-volume/kubeflow-tf-operator-presubmit-v1-1103-1b41797-7040-a3f4/src/kubeflow/testing/py/kubeflow/testing/util.py",
line 638, in install_gpu_drivers
   ext_client.create_namespaced_daemon_set(namespace, daemonset_spec)
 File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/apis/extensions_v1beta1_api.py", line 58, in create_namespaced_daemon_set
   (data) = self.create_namespaced_daemon_set_with_http_info(namespace, body, **kwargs)
 File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/apis/extensions_v1beta1_api.py", line 143, in create_namespaced_daemon_se
t_with_http_info
   collection_formats=collection_formats)
 File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/api_client.py", line 321, in call_api
   _return_http_data_only, collection_formats, _preload_content, _request_timeout)
 File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/api_client.py", line 155, in __call_api
   _request_timeout=_request_timeout)
 File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/api_client.py", line 364, in request
   body=body)
 File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/rest.py", line 266, in POST
   body=body)
 File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/rest.py", line 222, in request
   raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'Date': 'Fri, 06 Dec 2019 01:45:37 GMT', 'Audit-Id': '75dd6c49-0b89-4e95-bf67-2c544b71e848', 'Cont
ent-Length': '213', 'Content-Type': 'application/json'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"the API version in the data (apps/v1)
does not match the expected API version (extensions/v1beta1)","reason":"BadRequest","code":400}

TravisBuddy · 2019-12-06T02:44:44Z

Hey @jinchihe,
Your changes look good to me!

View build log

TravisBuddy Request Identifier: 5af30b30-17d2-11ea-b5b2-a33d2660b61c

TravisBuddy · 2019-12-06T05:30:26Z

Hey @jinchihe,
Your changes look good to me!

View build log

TravisBuddy Request Identifier: 80370650-17e9-11ea-95f4-abb8efddad88

TravisBuddy · 2019-12-06T05:59:48Z

Hey @jinchihe,
Your changes look good to me!

View build log

TravisBuddy Request Identifier: 9aec5910-17ed-11ea-95f4-abb8efddad88

TravisBuddy · 2019-12-06T06:39:36Z

Hey @jinchihe,
Your changes look good to me!

View build log

TravisBuddy Request Identifier: 2a125e00-17f3-11ea-95f4-abb8efddad88

TravisBuddy · 2019-12-06T07:55:35Z

Hey @jinchihe,
Your changes look good to me!

View build log

TravisBuddy Request Identifier: c7d82f20-17fd-11ea-95f4-abb8efddad88

TravisBuddy · 2019-12-06T08:14:56Z

Hey @jinchihe,
Your changes look good to me!

View build log

TravisBuddy Request Identifier: 7bfb12e0-1800-11ea-95f4-abb8efddad88

jinchihe · 2019-12-06T08:29:15Z

The CI tests hangs in setup kubeflow step.

INFO:root:level=error msg="handle object: patching object from cluster: merging object with existing state: configmaps \"jupyterhub-confi
g\" is forbidden: User \"[email protected]\" cannot get resource \"configmaps\" in API group \"\" in t
he namespace \"kubeflow\""

TravisBuddy · 2019-12-06T08:39:09Z

Hey @jinchihe,
Your changes look good to me!

View build log

TravisBuddy Request Identifier: ddefcbf0-1803-11ea-95f4-abb8efddad88

TravisBuddy · 2019-12-06T09:45:18Z

Hey @jinchihe,
Your changes look good to me!

View build log

TravisBuddy Request Identifier: 1b82cd60-180d-11ea-95f4-abb8efddad88

gaocegege · 2019-12-06T10:00:09Z

/cc @richardsliu @jlewi Do you have idea about it?

jinchihe · 2019-12-06T14:35:12Z

@gaocegege The above problem already passed now, the only failed is simple-tfjob-tests-v1 timeout. I'm going to take a look.

TravisBuddy · 2019-12-06T14:50:42Z

Hey @jinchihe,
Your changes look good to me!

View build log

TravisBuddy Request Identifier: c55b8a00-1837-11ea-95f4-abb8efddad88

TravisBuddy · 2019-12-06T16:00:12Z

Hey @jinchihe,
Your changes look good to me!

View build log

TravisBuddy Request Identifier: 7ab7d3f0-1841-11ea-95f4-abb8efddad88

TravisBuddy · 2019-12-06T16:29:39Z

Hey @jinchihe,
Your changes look good to me!

View build log

TravisBuddy Request Identifier: 980d25f0-1845-11ea-95f4-abb8efddad88

TravisBuddy · 2019-12-06T16:56:36Z

Hey @jinchihe,
Your changes look good to me!

View build log

TravisBuddy Request Identifier: 5b895eb0-1849-11ea-95f4-abb8efddad88

TravisBuddy · 2019-12-06T17:22:30Z

Hey @jinchihe,
Your changes look good to me!

View build log

TravisBuddy Request Identifier: fa0f28a0-184c-11ea-95f4-abb8efddad88

TravisBuddy · 2019-12-06T17:49:18Z

Hey @jinchihe,
Your changes look good to me!

View build log

TravisBuddy Request Identifier: b842fe70-1850-11ea-95f4-abb8efddad88

TravisBuddy · 2019-12-06T17:55:26Z

Hey @jinchihe,
Your changes look good to me!

View build log

TravisBuddy Request Identifier: 9391b980-1851-11ea-95f4-abb8efddad88

TravisBuddy · 2019-12-06T18:13:00Z

Hey @jinchihe,
Your changes look good to me!

View build log

TravisBuddy Request Identifier: 080caf70-1854-11ea-95f4-abb8efddad88

TravisBuddy · 2019-12-07T03:13:22Z

Hey @jinchihe,
Your changes look good to me!

View build log

TravisBuddy Request Identifier: 84ea0f60-189f-11ea-95f4-abb8efddad88

TravisBuddy · 2019-12-07T04:07:16Z

Hey @jinchihe,
Your changes look good to me!

View build log

TravisBuddy Request Identifier: 0c4895b0-18a7-11ea-95f4-abb8efddad88

TravisBuddy · 2019-12-07T04:30:07Z

Hey @jinchihe,
Your changes look good to me!

View build log

TravisBuddy Request Identifier: 3e248870-18aa-11ea-9822-dbdeb0b9a0b2

TravisBuddy · 2019-12-07T04:35:30Z

Hey @jinchihe,
Your changes look good to me!

View build log

TravisBuddy Request Identifier: fdf61010-18aa-11ea-9822-dbdeb0b9a0b2

jinchihe · 2019-12-07T05:25:23Z

/cc @gaocegege @richardsliu @jlewi @animeshsingh

The PR is ready for reviewing now. Thanks!

Next, will consider how to merge some useful functions in current py/kubeflow, especially py/kubeflow/tf_operator/tf_job_client.py, so that user can use that directly once install kubeflow-tfjob SDK from pip.

sdk/python/docs/TFJobClient.md

gaocegege · 2019-12-08T03:06:00Z

@jinchihe Thanks for your awesome contribution!

animeshsingh · 2019-12-08T21:32:46Z

Thanks @jinchihe - great work!

TravisBuddy · 2019-12-09T01:39:25Z

Hey @jinchihe,
Your changes look good to me!

View build log

TravisBuddy Request Identifier: b904b620-1a24-11ea-a744-67689bca182d

gaocegege · 2019-12-09T01:40:17Z

/lgtm

johnugeorge · 2019-12-09T08:03:33Z

/approve

k8s-ci-robot · 2019-12-09T08:03:38Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: johnugeorge

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [johnugeorge]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added do-not-merge/work-in-progress size/XXL labels Dec 4, 2019

k8s-ci-robot requested review from ChanYiLin and jimexist December 4, 2019 09:49

jinchihe mentioned this pull request Dec 6, 2019

Failed set up cluster while installing GPU Drivers kubeflow/testing#537

Closed

jinchihe mentioned this pull request Dec 7, 2019

Update openapi tag for SDK generating kubeflow/common#49

Merged

jinchihe changed the title ~~WIP: generate TFJob Python SDK~~ Generate TFJob Python SDK Dec 7, 2019

k8s-ci-robot removed the do-not-merge/work-in-progress label Dec 7, 2019

k8s-ci-robot requested review from animeshsingh, gaocegege, jlewi and richardsliu December 7, 2019 05:25

nrchakradhar reviewed Dec 7, 2019

View reviewed changes

sdk/python/docs/TFJobClient.md Outdated Show resolved Hide resolved

generate TFJob Python SDK

363a0d4

k8s-ci-robot assigned gaocegege Dec 9, 2019

k8s-ci-robot added the lgtm label Dec 9, 2019

k8s-ci-robot added the approved label Dec 9, 2019

k8s-ci-robot merged commit f3509e6 into kubeflow:master Dec 9, 2019

Generate TFJob Python SDK #1103

Generate TFJob Python SDK #1103

Conversation

jinchihe commented Dec 4, 2019 • edited Loading

The PR is includes:

Next:

coveralls commented Dec 4, 2019 • edited Loading

TravisBuddy commented Dec 5, 2019

TravisBuddy Request Identifier: d3419b40-170c-11ea-93d3-257117e2c129

jinchihe commented Dec 5, 2019

TravisBuddy commented Dec 5, 2019

TravisBuddy Request Identifier: c9a2c9f0-1721-11ea-b5b2-a33d2660b61c

TravisBuddy commented Dec 5, 2019

TravisBuddy Request Identifier: c2deba20-173a-11ea-b5b2-a33d2660b61c

TravisBuddy commented Dec 5, 2019

TravisBuddy Request Identifier: 80dbc120-1741-11ea-b5b2-a33d2660b61c

gaocegege commented Dec 5, 2019

jinchihe commented Dec 5, 2019

TravisBuddy commented Dec 5, 2019

TravisBuddy Request Identifier: d6c470e0-1742-11ea-b5b2-a33d2660b61c

TravisBuddy commented Dec 5, 2019

TravisBuddy Request Identifier: da6bc2a0-1771-11ea-b5b2-a33d2660b61c

jinchihe commented Dec 6, 2019

jinchihe commented Dec 6, 2019

TravisBuddy commented Dec 6, 2019

TravisBuddy Request Identifier: 5af30b30-17d2-11ea-b5b2-a33d2660b61c

TravisBuddy commented Dec 6, 2019

TravisBuddy Request Identifier: 80370650-17e9-11ea-95f4-abb8efddad88

TravisBuddy commented Dec 6, 2019

TravisBuddy Request Identifier: 9aec5910-17ed-11ea-95f4-abb8efddad88

TravisBuddy commented Dec 6, 2019

TravisBuddy Request Identifier: 2a125e00-17f3-11ea-95f4-abb8efddad88

TravisBuddy commented Dec 6, 2019

TravisBuddy Request Identifier: c7d82f20-17fd-11ea-95f4-abb8efddad88

TravisBuddy commented Dec 6, 2019

TravisBuddy Request Identifier: 7bfb12e0-1800-11ea-95f4-abb8efddad88

jinchihe commented Dec 6, 2019

TravisBuddy commented Dec 6, 2019

TravisBuddy Request Identifier: ddefcbf0-1803-11ea-95f4-abb8efddad88

TravisBuddy commented Dec 6, 2019

TravisBuddy Request Identifier: 1b82cd60-180d-11ea-95f4-abb8efddad88

gaocegege commented Dec 6, 2019

jinchihe commented Dec 6, 2019

TravisBuddy commented Dec 6, 2019

TravisBuddy Request Identifier: c55b8a00-1837-11ea-95f4-abb8efddad88

TravisBuddy commented Dec 6, 2019

TravisBuddy Request Identifier: 7ab7d3f0-1841-11ea-95f4-abb8efddad88

TravisBuddy commented Dec 6, 2019

TravisBuddy Request Identifier: 980d25f0-1845-11ea-95f4-abb8efddad88

TravisBuddy commented Dec 6, 2019

TravisBuddy Request Identifier: 5b895eb0-1849-11ea-95f4-abb8efddad88

TravisBuddy commented Dec 6, 2019

TravisBuddy Request Identifier: fa0f28a0-184c-11ea-95f4-abb8efddad88

TravisBuddy commented Dec 6, 2019

TravisBuddy Request Identifier: b842fe70-1850-11ea-95f4-abb8efddad88

TravisBuddy commented Dec 6, 2019

TravisBuddy Request Identifier: 9391b980-1851-11ea-95f4-abb8efddad88

TravisBuddy commented Dec 6, 2019

TravisBuddy Request Identifier: 080caf70-1854-11ea-95f4-abb8efddad88

TravisBuddy commented Dec 7, 2019

TravisBuddy Request Identifier: 84ea0f60-189f-11ea-95f4-abb8efddad88

TravisBuddy commented Dec 7, 2019

TravisBuddy Request Identifier: 0c4895b0-18a7-11ea-95f4-abb8efddad88

TravisBuddy commented Dec 7, 2019

TravisBuddy Request Identifier: 3e248870-18aa-11ea-9822-dbdeb0b9a0b2

TravisBuddy commented Dec 7, 2019

TravisBuddy Request Identifier: fdf61010-18aa-11ea-9822-dbdeb0b9a0b2

jinchihe commented Dec 7, 2019

gaocegege commented Dec 8, 2019

animeshsingh commented Dec 8, 2019

TravisBuddy commented Dec 9, 2019

TravisBuddy Request Identifier: b904b620-1a24-11ea-a744-67689bca182d

gaocegege commented Dec 9, 2019

johnugeorge commented Dec 9, 2019

k8s-ci-robot commented Dec 9, 2019

jinchihe commented Dec 4, 2019 •

edited

Loading

coveralls commented Dec 4, 2019 •

edited

Loading