Update scripts to generate sdk for all frameworks #1389

Jeffwan · 2021-08-29T22:15:09Z

Scripts to generate sdk for all frameworks

@jinchihe @andreyvelich @kubeflow/wg-training-leads

hack/python-sdk/main.go

andreyvelich · 2021-08-30T15:29:46Z

hack/python-sdk/main.go

+	switch framework {
+	case "tensorflow":
+		oAPIDefs = tfjob.GetOpenAPIDefinitions(func(name string) spec.Ref {
+			return spec.MustCreateRef("#/definitions/" + common.EscapeJsonPointer(swaggify(name, framework)))
+		})
+	case "pytorch":
+		oAPIDefs = pytorchJob.GetOpenAPIDefinitions(func(name string) spec.Ref {
+			return spec.MustCreateRef("#/definitions/" + common.EscapeJsonPointer(swaggify(name, framework)))
+		})
+	case "mxnet":
+		oAPIDefs = mxJob.GetOpenAPIDefinitions(func(name string) spec.Ref {
+			return spec.MustCreateRef("#/definitions/" + common.EscapeJsonPointer(swaggify(name, framework)))
+		})
+	case "xgboost":
+		oAPIDefs = xgboostJob.GetOpenAPIDefinitions(func(name string) spec.Ref {
+			return spec.MustCreateRef("#/definitions/" + common.EscapeJsonPointer(swaggify(name, framework)))
+		})
+	}


@kubeflow/wg-training-leads Do we want to have to have multiple or one Swagger for each framework?

You mean compress all swagger into a big one?

Yes, I am just thinking what is the best way to locate these swaggers.

We still need to generate each for different framework. Then I mixin into one and use it to generate SDK.

andreyvelich · 2021-08-30T15:37:21Z

hack/python-sdk/main.go

-	oAPIDefs := tfjob.GetOpenAPIDefinitions(func(name string) spec.Ref {
-		return spec.MustCreateRef("#/definitions/" + common.EscapeJsonPointer(swaggify(name)))
-	})


@Jeffwan We might need to generate OpenAPI Definition for our APIs to avoid corev1 definition as you mentioned here: #1389 (comment).
Then, our Swagger will have only Training related APIs.
What do you think ?

@andreyvelich

Yeah. I think this is the problem. Not sure what corev1 are populated. Do you have any ideas? Seems something is misconfigured in this hack/python-sdk/main.go?

Do we need to use code-generator to generate it ? Similar to this: https://github.com/kubeflow/katib/blob/master/hack/update-openapigen.sh#L49-L53.

Yeah. Same tool. I update the --input-dirs to exclude those unrelated files. Thanks @andreyvelich

jinchihe · 2021-09-02T07:08:17Z

@Jeffwan are you want to copy the script to each framework repo? Personally I think we should generate swagger file for each repo.
Others LGTM.

openapi-gen input-dirs includes a few k8s.io/api and k8s.io/apimachinery packages which makes openapi_generated.go very large. Swagger files created by them are large as well. This change help remove unnecessary references. Signed-off-by: Jiaxin Shan <[email protected]>

1. Update gen-sdk.sh and main.go to support all frameworks 2. Generate swagger.json for all frameworks Signed-off-by: Jiaxin Shan <[email protected]>

The swagger.json can not be merged smartly. I manually update info.description, info.title in hack/python-sdk/swagger.json Signed-off-by: Jiaxin Shan <[email protected]>

alembiewski · 2021-09-21T17:04:21Z

The PR hasn't been updated for quite some time now. Are there any plans to finalize and merge this one?

Jeffwan · 2021-09-21T18:40:35Z

@alembiewski Hi, I am kind of overwhelmed recently for some internal work. If anyone has time, feel free to pick it up. otherwise, I will find some time to make it.

alembiewski · 2021-09-22T12:06:00Z

@Jeffwan, trying to understand what needs to be done in terms of tooling here. I was able to generate Python models using the script, provided in this PR. I could start working on updating/adding the models to sdk/python and update the existing API clients (we should probably copy the pytorch_job client here as well) to operate with the new models, updating the tests etc. Does that sound good?

Jeffwan · 2021-09-24T05:22:34Z

@alembiewski Can you help review this change as well? I don't have further changes for this PR.

alembiewski · 2021-09-25T20:23:32Z

Thanks for updating the tooling, @Jeffwan! LGTM!

Jeffwan · 2021-09-25T22:25:29Z

@andreyvelich Alex has used update tools to generate sdk in #1420. Please help double check if further changes are needed.

andreyvelich · 2021-09-28T18:52:33Z

/lgtm

Jeffwan · 2021-09-28T19:25:27Z

/approve

google-oss-robot · 2021-09-28T19:25:44Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Jeffwan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [Jeffwan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

google-oss-robot added the do-not-merge/work-in-progress label Aug 29, 2021

google-oss-robot requested review from jinchihe and terrytangyuan August 29, 2021 22:15

google-oss-robot added the size/M label Aug 29, 2021

Jeffwan mentioned this pull request Aug 29, 2021

Python SDK for Kubeflow Training Operator #1380

Closed

andreyvelich reviewed Aug 30, 2021

View reviewed changes

Jeffwan added 3 commits September 5, 2021 18:37

Update OpenAPI generation scripts

c28712a

1. Update gen-sdk.sh and main.go to support all frameworks 2. Generate swagger.json for all frameworks Signed-off-by: Jiaxin Shan <[email protected]>

Merge swagger files into one

893acfb

The swagger.json can not be merged smartly. I manually update info.description, info.title in hack/python-sdk/swagger.json Signed-off-by: Jiaxin Shan <[email protected]>

Jeffwan force-pushed the sdk branch from d34732a to 893acfb Compare September 6, 2021 02:24

google-oss-robot added size/XXL and removed size/M labels Sep 6, 2021

alembiewski mentioned this pull request Sep 22, 2021

Add Python SDK for Kubeflow Training Operator #1420

Merged

Jeffwan changed the title ~~[WIP] Update scripts to generate sdk for all frameworks~~ Update scripts to generate sdk for all frameworks Sep 24, 2021

google-oss-robot removed the do-not-merge/work-in-progress label Sep 24, 2021

google-oss-robot assigned andreyvelich Sep 28, 2021

google-oss-robot added the lgtm label Sep 28, 2021

google-oss-robot added the approved label Sep 28, 2021

google-oss-robot merged commit f1b2b13 into kubeflow:master Sep 28, 2021

Jeffwan mentioned this pull request Sep 29, 2021

Cut official release of 1.3.0 #1425

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update scripts to generate sdk for all frameworks #1389

Update scripts to generate sdk for all frameworks #1389

Jeffwan commented Aug 29, 2021 •

edited

Loading

andreyvelich Aug 30, 2021

Jeffwan Aug 30, 2021

andreyvelich Aug 31, 2021

Jeffwan Sep 6, 2021 •

edited

Loading

andreyvelich Aug 30, 2021

Jeffwan Aug 30, 2021 •

edited

Loading

andreyvelich Aug 31, 2021

Jeffwan Sep 6, 2021

jinchihe commented Sep 2, 2021

alembiewski commented Sep 21, 2021

Jeffwan commented Sep 21, 2021

alembiewski commented Sep 22, 2021

Jeffwan commented Sep 24, 2021

alembiewski commented Sep 25, 2021

Jeffwan commented Sep 25, 2021

andreyvelich commented Sep 28, 2021

Jeffwan commented Sep 28, 2021

google-oss-robot commented Sep 28, 2021

Update scripts to generate sdk for all frameworks #1389

Update scripts to generate sdk for all frameworks #1389

Conversation

Jeffwan commented Aug 29, 2021 • edited Loading

andreyvelich Aug 30, 2021

Choose a reason for hiding this comment

Jeffwan Aug 30, 2021

Choose a reason for hiding this comment

andreyvelich Aug 31, 2021

Choose a reason for hiding this comment

Jeffwan Sep 6, 2021 • edited Loading

Choose a reason for hiding this comment

andreyvelich Aug 30, 2021

Choose a reason for hiding this comment

Jeffwan Aug 30, 2021 • edited Loading

Choose a reason for hiding this comment

andreyvelich Aug 31, 2021

Choose a reason for hiding this comment

Jeffwan Sep 6, 2021

Choose a reason for hiding this comment

jinchihe commented Sep 2, 2021

alembiewski commented Sep 21, 2021

Jeffwan commented Sep 21, 2021

alembiewski commented Sep 22, 2021

Jeffwan commented Sep 24, 2021

alembiewski commented Sep 25, 2021

Jeffwan commented Sep 25, 2021

andreyvelich commented Sep 28, 2021

Jeffwan commented Sep 28, 2021

google-oss-robot commented Sep 28, 2021

Jeffwan commented Aug 29, 2021 •

edited

Loading

Jeffwan Sep 6, 2021 •

edited

Loading

Jeffwan Aug 30, 2021 •

edited

Loading