1322: Modified manifests to use all-in-one training-operator #1346

deepak-muley · 2021-08-12T19:31:35Z

Actions taken:
- replaced tf-job-operator => training-operator
- replaced kubeflow-tfjobs- => kubeflow-training-
- moved crds for mxjobs, tgjobs, pytorchjobs and xgboostjobs from
config/crd/bases to manifests/base/ and prefixed them with crd_
- merged config/manager deployment with manifests/base/deployment
Ref: #1322
Testing steps:

Deleted exsting operator deployments for tfjob, pytorch etc
Built the training-operator as follows
docker build -f build/images/training-operator/Dockerfile -t deepakmuley/kubeflow-training-operator:1 .
updated the manifests to use this new image
deployed the training-operator
Build and ran tfjob as per developer guide
k get pods -n test-tf-operator
NAME READY STATUS RESTARTS AGE
dist-mnist-for-e2e-test-ps-0 1/1 Running 0 77s
dist-mnist-for-e2e-test-ps-1 1/1 Running 0 77s
dist-mnist-for-e2e-test-worker-0 1/1 Running 0 77s
dist-mnist-for-e2e-test-worker-1 1/1 Running 0 77s
dist-mnist-for-e2e-test-worker-2 1/1 Running 0 77s
dist-mnist-for-e2e-test-worker-3 1/1 Running 0 77s

Actions taken: - replaced tf-job-operator => training-operator - replaced kubeflow-tfjobs- => kubeflow-training- - moved crds for mxjobs, tgjobs, pytorchjobs and xgboostjobs from config/crd/bases to manifests/base/ and prefixed them with crd_ Ref: kubeflow#1322 Testing steps: To be added Work in Progress

google-cla · 2021-08-12T19:31:41Z

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.

What to do if you already signed the CLA

Individual signers

It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

aws-kf-ci-bot · 2021-08-12T19:31:48Z

Hi @deepak-muley. Thanks for your PR.

I'm waiting for a kubeflow member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

deepak-muley · 2021-08-12T19:33:17Z

@googlebot I signed it!

johnugeorge · 2021-08-12T19:33:19Z

/ok-to-test

johnugeorge · 2021-08-12T19:35:27Z

/cc @Jeffwan /cc @gaocegege

google-oss-robot · 2021-08-12T19:35:29Z

@johnugeorge: GitHub didn't allow me to request PR reviews from the following users: /cc.

Note that only kubeflow members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/cc @Jeffwan /cc @gaocegege

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

deepak-muley · 2021-08-12T19:42:57Z

/hold

manifests/base/cluster-role.yaml

Jeffwan · 2021-08-12T21:24:15Z

Please help compare https://github.com/kubeflow/tf-operator/blob/c60dabb7daa3ff9ffb4eecfc24ee4a39bce4b7ca/config/manager/manager.yaml#L7-L9 and your deployment.yaml

andreyvelich

Thank you for doing this @deepak-muley!
@kubeflow/wg-training-leads Since we are using /manifests folder for our deployment, do we need /config folder ?

manifests/base/cluster-role.yaml

manifests/base/kustomization.yaml

manifests/overlays/kubeflow/kustomization.yaml

manifests/overlays/standalone/kustomization.yaml

Training operator was found to be working <pre> k -n kubeflow logs -f training-operator-694766989-pp2j4 I0812 21:43:24.739862 1 request.go:645] Throttling request took 1.048945631s, request: GET:https://172.19.0.1:443/apis/networking.k8s.io/v1?timeout=32s 2021-08-12T21:43:25.694Z INFO controller-runtime.metrics metrics server is starting to listen {"addr": ":8080"} 2021-08-12T21:43:25.790Z INFO setup starting manager 2021-08-12T21:43:25.790Z INFO controller-runtime.manager starting metrics server {"path": "/metrics"} 2021-08-12T21:43:25.790Z INFO controller-runtime.manager.controller.tf-operator Starting EventSource {"source": "kind source: /, Kind="} 2021-08-12T21:43:25.790Z INFO controller-runtime.manager.controller.mxnet-operator Starting EventSource {"source": "kind source: /, Kind="} 2021-08-12T21:43:25.791Z INFO controller-runtime.manager.controller.pytorchjob-operator Starting EventSource {"source": "kind source: /, Kind="} 2021-08-12T21:43:25.791Z INFO controller-runtime.manager.controller.xgboostjob-operator Starting EventSource {"source": "kind source: /, Kind="} 2021-08-12T21:43:26.289Z INFO controller-runtime.manager.controller.xgboostjob-operator Starting EventSource {"source": "kind source: /, Kind="} 2021-08-12T21:43:26.294Z INFO controller-runtime.manager.controller.pytorchjob-operator Starting EventSource {"source": "kind source: /, Kind="} 2021-08-12T21:43:26.589Z INFO controller-runtime.manager.controller.mxnet-operator Starting EventSource {"source": "kind source: /, Kind="} 2021-08-12T21:43:26.688Z INFO controller-runtime.manager.controller.tf-operator Starting EventSource {"source": "kind source: /, Kind="} 2021-08-12T21:43:26.889Z INFO controller-runtime.manager.controller.tf-operator Starting EventSource {"source": "kind source: /, Kind="} 2021-08-12T21:43:26.889Z INFO controller-runtime.manager.controller.pytorchjob-operator Starting EventSource {"source": "kind source: /, Kind="} 2021-08-12T21:43:26.890Z INFO controller-runtime.manager.controller.xgboostjob-operator Starting EventSource {"source": "kind source: /, Kind="} 2021-08-12T21:43:26.890Z INFO controller-runtime.manager.controller.mxnet-operator Starting EventSource {"source": "kind source: /, Kind="} 2021-08-12T21:43:26.990Z INFO controller-runtime.manager.controller.xgboostjob-operator Starting Controller 2021-08-12T21:43:26.990Z INFO controller-runtime.manager.controller.tf-operator Starting Controller 2021-08-12T21:43:26.990Z INFO controller-runtime.manager.controller.tf-operator Starting workers {"worker count": 1} 2021-08-12T21:43:26.990Z INFO controller-runtime.manager.controller.pytorchjob-operator Starting Controller 2021-08-12T21:43:26.991Z INFO controller-runtime.manager.controller.xgboostjob-operator Starting workers {"worker count": 1} 2021-08-12T21:43:26.991Z INFO controller-runtime.manager.controller.pytorchjob-operator Starting workers {"worker count": 1} 2021-08-12T21:43:26.991Z INFO controller-runtime.manager.controller.mxnet-operator Starting Controller 2021-08-12T21:43:26.991Z INFO controller-runtime.manager.controller.mxnet-operator Starting workers {"worker count": 1} </pre>

deepak-muley · 2021-08-12T22:23:30Z

/hold cancel

terrytangyuan · 2021-08-12T23:26:33Z

manifests/base/deployment.yaml

-      serviceAccountName: tf-job-operator
+        - command:
+            - /manager
+          # disable leader-elect now


Remove the lines that are no longer needed?

@Jeffwan should we be removing it or can be handled through another tracker which enables leader election. where do i find details of why it was disabled?

This will be needed for beta release. @deepak-muley please create an issue to track it.

terrytangyuan

.

- now controller-gen generates the crds directly in manifests/base instead of config/crd/bases - updated setup-training-operator.sh to use manifests/overlays/standalone

Jeffwan · 2021-08-13T06:10:11Z

/lgtm
/approve

google-oss-robot · 2021-08-13T06:10:18Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Jeffwan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [Jeffwan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

aws-kf-ci-bot added the needs-ok-to-test label Aug 12, 2021

google-oss-robot added the size/XXL label Aug 12, 2021

google-oss-robot requested review from jinchihe and terrytangyuan August 12, 2021 19:32

aws-kf-ci-bot added ok-to-test and removed needs-ok-to-test labels Aug 12, 2021

google-oss-robot requested review from Jeffwan and gaocegege August 12, 2021 19:35

google-oss-robot added the do-not-merge/hold label Aug 12, 2021

Jeffwan reviewed Aug 12, 2021

View reviewed changes

manifests/base/cluster-role.yaml Outdated Show resolved Hide resolved

andreyvelich reviewed Aug 12, 2021

View reviewed changes

deepak-muley added 2 commits August 12, 2021 14:46

1322: incorporated review comments - added all resources in ClusterRole

920986a

google-oss-robot removed the do-not-merge/hold label Aug 12, 2021

deepak-muley changed the title ~~1322: Modified manifests to use all-in-one training-operator WIP~~ 1322: Modified manifests to use all-in-one training-operator Aug 12, 2021

terrytangyuan reviewed Aug 12, 2021

View reviewed changes

deepak-muley added 2 commits August 12, 2021 21:49

1322: incorporated review comments

c693173

- now controller-gen generates the crds directly in manifests/base instead of config/crd/bases - updated setup-training-operator.sh to use manifests/overlays/standalone

1322: removed config/crd/bases as its now getting generated in manifests

4c81ab5

google-oss-robot removed the size/XXL label Aug 13, 2021

google-oss-robot added the size/L label Aug 13, 2021

1322: incorporated review comments related to using separate role files

eb3b537

google-oss-robot added size/XL and removed size/L labels Aug 13, 2021

1322: removed image name replacement

146364a

Jeffwan approved these changes Aug 13, 2021

View reviewed changes

google-oss-robot added the approved label Aug 13, 2021

google-oss-robot assigned Jeffwan Aug 13, 2021

google-oss-robot added the lgtm label Aug 13, 2021

google-oss-robot merged commit 46c5864 into kubeflow:master Aug 13, 2021

Jeffwan mentioned this pull request Aug 13, 2021

Clean up manifests and remove unused files #1349

Merged

deepak-muley mentioned this pull request Aug 13, 2021

Enable leader election in controller manager using controllermanagerconfig #1350

Closed

johnugeorge mentioned this pull request Aug 23, 2021

Adding Deepak to member list kubeflow/internal-acls#486

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

1322: Modified manifests to use all-in-one training-operator #1346

1322: Modified manifests to use all-in-one training-operator #1346

deepak-muley commented Aug 12, 2021 •

edited

Loading

google-cla bot commented Aug 12, 2021

aws-kf-ci-bot commented Aug 12, 2021

deepak-muley commented Aug 12, 2021

johnugeorge commented Aug 12, 2021

johnugeorge commented Aug 12, 2021

google-oss-robot commented Aug 12, 2021

deepak-muley commented Aug 12, 2021

Jeffwan commented Aug 12, 2021

andreyvelich left a comment

deepak-muley commented Aug 12, 2021

terrytangyuan Aug 12, 2021

deepak-muley Aug 13, 2021

Jeffwan Aug 13, 2021 •

edited

Loading

terrytangyuan left a comment •

edited

Loading

Jeffwan commented Aug 13, 2021

google-oss-robot commented Aug 13, 2021

1322: Modified manifests to use all-in-one training-operator #1346

1322: Modified manifests to use all-in-one training-operator #1346

Conversation

deepak-muley commented Aug 12, 2021 • edited Loading

google-cla bot commented Aug 12, 2021

What to do if you already signed the CLA

Individual signers

Corporate signers

aws-kf-ci-bot commented Aug 12, 2021

deepak-muley commented Aug 12, 2021

johnugeorge commented Aug 12, 2021

johnugeorge commented Aug 12, 2021

google-oss-robot commented Aug 12, 2021

deepak-muley commented Aug 12, 2021

Jeffwan commented Aug 12, 2021

andreyvelich left a comment

Choose a reason for hiding this comment

deepak-muley commented Aug 12, 2021

terrytangyuan Aug 12, 2021

Choose a reason for hiding this comment

deepak-muley Aug 13, 2021

Choose a reason for hiding this comment

Jeffwan Aug 13, 2021 • edited Loading

Choose a reason for hiding this comment

terrytangyuan left a comment • edited Loading

Choose a reason for hiding this comment

Jeffwan commented Aug 13, 2021

google-oss-robot commented Aug 13, 2021

deepak-muley commented Aug 12, 2021 •

edited

Loading

Jeffwan Aug 13, 2021 •

edited

Loading

terrytangyuan left a comment •

edited

Loading