Feat: switch to v1 API paths for backend. #1121

ScrapCodes · 2023-01-17T13:43:34Z

Which issue is resolved by this Pull Request:
Resolves #1081

Description of your changes:

Environment tested:

Python Version (use python --version):
Tekton Version (use tkn version):
Kubernetes Version (use kubectl version):
OS (e.g. from /etc/os-release):

Checklist:

The title for your pull request (PR) should follow our title convention. Learn more about the pull request title convention used in this repository.

google-oss-prow · 2023-01-17T13:43:47Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ScrapCodes

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [ScrapCodes]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ScrapCodes · 2023-01-18T14:27:45Z

currently trying to figure out, why it is querying v1beta1

    at emitErrorCloseNT (internal/streams/destroy.js:74:3)
    at processTicksAndRejections (internal/process/task_queues.js:82:21)
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:1) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
(node:1) UnhandledPromiseRejectionWarning: FetchError: request to http://metadata/computeMetadata/v1/project/project-id failed, reason: getaddrinfo ENOTFOUND metadata
    at ClientRequest.<anonymous> (/server/node_modules/node-fetch/lib/index.js:1461:11)
    at ClientRequest.emit (events.js:400:28)
    at Socket.socketErrorListener (_http_client.js:475:9)
    at Socket.emit (events.js:400:28)
    at emitErrorNT (internal/streams/destroy.js:106:8)
    at emitErrorCloseNT (internal/streams/destroy.js:74:3)
    at processTicksAndRejections (internal/process/task_queues.js:82:21)
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2)
GET /apis/v1beta1/healthz
GET /apis/v1beta1/healthz
GET /apis/v1beta1/healthz
GET /apis/v1beta1/healthz
GET /apis/v1beta1/healthz
GET /apis/v1beta1/healthz
GET /apis/v1beta1/experiments?page_token=&page_size=10&sort_by=created_at%20desc&filter=%257B%2522predicates%2522%253A%255B%257B%2522key%2522%253A%2522storage_state%2522%252C%2522op%2522%253A%2522NOT_EQUALS%2522%252C%2522string_value%2522%253A%2522STORAGESTATE_ARCHIVED%2522%257D%255D%257D
Proxied request:  /apis/v1beta1/experiments?page_token=&page_size=10&sort_by=created_at%20desc&filter=%257B%2522predicates%2522%253A%255B%257B%2522key%2522%253A%2522storage_state%2522%252C%2522op%2522%253A%2522NOT_EQUALS%2522%252C%2522string_value%2522%253A%2522STORAGESTATE_ARCHIVED%2522%257D%255D%257D
GET /apis/v1beta1/healthz
GET /apis/v1beta1/healthz
GET /apis/v1beta1/healthz
GET /apis/v1beta1/healthz

Tomcli · 2023-01-18T17:43:27Z

currently trying to figure out, why it is querying v1beta1

    at emitErrorCloseNT (internal/streams/destroy.js:74:3)
    at processTicksAndRejections (internal/process/task_queues.js:82:21)
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:1) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
(node:1) UnhandledPromiseRejectionWarning: FetchError: request to http://metadata/computeMetadata/v1/project/project-id failed, reason: getaddrinfo ENOTFOUND metadata
    at ClientRequest.<anonymous> (/server/node_modules/node-fetch/lib/index.js:1461:11)
    at ClientRequest.emit (events.js:400:28)
    at Socket.socketErrorListener (_http_client.js:475:9)
    at Socket.emit (events.js:400:28)
    at emitErrorNT (internal/streams/destroy.js:106:8)
    at emitErrorCloseNT (internal/streams/destroy.js:74:3)
    at processTicksAndRejections (internal/process/task_queues.js:82:21)
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2)
GET /apis/v1beta1/healthz
GET /apis/v1beta1/healthz
GET /apis/v1beta1/healthz
GET /apis/v1beta1/healthz
GET /apis/v1beta1/healthz
GET /apis/v1beta1/healthz
GET /apis/v1beta1/experiments?page_token=&page_size=10&sort_by=created_at%20desc&filter=%257B%2522predicates%2522%253A%255B%257B%2522key%2522%253A%2522storage_state%2522%252C%2522op%2522%253A%2522NOT_EQUALS%2522%252C%2522string_value%2522%253A%2522STORAGESTATE_ARCHIVED%2522%257D%255D%257D
Proxied request:  /apis/v1beta1/experiments?page_token=&page_size=10&sort_by=created_at%20desc&filter=%257B%2522predicates%2522%253A%255B%257B%2522key%2522%253A%2522storage_state%2522%252C%2522op%2522%253A%2522NOT_EQUALS%2522%252C%2522string_value%2522%253A%2522STORAGESTATE_ARCHIVED%2522%257D%255D%257D
GET /apis/v1beta1/healthz
GET /apis/v1beta1/healthz
GET /apis/v1beta1/healthz
GET /apis/v1beta1/healthz

which microservice is this?

Tomcli · 2023-01-18T17:45:14Z

@ScrapCodes you probably want to update the deployment manifests too, seems like you are still deploying with the health checking pinning the v1beta1 api.
https://github.com/kubeflow/kfp-tekton/blob/559f8b6df04629cd2be43935f30268b406385001/manifests/kustomize/base/pipeline/ml-pipeline-apiserver-deployment.yaml

ScrapCodes · 2023-01-20T10:20:06Z

In the package frontend/ , how does npm run apis figure out where to find swagger files.

npm run apis

> [email protected] apis /Users/prashantsharma/work/kfp-tekton/frontend
> npm run apis:experiment && npm run apis:job && npm run apis:pipeline && npm run apis:run && npm run apis:filter && npm run apis:visualization && npm run apis:auth


> [email protected] apis:experiment /Users/prashantsharma/work/kfp-tekton/frontend
> java -jar swagger-codegen-cli.jar generate -i ../backend/api/swagger/experiment.swagger.json -l typescript-fetch -o ./src/apis/experiment -c ./swagger-config.json

[main] INFO io.swagger.parser.Swagger20Parser - reading from ../backend/api/swagger/experiment.swagger.json
[main] INFO io.swagger.parser.Swagger20Parser - reading from ../backend/api/swagger/experiment.swagger.json
[main] ERROR io.swagger.parser.SwaggerCompatConverter - failed to read resource listing
java.lang.RuntimeException: Could not find ../backend/api/swagger/experiment.swagger.json on the classpath
	at io.swagger.parser.util.ClasspathHelper.loadFileFromClasspath(ClasspathHelper.java:31)
	at io.swagger.parser.SwaggerCompatConverter.readResourceListing(SwaggerCompatConverter.java:206)
	at io.swagger.parser.SwaggerCompatConverter.read(SwaggerCompatConverter.java:122)
	at io.swagger.parser.SwaggerParser.read(SwaggerParser.java:105)
	at io.swagger.codegen.config.CodegenConfigurator.toClientOptInput(CodegenConfigurator.java:452)
	at io.swagger.codegen.cmd.Generate.run(Generate.java:305)

ScrapCodes · 2023-01-22T13:16:04Z

Hi @Tomcli, this PR is ready for review. To be noted : change set of API (v1beta1) -> v1 is large and done by manual replacements.

ScrapCodes · 2023-01-23T04:26:57Z

/retest

Tomcli · 2023-01-23T21:35:06Z

@ScrapCodes for the apiserver it seems like your http proxy wasn't updated properly, so the pod just start crashing after the server enters the main program.
https://github.com/ScrapCodes/kfp-tekton/blob/v1_part2/backend/src/apiserver/main.go#L142-L151

Tomcli · 2023-01-23T21:36:09Z

@ScrapCodes same thing with the persistent agent
https://github.com/ScrapCodes/kfp-tekton/blob/v1_part2/backend/src/agent/persistence/main.go#L133-L134

Tomcli · 2023-01-23T21:40:30Z

@ScrapCodes also this const needs to change to v1 as well
https://github.com/ScrapCodes/kfp-tekton/blob/v1_part2/backend/src/common/client/api_server/pipeline_upload_client.go#L21

Tomcli · 2023-01-23T21:47:44Z

Also we haven't update the python client so all the tests with python client so the e2e test and integration will be broken for this PR, but we should at least pass the deployment test after fixing these apis and see "Failed to upload pipeline" in our testing script for the python client errors. https://github.com/kubeflow/kfp-tekton/blob/master/scripts/deploy/iks/test-flip-coin.sh#L30

Tomcli · 2023-01-23T21:51:11Z

scripts/deploy/github/build-images.sh

@@ -25,5 +25,5 @@ docker build -t "${REGISTRY}/kfp-tekton/persistenceagent:latest" -f backend/Dock
 docker build -t "${REGISTRY}/kfp-tekton/metadata-writer:latest" -f backend/metadata_writer/Dockerfile . && docker push "${REGISTRY}/kfp-tekton/metadata-writer:latest" &
 docker build -t "${REGISTRY}/kfp-tekton/scheduledworkflow:latest" -f backend/Dockerfile.scheduledworkflow . && docker push "${REGISTRY}/kfp-tekton/scheduledworkflow:latest" &
 docker build -t "${REGISTRY}/kfp-tekton/cache-server:latest" -f backend/Dockerfile.cacheserver . && docker push "${REGISTRY}/kfp-tekton/cache-server:latest" &
-
+docker build -t "${REGISTRY}/kfp-tekton/frontend:latest" -f frontend/Dockerfile . && docker push "${REGISTRY}/kfp-tekton/frontend:latest" &


just fyi we didn't add frontend in our github actions because it sometimes get over the github action vm quota. We do test our frontend code in our ibm cluster ci by running the Manual Trigger - e2e at https://cloud.ibm.com/devops/pipelines/tekton/f85a45d2-36fc-45fb-a449-b01c13cf5191?env_id=ibm%3Ayp%3Aus-south

The github action e2e script itself is communicating between the python client to api server, so it never talks to the ui itself.

@yhwang fyi maybe we should have another github action to build ui images when the ui code is modified.

Tomcli · 2023-01-24T18:31:42Z

Thanks @ScrapCodes with the new fixes I verified the UI client works with the new V1 API. This PR haven't add any changes to the python sdk client so the sdk client failing since there's no v1beta1 API anymore. Since this PR is getting too big, I'm planning just merge this PR first and have the add the new changes to the SDK client as soon as possible. I will cut a v1beta1-api branch for now during this migration.

@yhwang WDYT?

yhwang · 2023-01-24T18:46:10Z

agree. it's a good idea to cut a v1beta1-api branch. Another way around is to have a v1-api branch and land those v1 API related PRs there. once everything is settled, merge the branch back to the master.

But this PR is open against the master now. so I guess you can just land this change into the master. we may do that next time.

Tomcli · 2023-01-24T21:16:34Z

@ScrapCodes fyi i will modify the python sdk client to use the new api, once i have that ready working with this PR. Then I will merge this and the python client pr so that the master branch won't be failing.

1. from modified swagger definition and python generator script. 2. Also deleted old python client files.

ScrapCodes · 2023-01-27T11:20:37Z

Hi @Tomcli ,
Thank you for taking a look,
python client sdk is generated and uploaded to Testpypi, https://test.pypi.org/project/kfp-tekton-server-api/#files

I was unable to figure - where it is referred by python SDK.

Tomcli · 2023-01-27T17:20:19Z

Thanks @ScrapCodes, can you also add @yhwang and my ibm email to the maintainer list in the kfp-tekton-server-api package?
[email protected]
[email protected]

Also, the original api client is referred in the kfp package. We need to create a new inherent client class to replace the v1beta1 client code to v1 as a new client function. https://github.com/kubeflow/pipelines/blob/sdk/release-1.8/sdk/python/requirements.in#L20

ScrapCodes · 2023-01-29T14:06:34Z

Hi Tommy, I will need your username,

https://test.pypi.org/account/register/
Also:
https://pypi.org/account/register/

Tomcli · 2023-01-30T16:59:31Z

@ScrapCodes my pypi user name is tomcli

ScrapCodes · 2023-01-31T10:35:11Z

Thank you @Tomcli and @yhwang , I have added @yhwang with username yhwang (Can you confirm? )

… containing the V1 API.

yhwang · 2023-01-31T16:29:12Z

@ScrapCodes this is me: https://pypi.org/user/yihongwang/
my username is yihongwang

Tomcli · 2023-01-31T20:39:46Z

sdk/python/requirements.in

+protobuf>=3.20.1,<4
+kfp-tekton-server-api==1.5.0


Suggested change

kfp-tekton-server-api==1.5.0

kfp-tekton-server-api>=1.5.0

This way is more flexible if we need to add bug patches to the api package.

Tomcli · 2023-01-31T20:42:04Z

Thanks @ScrapCodes, I will open a PR shortly with the new api package you have. Can you fix the python lint so that I can merge this PR? Thanks

make lint

ScrapCodes · 2023-02-01T00:03:43Z

Thanks @ScrapCodes, I will open a PR shortly with the new api package you have. Can you fix the python lint so that I can merge this PR? Thanks

@Tomcli done!

Also did you receive the invite to collaborate on pypi published package?

Tomcli · 2023-02-01T18:12:07Z

thanks @ScrapCodes, I will merge this for now and add a patch for the e2e test. Some tests need to be updated.

google-oss-prow bot added do-not-merge/work-in-progress size/XXL labels Jan 17, 2023

google-oss-prow bot requested review from ckadner and Tomcli January 17, 2023 13:43

google-oss-prow bot added the approved label Jan 17, 2023

ScrapCodes force-pushed the v1_part2 branch 2 times, most recently from 87c42ff to 6bed352 Compare January 18, 2023 09:44

ScrapCodes changed the title ~~WIP: switch to v1 API paths for backend.~~ Feat: switch to v1 API paths for backend. Jan 22, 2023

google-oss-prow bot removed the do-not-merge/work-in-progress label Jan 22, 2023

Tomcli reviewed Jan 23, 2023

View reviewed changes

ScrapCodes force-pushed the v1_part2 branch from 61b3c7a to 71e650e Compare January 24, 2023 12:50

ScrapCodes force-pushed the v1_part2 branch from 71e650e to 242eff7 Compare January 27, 2023 10:48

ScrapCodes added 4 commits January 27, 2023 16:30

switch to v1 API paths for backend.

144f0fe

python api update.

b53f51e

updated deployment to query v1

738e930

generated swagger api for frontend and replaced previous Api with V1.

9383e58

changed remaining API -> V1 api

0b74e81

ScrapCodes force-pushed the v1_part2 branch from 242eff7 to 7d73777 Compare January 27, 2023 11:01

Generated python api client

fb76f71

1. from modified swagger definition and python generator script. 2. Also deleted old python client files.

ScrapCodes force-pushed the v1_part2 branch from 7d73777 to fb76f71 Compare January 27, 2023 11:16

Updating the requirements to import the kfp-tekton-server-api package…

0692219

… containing the V1 API.

Tomcli reviewed Jan 31, 2023

View reviewed changes

Tomcli mentioned this pull request Jan 31, 2023

feat(sdk): Add sdk v1 api patch #1138

Closed

1 task

fixed the lint

e3d7c45

add sdk v0 api patch

1a4ccf5

Tomcli merged commit 4637dcb into kubeflow:master Feb 1, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat: switch to v1 API paths for backend. #1121

Feat: switch to v1 API paths for backend. #1121

ScrapCodes commented Jan 17, 2023

google-oss-prow bot commented Jan 17, 2023

ScrapCodes commented Jan 18, 2023

Tomcli commented Jan 18, 2023

Tomcli commented Jan 18, 2023

ScrapCodes commented Jan 20, 2023

ScrapCodes commented Jan 22, 2023

ScrapCodes commented Jan 23, 2023

Tomcli commented Jan 23, 2023

Tomcli commented Jan 23, 2023

Tomcli commented Jan 23, 2023

Tomcli commented Jan 23, 2023

Tomcli Jan 23, 2023

Tomcli Jan 23, 2023

Tomcli Jan 23, 2023

Tomcli commented Jan 24, 2023

yhwang commented Jan 24, 2023

Tomcli commented Jan 24, 2023

ScrapCodes commented Jan 27, 2023

Tomcli commented Jan 27, 2023

ScrapCodes commented Jan 29, 2023 •

edited

Loading

Tomcli commented Jan 30, 2023

ScrapCodes commented Jan 31, 2023

yhwang commented Jan 31, 2023

Tomcli Jan 31, 2023

Tomcli commented Jan 31, 2023

ScrapCodes commented Feb 1, 2023

Tomcli commented Feb 1, 2023

		protobuf>=3.20.1,<4
		kfp-tekton-server-api==1.5.0

Feat: switch to v1 API paths for backend. #1121

Feat: switch to v1 API paths for backend. #1121

Conversation

ScrapCodes commented Jan 17, 2023

google-oss-prow bot commented Jan 17, 2023

ScrapCodes commented Jan 18, 2023

Tomcli commented Jan 18, 2023

Tomcli commented Jan 18, 2023

ScrapCodes commented Jan 20, 2023

ScrapCodes commented Jan 22, 2023

ScrapCodes commented Jan 23, 2023

Tomcli commented Jan 23, 2023

Tomcli commented Jan 23, 2023

Tomcli commented Jan 23, 2023

Tomcli commented Jan 23, 2023

Tomcli Jan 23, 2023

Choose a reason for hiding this comment

Tomcli Jan 23, 2023

Choose a reason for hiding this comment

Tomcli Jan 23, 2023

Choose a reason for hiding this comment

Tomcli commented Jan 24, 2023

yhwang commented Jan 24, 2023

Tomcli commented Jan 24, 2023

ScrapCodes commented Jan 27, 2023

Tomcli commented Jan 27, 2023

ScrapCodes commented Jan 29, 2023 • edited Loading

Tomcli commented Jan 30, 2023

ScrapCodes commented Jan 31, 2023

yhwang commented Jan 31, 2023

Tomcli Jan 31, 2023

Choose a reason for hiding this comment

Tomcli commented Jan 31, 2023

ScrapCodes commented Feb 1, 2023

Tomcli commented Feb 1, 2023

ScrapCodes commented Jan 29, 2023 •

edited

Loading