Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat: switch to v1 API paths for backend. #1121

Merged
merged 9 commits into from
Feb 1, 2023

Conversation

ScrapCodes
Copy link
Contributor

Which issue is resolved by this Pull Request:
Resolves #1081

Description of your changes:

Environment tested:

  • Python Version (use python --version):
  • Tekton Version (use tkn version):
  • Kubernetes Version (use kubectl version):
  • OS (e.g. from /etc/os-release):

Checklist:

@google-oss-prow
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ScrapCodes

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ScrapCodes
Copy link
Contributor Author

currently trying to figure out, why it is querying v1beta1

    at emitErrorCloseNT (internal/streams/destroy.js:74:3)
    at processTicksAndRejections (internal/process/task_queues.js:82:21)
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:1) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
(node:1) UnhandledPromiseRejectionWarning: FetchError: request to http://metadata/computeMetadata/v1/project/project-id failed, reason: getaddrinfo ENOTFOUND metadata
    at ClientRequest.<anonymous> (/server/node_modules/node-fetch/lib/index.js:1461:11)
    at ClientRequest.emit (events.js:400:28)
    at Socket.socketErrorListener (_http_client.js:475:9)
    at Socket.emit (events.js:400:28)
    at emitErrorNT (internal/streams/destroy.js:106:8)
    at emitErrorCloseNT (internal/streams/destroy.js:74:3)
    at processTicksAndRejections (internal/process/task_queues.js:82:21)
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2)
GET /apis/v1beta1/healthz
GET /apis/v1beta1/healthz
GET /apis/v1beta1/healthz
GET /apis/v1beta1/healthz
GET /apis/v1beta1/healthz
GET /apis/v1beta1/healthz
GET /apis/v1beta1/experiments?page_token=&page_size=10&sort_by=created_at%20desc&filter=%257B%2522predicates%2522%253A%255B%257B%2522key%2522%253A%2522storage_state%2522%252C%2522op%2522%253A%2522NOT_EQUALS%2522%252C%2522string_value%2522%253A%2522STORAGESTATE_ARCHIVED%2522%257D%255D%257D
Proxied request:  /apis/v1beta1/experiments?page_token=&page_size=10&sort_by=created_at%20desc&filter=%257B%2522predicates%2522%253A%255B%257B%2522key%2522%253A%2522storage_state%2522%252C%2522op%2522%253A%2522NOT_EQUALS%2522%252C%2522string_value%2522%253A%2522STORAGESTATE_ARCHIVED%2522%257D%255D%257D
GET /apis/v1beta1/healthz
GET /apis/v1beta1/healthz
GET /apis/v1beta1/healthz
GET /apis/v1beta1/healthz

@Tomcli
Copy link
Member

Tomcli commented Jan 18, 2023

currently trying to figure out, why it is querying v1beta1

    at emitErrorCloseNT (internal/streams/destroy.js:74:3)
    at processTicksAndRejections (internal/process/task_queues.js:82:21)
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:1) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
(node:1) UnhandledPromiseRejectionWarning: FetchError: request to http://metadata/computeMetadata/v1/project/project-id failed, reason: getaddrinfo ENOTFOUND metadata
    at ClientRequest.<anonymous> (/server/node_modules/node-fetch/lib/index.js:1461:11)
    at ClientRequest.emit (events.js:400:28)
    at Socket.socketErrorListener (_http_client.js:475:9)
    at Socket.emit (events.js:400:28)
    at emitErrorNT (internal/streams/destroy.js:106:8)
    at emitErrorCloseNT (internal/streams/destroy.js:74:3)
    at processTicksAndRejections (internal/process/task_queues.js:82:21)
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2)
GET /apis/v1beta1/healthz
GET /apis/v1beta1/healthz
GET /apis/v1beta1/healthz
GET /apis/v1beta1/healthz
GET /apis/v1beta1/healthz
GET /apis/v1beta1/healthz
GET /apis/v1beta1/experiments?page_token=&page_size=10&sort_by=created_at%20desc&filter=%257B%2522predicates%2522%253A%255B%257B%2522key%2522%253A%2522storage_state%2522%252C%2522op%2522%253A%2522NOT_EQUALS%2522%252C%2522string_value%2522%253A%2522STORAGESTATE_ARCHIVED%2522%257D%255D%257D
Proxied request:  /apis/v1beta1/experiments?page_token=&page_size=10&sort_by=created_at%20desc&filter=%257B%2522predicates%2522%253A%255B%257B%2522key%2522%253A%2522storage_state%2522%252C%2522op%2522%253A%2522NOT_EQUALS%2522%252C%2522string_value%2522%253A%2522STORAGESTATE_ARCHIVED%2522%257D%255D%257D
GET /apis/v1beta1/healthz
GET /apis/v1beta1/healthz
GET /apis/v1beta1/healthz
GET /apis/v1beta1/healthz

which microservice is this?

@Tomcli
Copy link
Member

Tomcli commented Jan 18, 2023

@ScrapCodes you probably want to update the deployment manifests too, seems like you are still deploying with the health checking pinning the v1beta1 api.
https://github.com/kubeflow/kfp-tekton/blob/559f8b6df04629cd2be43935f30268b406385001/manifests/kustomize/base/pipeline/ml-pipeline-apiserver-deployment.yaml

@ScrapCodes
Copy link
Contributor Author

In the package frontend/ , how does npm run apis figure out where to find swagger files.

npm run apis

> [email protected] apis /Users/prashantsharma/work/kfp-tekton/frontend
> npm run apis:experiment && npm run apis:job && npm run apis:pipeline && npm run apis:run && npm run apis:filter && npm run apis:visualization && npm run apis:auth


> [email protected] apis:experiment /Users/prashantsharma/work/kfp-tekton/frontend
> java -jar swagger-codegen-cli.jar generate -i ../backend/api/swagger/experiment.swagger.json -l typescript-fetch -o ./src/apis/experiment -c ./swagger-config.json

[main] INFO io.swagger.parser.Swagger20Parser - reading from ../backend/api/swagger/experiment.swagger.json
[main] INFO io.swagger.parser.Swagger20Parser - reading from ../backend/api/swagger/experiment.swagger.json
[main] ERROR io.swagger.parser.SwaggerCompatConverter - failed to read resource listing
java.lang.RuntimeException: Could not find ../backend/api/swagger/experiment.swagger.json on the classpath
	at io.swagger.parser.util.ClasspathHelper.loadFileFromClasspath(ClasspathHelper.java:31)
	at io.swagger.parser.SwaggerCompatConverter.readResourceListing(SwaggerCompatConverter.java:206)
	at io.swagger.parser.SwaggerCompatConverter.read(SwaggerCompatConverter.java:122)
	at io.swagger.parser.SwaggerParser.read(SwaggerParser.java:105)
	at io.swagger.codegen.config.CodegenConfigurator.toClientOptInput(CodegenConfigurator.java:452)
	at io.swagger.codegen.cmd.Generate.run(Generate.java:305)

@ScrapCodes ScrapCodes changed the title WIP: switch to v1 API paths for backend. Feat: switch to v1 API paths for backend. Jan 22, 2023
@ScrapCodes
Copy link
Contributor Author

Hi @Tomcli, this PR is ready for review. To be noted : change set of API (v1beta1) -> v1 is large and done by manual replacements.

@ScrapCodes
Copy link
Contributor Author

/retest

@Tomcli
Copy link
Member

Tomcli commented Jan 23, 2023

@ScrapCodes for the apiserver it seems like your http proxy wasn't updated properly, so the pod just start crashing after the server enters the main program.
https://github.com/ScrapCodes/kfp-tekton/blob/v1_part2/backend/src/apiserver/main.go#L142-L151

@Tomcli
Copy link
Member

Tomcli commented Jan 23, 2023

@Tomcli
Copy link
Member

Tomcli commented Jan 23, 2023

@Tomcli
Copy link
Member

Tomcli commented Jan 23, 2023

Also we haven't update the python client so all the tests with python client so the e2e test and integration will be broken for this PR, but we should at least pass the deployment test after fixing these apis and see "Failed to upload pipeline" in our testing script for the python client errors. https://github.com/kubeflow/kfp-tekton/blob/master/scripts/deploy/iks/test-flip-coin.sh#L30

@@ -25,5 +25,5 @@ docker build -t "${REGISTRY}/kfp-tekton/persistenceagent:latest" -f backend/Dock
docker build -t "${REGISTRY}/kfp-tekton/metadata-writer:latest" -f backend/metadata_writer/Dockerfile . && docker push "${REGISTRY}/kfp-tekton/metadata-writer:latest" &
docker build -t "${REGISTRY}/kfp-tekton/scheduledworkflow:latest" -f backend/Dockerfile.scheduledworkflow . && docker push "${REGISTRY}/kfp-tekton/scheduledworkflow:latest" &
docker build -t "${REGISTRY}/kfp-tekton/cache-server:latest" -f backend/Dockerfile.cacheserver . && docker push "${REGISTRY}/kfp-tekton/cache-server:latest" &

docker build -t "${REGISTRY}/kfp-tekton/frontend:latest" -f frontend/Dockerfile . && docker push "${REGISTRY}/kfp-tekton/frontend:latest" &
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just fyi we didn't add frontend in our github actions because it sometimes get over the github action vm quota. We do test our frontend code in our ibm cluster ci by running the Manual Trigger - e2e at https://cloud.ibm.com/devops/pipelines/tekton/f85a45d2-36fc-45fb-a449-b01c13cf5191?env_id=ibm%3Ayp%3Aus-south

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The github action e2e script itself is communicating between the python client to api server, so it never talks to the ui itself.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yhwang fyi maybe we should have another github action to build ui images when the ui code is modified.

@Tomcli
Copy link
Member

Tomcli commented Jan 24, 2023

Thanks @ScrapCodes with the new fixes I verified the UI client works with the new V1 API. This PR haven't add any changes to the python sdk client so the sdk client failing since there's no v1beta1 API anymore. Since this PR is getting too big, I'm planning just merge this PR first and have the add the new changes to the SDK client as soon as possible. I will cut a v1beta1-api branch for now during this migration.

@yhwang WDYT?

@yhwang
Copy link
Member

yhwang commented Jan 24, 2023

agree. it's a good idea to cut a v1beta1-api branch. Another way around is to have a v1-api branch and land those v1 API related PRs there. once everything is settled, merge the branch back to the master.

But this PR is open against the master now. so I guess you can just land this change into the master. we may do that next time.

@Tomcli
Copy link
Member

Tomcli commented Jan 24, 2023

@ScrapCodes fyi i will modify the python sdk client to use the new api, once i have that ready working with this PR. Then I will merge this and the python client pr so that the master branch won't be failing.

1. from modified swagger definition and python generator script.
2. Also deleted old python client files.
@ScrapCodes
Copy link
Contributor Author

Hi @Tomcli ,
Thank you for taking a look,
python client sdk is generated and uploaded to Testpypi, https://test.pypi.org/project/kfp-tekton-server-api/#files

I was unable to figure - where it is referred by python SDK.

@Tomcli
Copy link
Member

Tomcli commented Jan 27, 2023

Thanks @ScrapCodes, can you also add @yhwang and my ibm email to the maintainer list in the kfp-tekton-server-api package?
[email protected]
[email protected]

Also, the original api client is referred in the kfp package. We need to create a new inherent client class to replace the v1beta1 client code to v1 as a new client function. https://github.com/kubeflow/pipelines/blob/sdk/release-1.8/sdk/python/requirements.in#L20

@ScrapCodes
Copy link
Contributor Author

ScrapCodes commented Jan 29, 2023

Hi Tommy, I will need your username,

Screenshot 2023-01-29 at 7 35 37 PM

https://test.pypi.org/account/register/
Also:
https://pypi.org/account/register/

@Tomcli
Copy link
Member

Tomcli commented Jan 30, 2023

@ScrapCodes my pypi user name is tomcli

@ScrapCodes
Copy link
Contributor Author

Thank you @Tomcli and @yhwang , I have added @yhwang with username yhwang (Can you confirm? )

@yhwang
Copy link
Member

yhwang commented Jan 31, 2023

@ScrapCodes this is me: https://pypi.org/user/yihongwang/
my username is yihongwang

protobuf>=3.20.1,<4
kfp-tekton-server-api==1.5.0
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
kfp-tekton-server-api==1.5.0
kfp-tekton-server-api>=1.5.0

This way is more flexible if we need to add bug patches to the api package.

@Tomcli
Copy link
Member

Tomcli commented Jan 31, 2023

Thanks @ScrapCodes, I will open a PR shortly with the new api package you have. Can you fix the python lint so that I can merge this PR? Thanks

make lint

@Tomcli Tomcli mentioned this pull request Jan 31, 2023
1 task
@ScrapCodes
Copy link
Contributor Author

Thanks @ScrapCodes, I will open a PR shortly with the new api package you have. Can you fix the python lint so that I can merge this PR? Thanks

@Tomcli done!

Also did you receive the invite to collaborate on pypi published package?

@Tomcli
Copy link
Member

Tomcli commented Feb 1, 2023

thanks @ScrapCodes, I will merge this for now and add a patch for the e2e test. Some tests need to be updated.

@Tomcli Tomcli merged commit 4637dcb into kubeflow:master Feb 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

add v1 api endpoint support for backend server
3 participants