Support local file upload when creating new pipeline version

[jingzhang36]

No description provided.

[Bobgy]

@jingzhang36 I think all work is done here, is that right?
Can you link all relevant PRs for tracking purpose?

Added a PR for api client on version upload below.

I believe we can close this issue.

[pselden]

Does the sdk still need to be updated? I looked through some of the client.py files and couldn't find anything related to this.

[Bobgy]

@numerology @Ark-kun Do you know answer to above question?
Is regenerating server api client enough? (I did that in 0.2.5 release)

[jingzhang36]

I feel using the generated server api client would be ok as long as I can find a way to load a local file to io object and pass to the generated api method. Let me try to make up an example and post it here

[jingzhang36]

@pselden with kfp 0.2.5, you can do

import os
import kfp
myfile = os.path.join(<your local pipeline file>)
client = kfp.Client(<host>)
client.pipeline_uploads.upload_pipeline_version(myfile, pipelineid=<id of the pipeline you want to create version under>, name=<version name by your choice>)

Also note: pipelineid and name are both strings.

[Bobgy]

To add to above comment, you also need to upgrade kfp-server-api package to 0.2.5

[pselden]

Thanks. Do you happen to know if this will be added to the CLI as well?

Something like kfp pipelines upload --name <name> --version <version>?

[pahask8]

@pselden with kfp 0.2.5, you can do

import os
import kfp
myfile = os.path.join()
client = kfp.Client()
client.pipeline_uploads.upload_pipeline_version(myfile, pipelineid=, name=)

Also note: pipelineid and name are both strings.

@jingzhang36 Thank you for sharing this example. Right now the kfp package does not support this version:

pipelines/sdk/python/requirements.txt

Line 13 in ee525b9

kfp-server-api >= 0.1.18, < 0.1.19

. Will it be updated soon?
Thank you

[numerology]

Will it be updated soon?

Sorry for the confusion. kfp-server-api does not have a consistent version number with KFP SDK package.

So if you deploy KFP with version 0.2.5 and use KFP SDK version higher than 0.2.5, it works.

[Bobgy]

@numerology I just released kfp-server-api 0.2.5
Looks like we need to update

pipelines/sdk/python/setup.py

Line 35 in ee525b9

'kfp-server-api >= 0.1.18, <= 0.1.40', #Update the upper version whenever a new version of the kfp-server-api package is released. Update the lower version when there is a breaking change in kfp-server-api.

[numerology]

@numerology I just released kfp-server-api 0.2.5
Looks like we need to update

pipelines/sdk/python/setup.py

Line 35 in ee525b9

'kfp-server-api >= 0.1.18, <= 0.1.40', #Update the upper version whenever a new version of the kfp-server-api package is released. Update the lower version when there is a breaking change in kfp-server-api.

Ah got it.

[pselden]

Just checking -- this is NOT available in any officially published version of the KFP client sdk right?

[Bobgy]

@pselden it's available in 0.2.5 and 0.3.0.
Refer to #2977 (comment)

but you need to make sure your kfp server is the same version.

[Tauranis]

Unfortunately, uploading a new version is crashing.

When I run

kfp --endpoint <URL> pipeline upload-version -p <PIPE_ID> -v <PIPE_VERSION> ./dist/pipeline.tar.gz

It throws (500) Bad Request. Reason: Internal Server Error

The same happens when I use the Python SDK

import os
import kfp
client = kfp.Client()
client.pipeline_uploads.upload_pipeline_version(<FILE>, pipelineid=<PIPE_ID>, name=<PIPE_VERSION>)

Currently I'm using Kubeflow deployed at Google AI Platform with kfp==0.5.1 and kfp-server-api==0.5.0

Does someone else have the same problem?

[jingzhang36]

I tested with kfp===0.4.0 kfp-server-api===0.3.9 and kfp standalone deployment with 0.5.1
By following the example at https://www.kubeflow.org/docs/pipelines/tutorials/sdk-examples/
I can not reproduce the error.
So that means the API server's feature is fine. Suspecting issues with cluster setting up.
Since this Github issue is for feature request and the feature is good per my test, let's close this one and open another issue for investigating the particular user issue with @Tauranis deployment.
@Tauranis, do you mind moving to #3805?

[Tauranis]

I found the error.
I was passing the pipeline name and not the pipeline id.
My bad. Sorry.

For those who want to get the pipeline id from its name the following will work.

PIPELINE_ID=$(kfp --endpoint ${KUBEFLOW_ENDPOINT} pipeline list | grep -w  ${PIPELINE_NAME}  | grep -E -o -e "([a-z0-9]){8}-([a-z0-9]){4}-([a-z0-9]){4}-([a-z0-9]){4}-([a-z0-9]){12}")

I'm currently using this for my CI/CD pipeline.