Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(sdk): fix nested placeholders and block illegal IfPresent form in Concat #8414

Merged
merged 5 commits into from
Nov 4, 2022
Merged

fix(sdk): fix nested placeholders and block illegal IfPresent form in Concat #8414

merged 5 commits into from
Nov 4, 2022

Conversation

connor-mccarthy
Copy link
Member

Description of your changes:
This PR:

  • Adds a check to prevent then or else_ arguments to IfPresentPlaceholder from
    being lists if the IfPresentPlaceholder is within a ConcatPlaceholder.
  • Fixes a bug in handling of then and else_ when either is a string element (instead of
    a list of elements)

Checklist:

@google-oss-prow
Copy link

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@connor-mccarthy connor-mccarthy force-pushed the fix-nested-ifpresent-in-concat branch from 43b6b42 to f2f1a7d Compare November 2, 2022 18:55
@connor-mccarthy
Copy link
Member Author

/test all

@connor-mccarthy
Copy link
Member Author

Note that the kubeflow-pipelines-component-yaml failures are attributed to two Google Cloud Pipeline Components components which will never be loaded by users directly (only via the google_cloud_pipeline_components library). Since the current versions of google_cloud_pipeline_components is incompatible with kfp==2.x.x, this is not an error users will ever experience. Therefore, it is safe to ignore these two YAML failures and merge.

@connor-mccarthy
Copy link
Member Author

/assign @chensun

@connor-mccarthy
Copy link
Member Author

/retest kubeflow-pipelines-sdk-python39

@google-oss-prow
Copy link

@connor-mccarthy: The /retest command does not accept any targets.
The following commands are available to trigger required jobs:

  • /test kubeflow-pipeline-backend-test
  • /test kubeflow-pipeline-e2e-test
  • /test kubeflow-pipeline-frontend-test
  • /test kubeflow-pipeline-mkp-snapshot-test
  • /test kubeflow-pipeline-mkp-test
  • /test kubeflow-pipeline-upgrade-test
  • /test kubeflow-pipelines-backend-visualization
  • /test kubeflow-pipelines-component-yaml
  • /test kubeflow-pipelines-components-gcp-python37
  • /test kubeflow-pipelines-components-google-cloud-python38
  • /test kubeflow-pipelines-integration-v2
  • /test kubeflow-pipelines-manifests
  • /test kubeflow-pipelines-samples-v2
  • /test kubeflow-pipelines-sdk-docformatter
  • /test kubeflow-pipelines-sdk-isort
  • /test kubeflow-pipelines-sdk-python310
  • /test kubeflow-pipelines-sdk-python37
  • /test kubeflow-pipelines-sdk-python38
  • /test kubeflow-pipelines-sdk-python39
  • /test kubeflow-pipelines-sdk-yapf
  • /test kubeflow-pipelines-tfx-python37

The following commands are available to trigger optional jobs:

  • /test kubeflow-pipelines-sdk-execution-tests
  • /test kubeflow-pipelines-sdk-python311

Use /test all to run the following jobs that were automatically triggered:

  • kubeflow-pipelines-component-yaml
  • kubeflow-pipelines-sdk-docformatter
  • kubeflow-pipelines-sdk-execution-tests
  • kubeflow-pipelines-sdk-isort
  • kubeflow-pipelines-sdk-python310
  • kubeflow-pipelines-sdk-python311
  • kubeflow-pipelines-sdk-python37
  • kubeflow-pipelines-sdk-python38
  • kubeflow-pipelines-sdk-python39
  • kubeflow-pipelines-sdk-yapf
  • kubeflow-pipelines-tfx-python37

In response to this:

/retest kubeflow-pipelines-sdk-python39

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@connor-mccarthy connor-mccarthy marked this pull request as ready for review November 2, 2022 20:07
@connor-mccarthy
Copy link
Member Author

/retest

@chensun
Copy link
Member

chensun commented Nov 3, 2022

Note that the kubeflow-pipelines-component-yaml failures are attributed to two Google Cloud Pipeline Components components which will never be loaded by users directly (only via the google_cloud_pipeline_components library). Since the current versions of google_cloud_pipeline_components is incompatible with kfp==2.x.x, this is not an error users will ever experience. Therefore, it is safe to ignore these two YAML failures and merge.

What is the plan for these two incompatible components? Are they going to be updated when GCPC migrate to support kfp 2.*?
We should exclude them from the test with some TODO comments. Otherwise, once this PR is merged, the same presubmit test will fail for all subsequent PRs.

@connor-mccarthy
Copy link
Member Author

What is the plan for these two incompatible components? Are they going to be updated when GCPC migrate to support kfp 2.*?

Yes. It will be impossible to author these incompatible components via KFP 2.*, so GCPC authors will be pushed in the correct direction.

We should exclude them from the test with some TODO comments. Otherwise, once this PR is merged, the same presubmit test will fail for all subsequent PRs.

Will follow up with this change.

@connor-mccarthy connor-mccarthy force-pushed the fix-nested-ifpresent-in-concat branch from 5347924 to 6f27c98 Compare November 3, 2022 22:55
chensun
chensun approved these changes Nov 4, 2022
View reviewed changes
Copy link
Member

@chensun chensun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/approve

Thanks!

@google-oss-prow google-oss-prow bot added the lgtm label Nov 4, 2022
@google-oss-prow
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: chensun

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@google-oss-prow google-oss-prow bot merged commit d33f359 into kubeflow:master Nov 4, 2022
@Ark-kun
Copy link
Contributor

Ark-kun commented Nov 8, 2022

Using if placeholder inside concat might look weird, but the lest-surprise way of handling it seems to be pretty simple - concatenate the then/else list items with the rest of the concat items.
Are there any backend issues handling this?

@connor-mccarthy
Copy link
Member Author

concatenate the then/else list items with the rest of the concat items.

@Ark-kun, this is the approach we've taken, with one restriction: then and else must be single elements (not lists)

jlyaoyuli pushed a commit to jlyaoyuli/pipelines that referenced this pull request Jan 5, 2023
@connor-mccarthy @jlyaoyuli
fix(sdk): fix nested placeholders and block illegal IfPresent form in…
f2e628c 
… Concat (kubeflow#8414)

* support single element IfPresentPlaceholders that contain a primitive placeholder

* block list then and else_ inside concat

* support single element IfPresent in v1

* clean up typing

* skip loading batch_predict_job yaml
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@chensun chensun chensun approved these changes

@IronPan IronPan Awaiting requested review from IronPan

@james-jwu james-jwu Awaiting requested review from james-jwu

@zijianjoy zijianjoy Awaiting requested review from zijianjoy

Assignees

@chensun chensun

Labels
approved lgtm size/L
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@connor-mccarthy @chensun @Ark-kun