Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add literal type to union literal #1144

Merged
merged 4 commits into from
Sep 9, 2022
Merged

Add literal type to union literal #1144

merged 4 commits into from
Sep 9, 2022

Conversation

pingsutw
Copy link
Member

Signed-off-by: Kevin Su [email protected]

TL;DR

flyteorg/flyte#2817

class Color(Enum):
    RED = "red"
    GREEN = "green"
    BLUE = "blue"


@task
def coalesce(a:  Union[List[str], str, Color]) -> str:
    if type(a) == Color:
        print(a.name)
        return a.name
    print(a)
    return a


@workflow
def wf(a:  Union[List[str], str, Color]) -> str:
    coalesced = coalesce(a=a)
    return coalesced


if __name__ == "__main__":
    wf(a=Color("red"))

Take the above code as an example. If we run pyflyte run --remote example_union.py wf --a "flyte", we'll get the below error.

Request failed with status code 400 invalid a input wrong type, expected union_type:<variants:<collection_type:
<simple:STRING > structure:<tag:"Typed List" > > variants:<simple:STRING structure:<tag:"str" > > variants:<enum_type:
<values:"red" values:"green" values:"blue" > structure:<tag:"DefaultEnumTransformer" > > > , got simple:STRING instead

That's because literal(string) can be converted to str or enum, which is ambiguous. To address this, we should add literal type in Union type, and it can force validator to cast "flyte" to string

Type

  • Bug Fix
  • Feature
  • Plugin

Are all requirements met?

  • Code completed
  • Smoke tested
  • Unit tests added
  • Code documentation added
  • Any pending items have an associated Issue

Complete description

How did you fix the bug, make the feature etc. Link to any design docs etc

Tracking Issue

https://github.com/flyteorg/flyte/issues/

Follow-up issue

NA
OR
https://github.com/flyteorg/flyte/issues/

@pingsutw pingsutw marked this pull request as draft August 29, 2022 19:45
Signed-off-by: Kevin Su <[email protected]>
@codecov
Copy link

codecov bot commented Aug 30, 2022

Codecov Report

Merging #1144 (304dd94) into master (3cf0639) will increase coverage by 0.07%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #1144      +/-   ##
==========================================
+ Coverage   68.36%   68.43%   +0.07%     
==========================================
  Files         287      288       +1     
  Lines       25931    25998      +67     
  Branches     2898     2900       +2     
==========================================
+ Hits        17728    17793      +65     
- Misses       7725     7727       +2     
  Partials      478      478              
Impacted Files Coverage Δ
flytekit/clis/sdk_in_container/run.py 84.19% <100.00%> (+0.59%) ⬆️
tests/flytekit/unit/cli/pyflyte/test_run.py 99.20% <100.00%> (+0.39%) ⬆️
flytekit/remote/remote.py 41.11% <0.00%> (ø)
flytekit/clis/sdk_in_container/constants.py 100.00% <0.00%> (ø)
tests/flytekit/unit/cli/pyflyte/imperative_wf.py 86.36% <0.00%> (ø)
flytekit/core/tracker.py 48.67% <0.00%> (+0.04%) ⬆️
flytekit/tools/script_mode.py 49.25% <0.00%> (+0.72%) ⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@pingsutw pingsutw marked this pull request as ready for review August 30, 2022 07:08
@eapolinario
Copy link
Collaborator

Can you add a unit test?

Signed-off-by: Kevin Su <[email protected]>
eapolinario
eapolinario previously approved these changes Aug 30, 2022
Signed-off-by: Kevin Su <[email protected]>
@pingsutw pingsutw marked this pull request as draft September 1, 2022 17:48
@pingsutw pingsutw marked this pull request as ready for review September 2, 2022 12:42
@wild-endeavor
Copy link
Contributor

this is dependent on the propeller pr going out first right?

@pingsutw
Copy link
Member Author

pingsutw commented Sep 9, 2022

yes, let's merge this after propeller pr.

@pingsutw pingsutw merged commit 3e080b5 into master Sep 9, 2022
eapolinario pushed a commit that referenced this pull request Sep 15, 2022
* Add literal type to union literal

Signed-off-by: Kevin Su <[email protected]>

* fix test

Signed-off-by: Kevin Su <[email protected]>

* Add tests

Signed-off-by: Kevin Su <[email protected]>

* more tests

Signed-off-by: Kevin Su <[email protected]>

Signed-off-by: Kevin Su <[email protected]>
eapolinario added a commit that referenced this pull request Sep 16, 2022
* Add deck to papermill plugin task (#1111)

Signed-off-by: Calvin Leather <[email protected]>

* Run compilation even in local execution for dynamic tasks to early detect errors (#1121)

Signed-off-by: Yee Hing Tong <[email protected]>

* Set to pyflyte run blob object remote when dealing with remote files (#1128)

Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>

* Override voidPromise resource (#1127)

* override void promise resource

Signed-off-by: Kevin Su <[email protected]>

* override void promise resource

Signed-off-by: Kevin Su <[email protected]>

* Fix how ShellTask retrieves the Pod class name (#1132)

* Fix how ShellTask retrieves the Pod class name

Signed-off-by: Matheus Moreno <[email protected]>

* Set Pod class name as a constant

Signed-off-by: Matheus Moreno <[email protected]>

* Revert last commit

Signed-off-by: Matheus Moreno <[email protected]>

* Execute automatic linting

Signed-off-by: Matheus Moreno <[email protected]>

Signed-off-by: Matheus Moreno <[email protected]>

* Add restriction for pandas to be >=1.2 for fsspec plugin (#1136)

Signed-off-by: Yee Hing Tong <[email protected]>

* Use joblib hashing to generate cache key to ensure repeatability (#1126)

* cherry pick 97b454b

Signed-off-by: Yee Hing Tong <[email protected]>

* requirements

Signed-off-by: Yee Hing Tong <[email protected]>

* Fix usage of save in ProtoJoblibHasher

Signed-off-by: Eduardo Apolinario <[email protected]>

* Regenerate requirements using python 3.7

Signed-off-by: Eduardo Apolinario <[email protected]>

* Add test_stable_cache_key

Signed-off-by: Eduardo Apolinario <[email protected]>

Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>
Co-authored-by: Eduardo Apolinario <[email protected]>

* Allow None protocol to mean all data persistence supported storage options in Structured Dataset (#1134)

Signed-off-by: Yee Hing Tong <[email protected]>

* handle ImportError and OSError in extras.pytorch (#1141)

* handle ImportError and OSError in extras.pytorch

Signed-off-by: Niels Bantilan <[email protected]>

* isolate exception to torch import

Signed-off-by: Niels Bantilan <[email protected]>

Signed-off-by: Niels Bantilan <[email protected]>

* Register dataframe renderers in structured dataset (#1140)

* Register dataframe renderers in structured dataset

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* fix test

Signed-off-by: Kevin Su <[email protected]>

* more tests

Signed-off-by: Kevin Su <[email protected]>

Signed-off-by: Kevin Su <[email protected]>

* pyflyte run imperative workflows (#1131)

Signed-off-by: Kevin Su <[email protected]>

* Using sidecar handler to run Papermill task (#1143)

* remove nb prefix

Signed-off-by: Kevin Su <[email protected]>

* add tests

Signed-off-by: Kevin Su <[email protected]>

* Update requirements.in

Signed-off-by: Kevin Su <[email protected]>

* remove _

Signed-off-by: Kevin Su <[email protected]>

Signed-off-by: Kevin Su <[email protected]>

* Properly raise error in NumpyArrayTransformer (#1146)

Signed-off-by: Rahul Mehta <[email protected]>

Signed-off-by: Rahul Mehta <[email protected]>

* Add assert_type in dataclass transformer (#1149)

* Add assert_type in dataclassTransformer

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* fix tests

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* fix tests

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* more tests

Signed-off-by: Kevin Su <[email protected]>

* fix lint

Signed-off-by: Kevin Su <[email protected]>

* Add one more test

Signed-off-by: Kevin Su <[email protected]>

Signed-off-by: Kevin Su <[email protected]>

* Pickle in Union Type (#1147)

* Pickel in Union type

Signed-off-by: Kevin Su <[email protected]>

* Pickel in Union type

Signed-off-by: Kevin Su <[email protected]>

* wip

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

* fix tests

Signed-off-by: Kevin Su <[email protected]>

* update tests

Signed-off-by: Kevin Su <[email protected]>

* fix tests

Signed-off-by: Kevin Su <[email protected]>

* fix tests

Signed-off-by: Kevin Su <[email protected]>

* fix tests

Signed-off-by: Kevin Su <[email protected]>

* fix tests

Signed-off-by: Kevin Su <[email protected]>

* fix tests

Signed-off-by: Kevin Su <[email protected]>

* fix tests

Signed-off-by: Kevin Su <[email protected]>

* fix tests

Signed-off-by: Kevin Su <[email protected]>

* fix tests

Signed-off-by: Kevin Su <[email protected]>

* Address comment

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

Signed-off-by: Kevin Su <[email protected]>

* Bump max docker version to 7.0.0 (#1138)

Signed-off-by: Rahul Mehta <[email protected]>

Signed-off-by: Rahul Mehta <[email protected]>

* Set flytekit<2.0 in plugins (#1152)

Signed-off-by: Eduardo Apolinario <[email protected]>

Signed-off-by: Eduardo Apolinario <[email protected]>
Co-authored-by: Eduardo Apolinario <[email protected]>

* Add literal type to union literal (#1144)

* Add literal type to union literal

Signed-off-by: Kevin Su <[email protected]>

* fix test

Signed-off-by: Kevin Su <[email protected]>

* Add tests

Signed-off-by: Kevin Su <[email protected]>

* more tests

Signed-off-by: Kevin Su <[email protected]>

Signed-off-by: Kevin Su <[email protected]>

* Fix the type of optional[int] in nested dataclass (#1148)

* Fix the type of optional[int] in nested dataclass

Signed-off-by: Kevin Su <[email protected]>

* update tests

Signed-off-by: Kevin Su <[email protected]>

* update comments

Signed-off-by: Kevin Su <[email protected]>

* nit

Signed-off-by: Kevin Su <[email protected]>

Signed-off-by: Kevin Su <[email protected]>

* Added symlink dereferencing in fast packaging and tests (#1151)

* Added symlink dereferencing and tests

Signed-off-by: Vanshika Chowdhary <[email protected]>

* Added flag to register as well

Signed-off-by: Vanshika Chowdhary <[email protected]>

* More flag propagation

Signed-off-by: Vanshika Chowdhary <[email protected]>

Signed-off-by: Vanshika Chowdhary <[email protected]>
Co-authored-by: Vanshika Chowdhary <[email protected]>

* Strip newline from client secret (#1163)

* Strip newline from client secret

* Add logging and rework the secret file comparison to work on windows

Signed-off-by: Eduardo Apolinario <[email protected]>

Signed-off-by: Eduardo Apolinario <[email protected]>
Co-authored-by: Eduardo Apolinario <[email protected]>

* Fix the type of optional[int] in dataclass (#1135)

Signed-off-by: Kevin Su <[email protected]>

* fix: plugins/flytekit-papermill/dev-requirements.txt to reduce vulnerabilities (#1154)

The following vulnerabilities are fixed by pinning transitive dependencies:
- https://snyk.io/vuln/SNYK-PYTHON-OAUTHLIB-3021142
- https://snyk.io/vuln/SNYK-PYTHON-PYSPARK-3021131

Signed-off-by: Eduardo Apolinario <[email protected]>

* Using sidecar handler to run Papermill task (#1143)

* remove nb prefix

Signed-off-by: Kevin Su <[email protected]>

* add tests

Signed-off-by: Kevin Su <[email protected]>

* Update requirements.in

Signed-off-by: Kevin Su <[email protected]>

* remove _

Signed-off-by: Kevin Su <[email protected]>

Signed-off-by: Kevin Su <[email protected]>

* fix: plugins/flytekit-papermill/dev-requirements.txt to reduce vulnerabilities (#1145)

The following vulnerabilities are fixed by pinning transitive dependencies:
- https://snyk.io/vuln/SNYK-PYTHON-COOKIECUTTER-2414281

* Bump pyspark from 3.2.1 to 3.2.2 in /plugins/flytekit-papermill (#1130)

Bumps [pyspark](https://github.com/apache/spark) from 3.2.1 to 3.2.2.
- [Release notes](https://github.com/apache/spark/releases)
- [Commits](apache/spark@v3.2.1...v3.2.2)

---
updated-dependencies:
- dependency-name: pyspark
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix: plugins/flytekit-papermill/dev-requirements.txt to reduce vulnerabilities (#1154)

The following vulnerabilities are fixed by pinning transitive dependencies:
- https://snyk.io/vuln/SNYK-PYTHON-OAUTHLIB-3021142
- https://snyk.io/vuln/SNYK-PYTHON-PYSPARK-3021131

Signed-off-by: Calvin Leather <[email protected]>
Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
Signed-off-by: Matheus Moreno <[email protected]>
Signed-off-by: Niels Bantilan <[email protected]>
Signed-off-by: Rahul Mehta <[email protected]>
Signed-off-by: Vanshika Chowdhary <[email protected]>
Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Calvin Leather <[email protected]>
Co-authored-by: Yee Hing Tong <[email protected]>
Co-authored-by: Kevin Su <[email protected]>
Co-authored-by: Matheus Moreno <[email protected]>
Co-authored-by: Eduardo Apolinario <[email protected]>
Co-authored-by: Niels Bantilan <[email protected]>
Co-authored-by: Rahul Mehta <[email protected]>
Co-authored-by: Vanshika Chowdhary <[email protected]>
Co-authored-by: Vanshika Chowdhary <[email protected]>
Co-authored-by: Snyk bot <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants