Skip to content

Commit

Permalink
Pydantic Transformer V2 (#2792)
Browse files Browse the repository at this point in the history 
* Pydantic Transformer V2

Signed-off-by: Future-Outlier <[email protected]>

* add __init__.py

Signed-off-by: Future-Outlier <[email protected]>

* add json schema

Signed-off-by: Future-Outlier <[email protected]>

* convert float to int

Signed-off-by: Future-Outlier <[email protected]>

* change gitsha in test script mode

Signed-off-by: Future-Outlier <[email protected]>

* change gitsha

Signed-off-by: Future-Outlier <[email protected]>

* use strict map=false

Signed-off-by: Future-Outlier <[email protected]>

* Test flytefile console input + attr access

Signed-off-by: Future-Outlier <[email protected]>

* add conditional branch

Signed-off-by: Future-Outlier <[email protected]>

* better rx

Signed-off-by: Future-Outlier <[email protected]>

* Add flytedir generic -> flytedir

Signed-off-by: Future-Outlier <[email protected]>

* support enum

Signed-off-by: Future-Outlier <[email protected]>

* update

Signed-off-by: Future-Outlier <[email protected]>

* add tests for input from flyte console

Signed-off-by: Future-Outlier <[email protected]>

* Add Tests for dataclass in BaseModel and pydantic.dataclass in BaseModel

Signed-off-by: Future-Outlier <[email protected]>

* update

Signed-off-by: Future-Outlier <[email protected]>

* update thomas's advice

Signed-off-by: Future-Outlier <[email protected]>

* change tree file structure

Signed-off-by: Future-Outlier <[email protected]>

* update Niel's advice

Signed-off-by: Future-Outlier <[email protected]>

* > to >=

Signed-off-by: Future-Outlier <[email protected]>

* try monodoc build again

Signed-off-by: Future-Outlier <[email protected]>

* add pydantic README.md

Signed-off-by: Future-Outlier <[email protected]>

* revert -vvv in monodocs

Signed-off-by: Future-Outlier <[email protected]>

* use model_validate_json to turn protobuf struct to python val

Signed-off-by: Future-Outlier <[email protected]>

* fix issue

Signed-off-by: Future-Outlier <[email protected]>

* handle flyte types in dict transformer from protobuf struct input (e.g FlyteConsole)

Signed-off-by: Future-Outlier <[email protected]>

* Add print

Signed-off-by: Future-Outlier <[email protected]>

* expected python type

Signed-off-by: Future-Outlier <[email protected]>

* switch call function order

Signed-off-by: Future-Outlier <[email protected]>

* try msgpack to handle protobug struct

Signed-off-by: Future-Outlier <[email protected]>

* Better Comment in Dict Transformer

Signed-off-by: Future-Outlier <[email protected]>

* Propeller -> FlytePropeller

Signed-off-by: Future-Outlier <[email protected]>

* dict_to_flyte_types

Signed-off-by: Future-Outlier <[email protected]>

* remove comments

Signed-off-by: Future-Outlier <[email protected]>

* add attr for protobuf struct . dict

Signed-off-by: Future-Outlier <[email protected]>

* Add Life Cycle for Flyte Types

Signed-off-by: Future-Outlier <[email protected]>

* better comments for derializing flyteschema and sd

Signed-off-by: Future-Outlier <[email protected]>

* nit

Signed-off-by: Future-Outlier <[email protected]>

* add back pv._remote_path = None to flytefile and flytedir

Signed-off-by: Future-Outlier <[email protected]>

* experiment

Signed-off-by: Future-Outlier <[email protected]>

* experiment

Signed-off-by: Future-Outlier <[email protected]>

* experiment

Signed-off-by: Future-Outlier <[email protected]>

* Add comments

Signed-off-by: Future-Outlier <[email protected]>

* update Yee's advice

Signed-off-by: Future-Outlier <[email protected]>

* code dc -> bm

Signed-off-by: Future-Outlier <[email protected]>

* Example dc -> bm and Example all flyte types

Signed-off-by: Future-Outlier <[email protected]>

* fix union dataclass, not yet add comments

Signed-off-by: Future-Outlier <[email protected]>

* add pydantic and dataclass optional test

Signed-off-by: Future-Outlier <[email protected]>

* NoneType=type(None)

Signed-off-by: Future-Outlier <[email protected]>

* fix union transformer none case with Eduardo

Signed-off-by: Future-Outlier <[email protected]>

* add comments for none type transformer + union transformer

Signed-off-by: Future-Outlier <[email protected]>

* add TODO

Signed-off-by: Future-Outlier <[email protected]>

* use deserailize = True

Signed-off-by: Future-Outlier <[email protected]>

* add all deserialize

Signed-off-by: Future-Outlier <[email protected]>

* better comments

Signed-off-by: Future-Outlier <[email protected]>

* better comments

Signed-off-by: Future-Outlier <[email protected]>

* test

Signed-off-by: Future-Outlier <[email protected]>

* Fix flyte directory issue by discussion with Kevin

Signed-off-by: Future-Outlier <[email protected]>
Co-authored-by: pingsutw  <[email protected]>

* add tests for providing conext when doing serialization

Signed-off-by: Future-Outlier <[email protected]>

* lint

Signed-off-by: Future-Outlier <[email protected]>

* test

Signed-off-by: Future-Outlier <[email protected]>

* move setattr to core

Signed-off-by: Future-Outlier <[email protected]>

* remove comments

Signed-off-by: Future-Outlier <[email protected]>

* lint

Signed-off-by: Future-Outlier <[email protected]>

* remove

Signed-off-by: Future-Outlier <[email protected]>

* testing

Signed-off-by: Future-Outlier <[email protected]>

* nit

Signed-off-by: Future-Outlier <[email protected]>

* flytekit/core/type_engine.py

Signed-off-by: Future-Outlier <[email protected]>

* add tests for Union

Signed-off-by: Future-Outlier <[email protected]>

* Trigger CI

Signed-off-by: Future-Outlier <[email protected]>

* remove nonetype

Signed-off-by: Future-Outlier <[email protected]>

* raw=Fasle as default

Signed-off-by: Future-Outlier <[email protected]>

* pydantic move to core test

Signed-off-by: Future-Outlier <[email protected]>

* move to core

Signed-off-by: Future-Outlier <[email protected]>

* log

Signed-off-by: Future-Outlier <[email protected]>

* update

Signed-off-by: Future-Outlier <[email protected]>

* lint

Signed-off-by: Future-Outlier <[email protected]>

* test

Signed-off-by: Future-Outlier <[email protected]>

* nit

Signed-off-by: Future-Outlier <[email protected]>

* nit

Signed-off-by: Future-Outlier <[email protected]>

* lint

Signed-off-by: Future-Outlier <[email protected]>

* move to type_engine

Signed-off-by: Future-Outlier <[email protected]>

* move back to init

Signed-off-by: Future-Outlier <[email protected]>

* update kevin's advice

Signed-off-by: Future-Outlier <[email protected]>
Co-authored-by: pingsutw  <[email protected]>

* wip

Signed-off-by: Future-Outlier <[email protected]>

* use decorator

Signed-off-by: Future-Outlier <[email protected]>

* decorator

Signed-off-by: Future-Outlier <[email protected]>

* fix syntax to support python 3.9

Signed-off-by: Future-Outlier <[email protected]>

* add Eduardo's advice

Signed-off-by: Future-Outlier <[email protected]>

* warning

Signed-off-by: Future-Outlier <[email protected]>

* Update Yee's advice

Signed-off-by: Future-Outlier <[email protected]>
Co-authored-by: Yee Hing Tong  <[email protected]>

* Show traceback by default (#2862)

Signed-off-by: Kevin Su <[email protected]>

* Support Identifier in generate_console_url (#2868)

Signed-off-by: Thomas J. Fan <[email protected]>

* Support overriding node metadata for array node (#2865)

* set parent node metadata on arraynode subnode

Signed-off-by: Paul Dittamo <[email protected]>

* clean up setting passed in metadata for array node

Signed-off-by: Paul Dittamo <[email protected]>

* add comment

Signed-off-by: Paul Dittamo <[email protected]>

* test

Signed-off-by: Paul Dittamo <[email protected]>

---------

Signed-off-by: Paul Dittamo <[email protected]>

* Fix Jupyter Versioning  (#2866)

* Update versioning

Signed-off-by: Mecoli1219 <[email protected]>

* Fix unit test

Signed-off-by: Mecoli1219 <[email protected]>

* Update path if jupyter file actually exist

Signed-off-by: Mecoli1219 <[email protected]>

* Fix pickled twice problem

Signed-off-by: Mecoli1219 <[email protected]>

---------

Signed-off-by: Mecoli1219 <[email protected]>

* improved output handling in notebooks (#2869)

---------

Signed-off-by: Future-Outlier <[email protected]>
Co-authored-by: pingsutw <[email protected]>
Co-authored-by: Yee Hing Tong <[email protected]>
  • Loading branch information
@Future-Outlier @pingsutw @wild-endeavor
3 people authored Oct 28, 2024
1 parent 6d3d3e7 commit ff2d0da
Show file tree
Hide file tree
Showing 28 changed files with 3,082 additions and 447 deletions.
1 change: 0 additions & 1 deletion .github/workflows/pythonbuild.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -356,7 +356,6 @@ jobs:
- flytekit-pandera
- flytekit-papermill
- flytekit-polars
- flytekit-pydantic
- flytekit-ray
- flytekit-snowflake
- flytekit-spark
Expand Down
1 change: 1 addition & 0 deletions dev-requirements.in
View file
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ torch<=1.12.1; python_version<'3.11'
# pytorch 2 supports python 3.11
# pytorch 2 does not support 3.12 yet: https://github.com/pytorch/pytorch/issues/110436
torch; python_version<'3.12'
pydantic

# TODO: Currently, the python-magic library causes build errors on Windows due to its dependency on DLLs for libmagic.
# We have temporarily disabled this feature on Windows and are using python-magic for Mac OS and Linux instead.
Expand Down
Loading

0 comments on commit ff2d0da

Please sign in to comment.