Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Docs] MessagePack IDL, Pydantic Support, and Attribute Access #6022

Merged
merged 20 commits into from
Nov 21, 2024
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion docs/user_guide/data_types_and_io/dataclass.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,16 @@ Flytekit uses the [Mashumaro library](https://github.com/Fatal1ty/mashumaro)
to serialize and deserialize dataclasses.
Future-Outlier marked this conversation as resolved.
Show resolved Hide resolved

:::{important}
If you're using Flytekit version below v1.11.1, you will need to add `from dataclasses_json import dataclass_json` to your imports and decorate your dataclass with `@dataclass_json`.
If you're using Flytekit version < v1.11.1, you will need to add `from dataclasses_json import dataclass_json` to your imports and decorate your dataclass with `@dataclass_json`.
:::

:::{important}
Future-Outlier marked this conversation as resolved.
Show resolved Hide resolved
Flytekit version < v1.14.0 will produce protobuf struct literal for dataclasses.

Flytekit version >= v1.14.0 will produce msgpack bytes literal for dataclasses.

If you're using Flytekit version >= v1.14.0 and you want to produce protobuf struct literal for dataclasses, you can
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be good to mention why would a user want to produce protobuf struct literal instead of msgpack bytes

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Flytekit version < v1.14.0 will produce protobuf struct literal for dataclasses.
Flytekit version >= v1.14.0 will produce msgpack bytes literal for dataclasses.
If you're using Flytekit version >= v1.14.0 and you want to produce protobuf struct literal for dataclasses, you can

set environment variable `FLYTE_USE_OLD_DC_FORMAT` to `true`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
set environment variable `FLYTE_USE_OLD_DC_FORMAT` to `true`.

This was already mentioned above

Also in the readthedocs build, you can see there are two important blocks nested

:::

```{note}
Expand Down
1 change: 1 addition & 0 deletions docs/user_guide/data_types_and_io/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -144,6 +144,7 @@ flytefile
flytedirectory
structureddataset
dataclass
pydantic
accessing_attributes
pytorch_type
enum_type
Expand Down
21 changes: 21 additions & 0 deletions docs/user_guide/data_types_and_io/pydantic.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
(pydantic)=

# Pydantic BaseModel

```{eval-rst}
.. tags:: Basic
```

When you've multiple values that you want to send across Flyte entities, and you want them to have, you can use a `pydantic.BaseModel`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure I understand this line

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed

Note:
You can put Dataclass and FlyteTypes (FlyteFile, FlyteDirectory, FlyteSchema, and StructuredDataset) in a pydantic BaseModel.

:::{important}
Pydantic BaseModel V2 only works when you are using flytekit version >= v1.14.0.
:::

:::{important}
If you're using Flytekit version >= v1.14.0 and you want to produce protobuf struct literal for pydantic basemodels,
you can set environment variable `FLYTE_USE_OLD_DC_FORMAT` to `true`.
:::

Loading