Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Docs] Use Pure Dataclass In Example #5829

Merged
merged 6 commits into from
Oct 9, 2024
Merged

[Docs] Use Pure Dataclass In Example #5829

Changes from 4 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
fa4a0d7
[Docs] Use Pure Dataclass In Example
Future-Outlier Oct 9, 2024
44d198f
Update docs/user_guide/data_types_and_io/dataclass.md
Future-Outlier Oct 9, 2024
6710b19
Update docs/user_guide/data_types_and_io/dataclass.md
Future-Outlier Oct 9, 2024
9a4078e
Update docs/user_guide/data_types_and_io/dataclass.md
Future-Outlier Oct 9, 2024
6f90ff4
better solution
Future-Outlier Oct 9, 2024
389b097
Update docs/user_guide/data_types_and_io/dataclass.md
Future-Outlier Oct 9, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 20 additions & 15 deletions docs/user_guide/data_types_and_io/dataclass.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,10 @@ Flytekit uses the [Mashumaro library](https://github.com/Fatal1ty/mashumaro)
to serialize and deserialize dataclasses.

:::{important}
If you're using Flytekit version below v1.10, you'll need to decorate with `@dataclass_json` using
`from dataclasses_json import dataclass_json` instead of inheriting from Mashumaro's `DataClassJSONMixin`.

If you're using Flytekit version >= v1.11.1, you don't need to decorate with `@dataclass_json` or
If you're using Flytekit version below v1.11.1, your dataclass will need to decorate with `@dataclass_json` or
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to clarify, this means that if you are using Flytekit < 1.11.1, you either need to decorate your dataclass with @dataclass_json or create a class that inherits DataClassJSONMixin, but that you don't need to do both? If you inherit from DataClassJSONMixin, do you need to do from mashumaro.mixins.json import DataClassJSONMixin as well?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
If you're using Flytekit version below v1.11.1, your dataclass will need to decorate with `@dataclass_json` or
If you're using Flytekit version greater than v1.10 and less than v1.11.1, you will need add `from mashumaro.mixins.json import DataClassJSONMixin` to your imports and either decorate your dataclass with `@dataclass_json` or create a class that inherits from `DataClassJSONMixin`.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me know if this is accurate!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's not, but I can update it

inherit from Mashumaro's `DataClassJSONMixin`.

If you're using Flytekit version less than v1.10, you'll need to add `from dataclasses_json import dataclass_json` to your imports and decorate your dataclass with `@dataclass_json`.
:::

```{note}
Expand All @@ -25,15 +24,21 @@ To clone and run the example code on this page, see the [Flytesnacks repo][flyte

To begin, import the necessary dependencies:

```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/dataclass.py
```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/cfb5ea3b0d0502ef7df1f2e14f4a0d9b78250b6a/examples/data_types_and_io/data_types_and_io/dataclass.py
:caption: data_types_and_io/dataclass.py
:lines: 1-9
```

Build your custom image with ImageSpec:
```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/cfb5ea3b0d0502ef7df1f2e14f4a0d9b78250b6a/examples/data_types_and_io/data_types_and_io/dataclass.py
:caption: data_types_and_io/dataclass.py
:lines: 1-10
:lines: 16-19
```

## Python types
We define a `dataclass` with `int`, `str` and `dict` as the data types.

```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/dataclass.py
```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/cfb5ea3b0d0502ef7df1f2e14f4a0d9b78250b6a/examples/data_types_and_io/data_types_and_io/dataclass.py
:caption: data_types_and_io/dataclass.py
:pyobject: Datum
```
Expand All @@ -46,41 +51,41 @@ All variables in a data class should be **annotated with their type**. Failure t

Once declared, a dataclass can be returned as an output or accepted as an input.

```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/dataclass.py
```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/cfb5ea3b0d0502ef7df1f2e14f4a0d9b78250b6a/examples/data_types_and_io/data_types_and_io/dataclass.py
:caption: data_types_and_io/dataclass.py
:lines: 28-43
:lines: 32-47
```

## Flyte types
We also define a data class that accepts {std:ref}`StructuredDataset <structured_dataset>`,
{std:ref}`FlyteFile <files>` and {std:ref}`FlyteDirectory <folder>`.

```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/dataclass.py
```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/cfb5ea3b0d0502ef7df1f2e14f4a0d9b78250b6a/examples/data_types_and_io/data_types_and_io/dataclass.py
:caption: data_types_and_io/dataclass.py
:lines: 47-84
:lines: 51-88
```

A data class supports the usage of data associated with Python types, data classes,
flyte file, flyte directory and structured dataset.

We define a workflow that calls the tasks created above.

```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/dataclass.py
```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/cfb5ea3b0d0502ef7df1f2e14f4a0d9b78250b6a/examples/data_types_and_io/data_types_and_io/dataclass.py
:caption: data_types_and_io/dataclass.py
:pyobject: dataclass_wf
```

You can run the workflow locally as follows:

```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/dataclass.py
```{rli} https://raw.githubusercontent.com/flyteorg/flytesnacks/cfb5ea3b0d0502ef7df1f2e14f4a0d9b78250b6a/examples/data_types_and_io/data_types_and_io/dataclass.py
:caption: data_types_and_io/dataclass.py
:lines: 97-98
:lines: 101-102
```

To trigger a task that accepts a dataclass as an input with `pyflyte run`, you can provide a JSON file as an input:
```
pyflyte run \
https://raw.githubusercontent.com/flyteorg/flytesnacks/69dbe4840031a85d79d9ded25f80397c6834752d/examples/data_types_and_io/data_types_and_io/dataclass.py \
https://raw.githubusercontent.com/flyteorg/flytesnacks/cfb5ea3b0d0502ef7df1f2e14f4a0d9b78250b6a/examples/data_types_and_io/data_types_and_io/dataclass.py \
add --x dataclass_input.json --y dataclass_input.json
```

Expand Down
Loading