Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add dask plugin #patch #1366

Merged
merged 31 commits into from
Jan 12, 2023
Merged

Add dask plugin #patch #1366

Show file tree
Hide file tree
Changes from 22 commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
eddb152
Add dummy task type to test backend plugin
bstadlbauer Jun 26, 2022
407d5cb
Add docs page
bstadlbauer Dec 10, 2022
992d5e7
Add dask models
bstadlbauer Dec 10, 2022
d7bdf47
Add function to convert resources
bstadlbauer Dec 10, 2022
641d2ac
Add tests to `dask` task
bstadlbauer Dec 10, 2022
96162bf
Remove namespace
bstadlbauer Dec 11, 2022
6092b6f
Update setup.py
bstadlbauer Dec 12, 2022
1caceff
Add dask to `plugin/README.md`
bstadlbauer Dec 12, 2022
afa54df
Add README.md for `dask`
bstadlbauer Dec 12, 2022
49bdab3
Top level export of `JopPodSpec` and `DaskCluster`
bstadlbauer Dec 12, 2022
96270f3
Update docs for images
bstadlbauer Dec 12, 2022
dfc4131
Update README.md
bstadlbauer Dec 12, 2022
de2af6d
Update models after `flyteidl` change
bstadlbauer Dec 17, 2022
8633ecc
Update task after `flyteidl` change
bstadlbauer Dec 17, 2022
528a085
Merge remote-tracking branch 'origin/master' into add-dask-plugin
bstadlbauer Dec 17, 2022
019529d
Raise error when less than 1 worker
bstadlbauer Dec 17, 2022
b04fb3d
Update flyteidl to >= 1.3.2
bstadlbauer Jan 5, 2023
4d5e445
Update doc requirements
bstadlbauer Jan 5, 2023
a5311ae
Merge branch 'master' into add-dask-plugin
bstadlbauer Jan 5, 2023
f703c09
Update doc-requirements.txt
bstadlbauer Jan 5, 2023
0465291
Re-lock dependencies on linux
Jan 5, 2023
f02d9dc
Update dask API docs
bstadlbauer Jan 5, 2023
b50be3e
Merge branch 'master' into add-dask-plugin
bstadlbauer Jan 9, 2023
167d654
Fix documentation links
bstadlbauer Jan 9, 2023
dab1bc6
Default optional model constructor arguments to `None`
bstadlbauer Jan 9, 2023
3722b0a
Refactor `convert_resources_to_resource_model` to `core.resources`
bstadlbauer Jan 9, 2023
4f829b7
Use `convert_resources_to_resource_model` in `core.node`
bstadlbauer Jan 9, 2023
43147cb
Merge branch 'master' into add-dask-plugin
bstadlbauer Jan 10, 2023
1e6e3e1
Merge branch 'master' into add-dask-plugin
eapolinario Jan 11, 2023
aadf78c
Incorporate review feedback
eapolinario Jan 12, 2023
8c62437
Lint
eapolinario Jan 12, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/pythonbuild.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,7 @@ jobs:
- flytekit-aws-batch
- flytekit-aws-sagemaker
- flytekit-bigquery
- flytekit-dask
- flytekit-data-fsspec
- flytekit-dbt
- flytekit-deck-standard
Expand Down
1 change: 1 addition & 0 deletions doc-requirements.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,4 +45,5 @@ whylogs # whylogs
whylabs-client # whylogs
ray # ray
scikit-learn # scikit-learn
dask[distributed] # dask
vaex # vaex
Loading