Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Caching of offloaded objects #762

Merged
merged 41 commits into from
Mar 2, 2022
Merged

Caching of offloaded objects #762

Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
06e960f
Remove flyteidl from install_requires
eapolinario Dec 2, 2021
c9047a8
Expose hash in Literal
eapolinario Dec 2, 2021
4d21957
Set hash in TypeEngine
eapolinario Dec 2, 2021
f9273db
Modify cache key calculation to take hash into account
eapolinario Dec 2, 2021
53dff4d
Opt-in PandasDataFrameTransformer
eapolinario Dec 2, 2021
5ae247c
Add unit tests
eapolinario Dec 2, 2021
9305746
Iterate using a flyteidl branch
eapolinario Dec 3, 2021
faff038
Merge remote-tracking branch 'origin' into offloaded-objects-caching
eapolinario Dec 3, 2021
ecd8b93
Regenerate requirements files
eapolinario Dec 3, 2021
44b72f6
Regenerate requirements files
eapolinario Dec 3, 2021
f46dc74
Merge remote-tracking branch 'origin' into offloaded-objects-caching
eapolinario Jan 19, 2022
22c90d3
Move _hash_overridable to StructureDatasetTransformerEngine
eapolinario Jan 21, 2022
fe7e8f7
Merge remote-tracking branch 'origin' into offloaded-objects-caching
eapolinario Jan 25, 2022
6ac4b44
Move HashMethod to flytekit.core.hash
eapolinario Jan 26, 2022
1702961
Merge remote-tracking branch 'origin' into offloaded-objects-caching
eapolinario Feb 17, 2022
4552e78
Fix `unit_test` make target
eapolinario Feb 17, 2022
c9fc044
Merge remote-tracking branch 'origin' into offloaded-objects-caching
eapolinario Feb 17, 2022
24f67a0
Split `unit_test` make target in two lines
eapolinario Feb 17, 2022
60ecf3a
Add assert to structured dataset compatibility test
eapolinario Feb 17, 2022
3aeedeb
Remove TODO
eapolinario Feb 17, 2022
c2dbb54
Regenerate plugins requirements files pointing to the right version o…
eapolinario Feb 17, 2022
0e58199
Set hash as a property of the literal
eapolinario Feb 17, 2022
5054861
Install plugins requirements in CI.
eapolinario Feb 17, 2022
9f2d06f
Add hash.setter
eapolinario Feb 18, 2022
e039836
Install flyteidl directly
eapolinario Feb 18, 2022
2da76f3
Revert "Regenerate plugins requirements files pointing to the right v…
eapolinario Feb 18, 2022
adaa448
wip - Add support for univariate lists
eapolinario Feb 18, 2022
4b5f608
Add support for lists of annotated objects
eapolinario Feb 18, 2022
ecffa04
Revamp generation of cache key (to cover case of literals collections…
eapolinario Feb 23, 2022
d4b0b49
Leave TODO for warning
eapolinario Feb 23, 2022
4d54c59
Revert "Add support for lists of annotated objects"
eapolinario Feb 23, 2022
82bbc1f
Revert "wip - Add support for univariate lists"
eapolinario Feb 23, 2022
a21631b
Remove docstring
eapolinario Mar 1, 2022
a432b1e
Merge remote-tracking branch 'origin' into offloaded-objects-caching
eapolinario Mar 1, 2022
3519149
Add flyteidl>=0.23.0
eapolinario Mar 2, 2022
09e736d
Remove mentions to branch flyteidl@add-hash-to-literal
eapolinario Mar 2, 2022
9ec103c
Bump flyteidl in plugins requirements
eapolinario Mar 2, 2022
fa4e3b2
Regenerate plugins requirements again
eapolinario Mar 2, 2022
373b3bb
Restore papermill/requirements.txt
eapolinario Mar 2, 2022
403830d
Point flytekitplugins-spark to the offloaded-objects-caching branch i…
eapolinario Mar 2, 2022
9754a0f
Set flyteidl>=0.23.0 in papermill dev-requirements
eapolinario Mar 2, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 21 additions & 26 deletions dev-requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#
# This file is autogenerated by pip-compile with python 3.8
# This file is autogenerated by pip-compile with python 3.9
# To update, run:
#
# make dev-requirements.txt
Expand Down Expand Up @@ -44,7 +44,7 @@ chardet==4.0.0
# via
# -c requirements.txt
# binaryornot
charset-normalizer==2.0.7
charset-normalizer==2.0.9
# via
# -c requirements.txt
# requests
Expand All @@ -67,13 +67,13 @@ cookiecutter==1.7.3
# via
# -c requirements.txt
# flytekit
coverage[toml]==6.1.1
coverage[toml]==6.2
# via -r dev-requirements.in
croniter==1.0.15
croniter==1.1.0
# via
# -c requirements.txt
# flytekit
cryptography==35.0.0
cryptography==36.0.0
# via
# -c requirements.txt
# paramiko
Expand All @@ -90,7 +90,7 @@ deprecated==1.2.13
# via
# -c requirements.txt
# flytekit
diskcache==5.2.1
diskcache==5.3.0
# via
# -c requirements.txt
# flytekit
Expand All @@ -112,21 +112,17 @@ dockerpty==0.4.1
# via docker-compose
docopt==0.6.2
# via docker-compose
docstring-parser==0.12
docstring-parser==0.13
# via
# -c requirements.txt
# flytekit
filelock==3.3.2
filelock==3.4.0
# via virtualenv
flyteidl==0.21.8
grpcio==1.42.0
# via
# -c requirements.txt
# flytekit
grpcio==1.41.1
# via
# -c requirements.txt
# flytekit
identify==2.3.5
identify==2.4.0
# via pre-commit
idna==3.3
# via
Expand Down Expand Up @@ -159,15 +155,15 @@ jsonschema==3.2.0
# via
# -c requirements.txt
# docker-compose
keyring==23.2.1
keyring==23.4.0
# via
# -c requirements.txt
# flytekit
markupsafe==2.0.1
# via
# -c requirements.txt
# jinja2
marshmallow==3.14.0
marshmallow==3.14.1
# via
# -c requirements.txt
# dataclasses-json
Expand Down Expand Up @@ -201,15 +197,15 @@ numpy==1.21.4
# -c requirements.txt
# pandas
# pyarrow
packaging==21.2
packaging==21.3
# via
# -c requirements.txt
# pytest
pandas==1.3.4
# via
# -c requirements.txt
# flytekit
paramiko==2.8.0
paramiko==2.8.1
# via
# -c requirements.txt
# docker
Expand All @@ -223,19 +219,18 @@ poyo==0.5.0
# via
# -c requirements.txt
# cookiecutter
pre-commit==2.15.0
pre-commit==2.16.0
# via -r dev-requirements.in
protobuf==3.19.1
# via
# -c requirements.txt
# flyteidl
# flytekit
py==1.11.0
# via
# -c requirements.txt
# pytest
# retry
pyarrow==6.0.0
pyarrow==6.0.1
# via
# -c requirements.txt
# flytekit
Expand All @@ -247,7 +242,7 @@ pynacl==1.4.0
# via
# -c requirements.txt
# paramiko
pyparsing==2.4.7
pyparsing==3.0.6
# via
# -c requirements.txt
# packaging
Expand All @@ -271,7 +266,7 @@ python-dateutil==2.8.1
# croniter
# flytekit
# pandas
python-dotenv==0.19.1
python-dotenv==0.19.2
# via docker-compose
python-json-logger==2.0.2
# via
Expand All @@ -295,7 +290,7 @@ pyyaml==5.4.1
# -c requirements.txt
# docker-compose
# pre-commit
regex==2021.11.9
regex==2021.11.10
# via
# -c requirements.txt
# docker-image-py
Expand All @@ -307,7 +302,7 @@ requests==2.26.0
# docker-compose
# flytekit
# responses
responses==0.15.0
responses==0.16.0
# via
# -c requirements.txt
# flytekit
Expand Down Expand Up @@ -356,7 +351,7 @@ tomli==1.2.2
# via
# -c requirements.txt
# coverage
typing-extensions==3.10.0.2
typing-extensions==4.0.1
# via
# -c requirements.txt
# mypy
Expand Down
Loading