Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TypeTransformers for PyTorch Tensor, Module, and Checkpoint #1032

Merged
merged 19 commits into from
Jul 7, 2022
Merged

TypeTransformers for PyTorch Tensor, Module, and Checkpoint #1032

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
4974353
TypeTransformers for PyTorch Tensor and Module
samhita-alla Jun 1, 2022
fa5e3ae
add torch to requirements
samhita-alla Jun 1, 2022
6d8f977
add module as a native type and PyTorchCheckpoint
samhita-alla Jun 13, 2022
3560adf
resolve merge conflict
samhita-alla Jun 13, 2022
bf61ef7
update requirements
samhita-alla Jun 14, 2022
f4b8704
resolve merge conflict
samhita-alla Jun 14, 2022
7b9218f
procedural to OOP approach
samhita-alla Jun 16, 2022
6332364
nit
samhita-alla Jun 16, 2022
9f098ba
verify device conversion
samhita-alla Jun 17, 2022
b8111a6
verify device conversion
samhita-alla Jun 17, 2022
13fd0f7
hyperparameters can be None
samhita-alla Jun 17, 2022
7b299f4
device conversion
samhita-alla Jun 20, 2022
4ef601c
device conversion
samhita-alla Jun 20, 2022
c8d5478
checkpoint code cleanup
samhita-alla Jun 20, 2022
b644a2e
resolve merge conflict
samhita-alla Jun 23, 2022
88a6ce1
move pytorch from types to extra; resolve merge conflict
samhita-alla Jul 4, 2022
6915913
fix pytorch api reference; resolve merge conflict
samhita-alla Jul 4, 2022
6ce6e19
fix pytorch import
samhita-alla Jul 6, 2022
a48472a
fix merge conflict
samhita-alla Jul 6, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions dev-requirements.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,3 +11,4 @@ codespell
google-cloud-bigquery
google-cloud-bigquery-storage
IPython
torch
39 changes: 15 additions & 24 deletions dev-requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@
# via
# -c requirements.txt
# pytest-flyte
appnope==0.1.3
# via ipython
arrow==1.2.2
# via
# -c requirements.txt
Expand All @@ -28,7 +30,7 @@ binaryornot==0.4.4
# cookiecutter
cached-property==1.5.2
# via docker-compose
cachetools==5.1.0
cachetools==5.2.0
# via google-auth
certifi==2022.5.18.1
# via
Expand Down Expand Up @@ -61,7 +63,7 @@ cloudpickle==2.1.0
# flytekit
codespell==2.1.0
# via -r dev-requirements.in
cookiecutter==1.7.3
cookiecutter==2.1.0
# via
# -c requirements.txt
# flytekit
Expand All @@ -76,7 +78,6 @@ cryptography==37.0.2
# -c requirements.txt
# paramiko
# pyopenssl
# secretstorage
dataclasses-json==0.5.7
# via
# -c requirements.txt
Expand Down Expand Up @@ -119,13 +120,13 @@ docstring-parser==0.14.1
# via
# -c requirements.txt
# flytekit
filelock==3.7.0
filelock==3.7.1
# via virtualenv
flyteidl==1.0.1
# via
# -c requirements.txt
# flytekit
google-api-core[grpc]==2.8.0
google-api-core[grpc]==2.8.1
# via
# google-cloud-bigquery
# google-cloud-bigquery-storage
Expand All @@ -146,7 +147,7 @@ google-crc32c==1.3.0
# via google-resumable-media
google-resumable-media==2.3.3
# via google-cloud-bigquery
googleapis-common-protos==1.56.1
googleapis-common-protos==1.56.2
# via
# -c requirements.txt
# flyteidl
Expand Down Expand Up @@ -182,15 +183,10 @@ importlib-metadata==4.11.4
# virtualenv
iniconfig==1.1.1
# via pytest
ipython==7.33.0
ipython==7.34.0
# via -r dev-requirements.in
jedi==0.18.1
# via ipython
jeepney==0.8.0
# via
# -c requirements.txt
# keyring
# secretstorage
jinja2==3.1.2
# via
# -c requirements.txt
Expand All @@ -215,7 +211,7 @@ markupsafe==2.1.1
# via
# -c requirements.txt
# jinja2
marshmallow==3.15.0
marshmallow==3.16.0
# via
# -c requirements.txt
# dataclasses-json
Expand Down Expand Up @@ -274,15 +270,11 @@ platformdirs==2.5.2
# via virtualenv
pluggy==1.0.0
# via pytest
poyo==0.5.0
# via
# -c requirements.txt
# cookiecutter
pre-commit==2.19.0
# via -r dev-requirements.in
prompt-toolkit==3.0.29
# via ipython
proto-plus==1.20.4
proto-plus==1.20.5
# via
# google-cloud-bigquery
# google-cloud-bigquery-storage
Expand Down Expand Up @@ -378,6 +370,7 @@ pytz==2022.1
pyyaml==5.4.1
# via
# -c requirements.txt
# cookiecutter
# docker-compose
# flytekit
# pre-commit
Expand Down Expand Up @@ -405,18 +398,13 @@ retry==0.9.2
# flytekit
rsa==4.8
# via google-auth
secretstorage==3.3.2
# via
# -c requirements.txt
# keyring
singledispatchmethod==1.0
# via
# -c requirements.txt
# flytekit
six==1.16.0
# via
# -c requirements.txt
# cookiecutter
# dockerpty
# google-auth
# grpcio
Expand Down Expand Up @@ -446,7 +434,9 @@ tomli==2.0.1
# coverage
# mypy
# pytest
traitlets==5.2.1.post0
torch==1.11.0
# via -r dev-requirements.in
traitlets==5.2.2.post1
# via
# ipython
# matplotlib-inline
Expand All @@ -460,6 +450,7 @@ typing-extensions==4.2.0
# importlib-metadata
# mypy
# responses
# torch
# typing-inspect
typing-inspect==0.7.1
# via
Expand Down
1 change: 1 addition & 0 deletions doc-requirements.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,3 +33,4 @@ papermill # papermill
jupyter # papermill
pyspark # spark
sqlalchemy # sqlalchemy
torch # pytorch
44 changes: 21 additions & 23 deletions doc-requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,10 @@ altair==4.2.0
# via great-expectations
ansiwrap==0.8.4
# via papermill
appnope==0.1.3
# via
# ipykernel
# ipython
argon2-cffi==21.3.0
# via notebook
argon2-cffi-bindings==21.2.0
Expand Down Expand Up @@ -42,9 +46,9 @@ binaryornot==0.4.4
# via cookiecutter
bleach==5.0.0
# via nbconvert
botocore==1.26.8
botocore==1.27.0
# via -r doc-requirements.in
cachetools==5.1.0
cachetools==5.2.0
# via google-auth
certifi==2022.5.18.1
# via
Expand All @@ -68,7 +72,7 @@ cloudpickle==2.1.0
# via flytekit
colorama==0.4.4
# via great-expectations
cookiecutter==1.7.3
cookiecutter==2.1.0
# via flytekit
croniter==1.3.5
# via flytekit
Expand All @@ -77,7 +81,6 @@ cryptography==36.0.2
# -r doc-requirements.in
# great-expectations
# pyopenssl
# secretstorage
css-html-js-minify==2.5.5
# via sphinx-material
cycler==0.11.0
Expand Down Expand Up @@ -130,7 +133,7 @@ fsspec==2022.5.0
# modin
furo @ git+https://github.com/flyteorg/furo@main
# via -r doc-requirements.in
google-api-core[grpc]==2.8.0
google-api-core[grpc]==2.8.1
# via
# google-cloud-bigquery
# google-cloud-bigquery-storage
Expand All @@ -152,12 +155,12 @@ google-crc32c==1.3.0
# via google-resumable-media
google-resumable-media==2.3.3
# via google-cloud-bigquery
googleapis-common-protos==1.56.1
googleapis-common-protos==1.56.2
# via
# flyteidl
# google-api-core
# grpcio-status
great-expectations==0.15.6
great-expectations==0.15.7
# via -r doc-requirements.in
greenlet==1.1.2
# via sqlalchemy
Expand Down Expand Up @@ -198,7 +201,7 @@ ipykernel==6.13.0
# jupyter-console
# notebook
# qtconsole
ipython==7.33.0
ipython==7.34.0
# via
# great-expectations
# ipykernel
Expand All @@ -213,10 +216,6 @@ ipywidgets==7.7.0
# via jupyter
jedi==0.18.1
# via ipython
jeepney==0.8.0
# via
# keyring
# secretstorage
jinja2==3.0.3
# via
# altair
Expand Down Expand Up @@ -275,7 +274,7 @@ kubernetes==23.6.0
# via -r doc-requirements.in
lazy-object-proxy==1.7.1
# via astroid
lxml==4.8.0
lxml==4.9.0
# via sphinx-material
markdown==3.3.7
# via -r doc-requirements.in
Expand All @@ -284,7 +283,7 @@ markupsafe==2.1.1
# jinja2
# nbconvert
# pandas-profiling
marshmallow==3.15.0
marshmallow==3.16.0
# via
# dataclasses-json
# marshmallow-enum
Expand Down Expand Up @@ -319,7 +318,7 @@ mypy-extensions==0.4.3
# via typing-inspect
natsort==8.1.0
# via flytekit
nbclient==0.6.3
nbclient==0.6.4
# via
# nbconvert
# papermill
Expand Down Expand Up @@ -415,15 +414,13 @@ pillow==9.1.1
# visions
plotly==5.8.0
# via -r doc-requirements.in
poyo==0.5.0
# via cookiecutter
prometheus-client==0.14.1
# via notebook
prompt-toolkit==3.0.29
# via
# ipython
# jupyter-console
proto-plus==1.20.4
proto-plus==1.20.5
# via
# google-cloud-bigquery
# google-cloud-bigquery-storage
Expand Down Expand Up @@ -518,6 +515,7 @@ pywavelets==1.3.0
# via imagehash
pyyaml==6.0
# via
# cookiecutter
# flytekit
# kubernetes
# pandas-profiling
Expand Down Expand Up @@ -572,16 +570,13 @@ seaborn==0.11.2
# via
# missingno
# pandas-profiling
secretstorage==3.3.2
# via keyring
send2trash==1.8.0
# via notebook
singledispatchmethod==1.0
# via flytekit
six==1.16.0
# via
# bleach
# cookiecutter
# google-auth
# grpcio
# imagehash
Expand Down Expand Up @@ -640,7 +635,7 @@ sphinxcontrib-serializinghtml==1.1.5
# via sphinx
sphinxcontrib-yt==0.2.2
# via -r doc-requirements.in
sqlalchemy==1.4.36
sqlalchemy==1.4.37
# via -r doc-requirements.in
statsd==3.3.0
# via flytekit
Expand All @@ -664,6 +659,8 @@ tinycss2==1.1.1
# via nbconvert
toolz==0.11.2
# via altair
torch==1.11.0
# via -r doc-requirements.in
tornado==6.1
# via
# ipykernel
Expand All @@ -675,7 +672,7 @@ tqdm==4.64.0
# great-expectations
# pandas-profiling
# papermill
traitlets==5.2.1.post0
traitlets==5.2.2.post1
# via
# ipykernel
# ipython
Expand Down Expand Up @@ -703,6 +700,7 @@ typing-extensions==4.2.0
# pandera
# pydantic
# responses
# torch
# typing-inspect
typing-inspect==0.7.1
# via
Expand Down
4 changes: 4 additions & 0 deletions docs/source/types.builtins.pytorch.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
.. automodule:: flytekit.types.pytorch
:no-members:
:no-inherited-members:
:no-special-members:
1 change: 1 addition & 0 deletions docs/source/types.extend.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,3 +11,4 @@ Feel free to follow the pattern of the built-in types.
types.builtins.structured
types.builtins.file
types.builtins.directory
types.builtins.pytorch
2 changes: 1 addition & 1 deletion flytekit/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -189,7 +189,7 @@
from flytekit.models.core.types import BlobType
from flytekit.models.literals import Blob, BlobMetadata, Literal, Scalar
from flytekit.models.types import LiteralType
from flytekit.types import directory, file, numpy, schema
from flytekit.types import directory, file, numpy, pytorch, schema
from flytekit.types.structured.structured_dataset import (
StructuredDataset,
StructuredDatasetFormat,
Expand Down
12 changes: 0 additions & 12 deletions flytekit/types/numpy/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1 @@
"""
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is removing this from docs intentional?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah. I don't think we'd want to have Transformer in the API reference cause the methods within the TypeTransformer class remain the same.

Flytekit Numpy
==============
.. currentmodule:: flytekit.types.numpy
.. autosummary::
:template: custom.rst
:toctree: generated/
NumpyArrayTransformer
"""

from .ndarray import NumpyArrayTransformer
Loading