Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cleanup flytekit dependencies that bloat the package and are not used #216

Merged
merged 3 commits into from
Oct 21, 2020
Merged

Cleanup flytekit dependencies that bloat the package and are not used #216

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a919ea4
Cleanup flytekit installations
Oct 21, 2020
bd683bc
up to 0.14.0
Oct 21, 2020
7af8a73
fmt fix
Oct 21, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,4 +49,4 @@ dev-requirements.txt: dev-requirements.in requirements.txt install-piptools
$(call PIP_COMPILE,dev-requirements.in)

.PHONY: requirements
requirements: requirements.txt dev-requirements.txt ## Compile requirements
requirements: requirements.txt dev-requirements.txt requirements-spark3.txt ## Compile requirements
8 changes: 4 additions & 4 deletions dev-requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,8 @@ coverage==5.3 # via -r dev-requirements.in
flake8-black==0.2.1 # via -r dev-requirements.in
flake8-isort==4.0.0 # via -r dev-requirements.in
flake8==3.8.4 # via -r dev-requirements.in, flake8-black, flake8-isort
iniconfig==1.0.1 # via pytest
isort==5.5.4 # via -r dev-requirements.in, flake8-isort
iniconfig==1.1.1 # via pytest
isort==5.6.4 # via -r dev-requirements.in, flake8-isort
mccabe==0.6.1 # via flake8
mock==4.0.2 # via -r dev-requirements.in
packaging==20.4 # via pytest
Expand All @@ -24,8 +24,8 @@ pycodestyle==2.6.0 # via flake8
pyflakes==2.2.0 # via flake8
pyparsing==2.4.7 # via packaging
pytest==6.1.1 # via -r dev-requirements.in
regex==2020.9.27 # via -c requirements.txt, black
regex==2020.10.15 # via -c requirements.txt, black
six==1.15.0 # via -c requirements.txt, packaging
testfixtures==6.14.2 # via flake8-isort
testfixtures==6.15.0 # via flake8-isort
toml==0.10.1 # via -c requirements.txt, black, pytest
typed-ast==1.4.1 # via -c requirements.txt, black
7 changes: 0 additions & 7 deletions flytekit/plugins/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,6 @@
hmsclient = _lazy_loader.lazy_load_module("hmsclient") # type: _lazy_loader._LazyLoadModule
type(hmsclient).add_sub_module("genthrift.hive_metastore.ttypes")

torch = _lazy_loader.lazy_load_module("torch") # type: _lazy_loader._LazyLoadModule

tensorflow = _lazy_loader.lazy_load_module("tensorflow") # type: _lazy_loader._LazyLoadModule

_lazy_loader.LazyLoadPlugin("spark", ["pyspark>=2.4.0,<3.0.0"], [pyspark])

_lazy_loader.LazyLoadPlugin("spark3", ["pyspark>=3.0.0"], [pyspark])
Expand All @@ -31,6 +27,3 @@

_lazy_loader.LazyLoadPlugin("hive_sensor", ["hmsclient>=0.0.1,<1.0.0"], [hmsclient])

_lazy_loader.LazyLoadPlugin("pytorch", ["torch>=1.0.0,<2.0.0"], [torch])

_lazy_loader.LazyLoadPlugin("tensorflow", ["tensorflow>=2.0.0,<3.0.0"], [tensorflow])
57 changes: 16 additions & 41 deletions requirements-spark3.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,33 +5,24 @@
# make requirements-spark3.txt
#
-e file:.#egg=flytekit # via -r requirements-spark3.in
absl-py==0.10.0 # via tensorboard, tensorflow
ansiwrap==0.8.4 # via papermill
appdirs==1.4.4 # via black
appnope==0.1.0 # via ipykernel, ipython
astunparse==1.6.3 # via tensorflow
async-generator==1.10 # via nbclient
attrs==20.2.0 # via black, jsonschema
backcall==0.2.0 # via ipython
black==19.10b0 # via flytekit, papermill
boto3==1.15.14 # via flytekit
botocore==1.18.14 # via boto3, s3transfer
cachetools==4.1.1 # via google-auth
boto3==1.16.1 # via flytekit
botocore==1.19.1 # via boto3, s3transfer
certifi==2020.6.20 # via requests
chardet==3.0.4 # via requests
click==7.1.2 # via black, flytekit, hmsclient, papermill
croniter==0.3.34 # via flytekit
croniter==0.3.35 # via flytekit
decorator==4.4.2 # via ipython, retry
deprecated==1.2.10 # via flytekit
entrypoints==0.3 # via papermill
flyteidl==0.18.9 # via flytekit
future==0.18.2 # via torch
gast==0.3.3 # via tensorflow
google-auth-oauthlib==0.4.1 # via tensorboard
google-auth==1.22.1 # via google-auth-oauthlib, tensorboard
google-pasta==0.2.0 # via tensorflow
grpcio==1.32.0 # via flytekit, tensorboard, tensorflow
h5py==2.10.0 # via tensorflow
grpcio==1.33.1 # via flytekit
hmsclient==0.1.1 # via flytekit
idna==2.10 # via requests
ipykernel==5.3.4 # via flytekit
Expand All @@ -43,30 +34,24 @@ jsonschema==3.2.0 # via nbformat
jupyter-client==6.1.7 # via ipykernel, nbclient
jupyter-core==4.6.3 # via jupyter-client, nbformat
k8s-proto==0.0.3 # via flytekit
keras-preprocessing==1.1.2 # via tensorflow
keyring==21.4.0 # via flytekit
markdown==3.3 # via tensorboard
natsort==7.0.1 # via croniter
nbclient==0.5.0 # via papermill
nbformat==5.0.7 # via nbclient, papermill
nbclient==0.5.1 # via papermill
nbformat==5.0.8 # via nbclient, papermill
nest-asyncio==1.4.1 # via nbclient
numpy==1.18.5 # via flytekit, h5py, keras-preprocessing, opt-einsum, pandas, pyarrow, tensorboard, tensorflow, torch
oauthlib==3.1.0 # via requests-oauthlib
opt-einsum==3.3.0 # via tensorflow
numpy==1.19.2 # via flytekit, pandas, pyarrow
pandas==1.1.3 # via flytekit
papermill==2.2.0 # via flytekit
parso==0.7.1 # via jedi
pathspec==0.8.0 # via black
pexpect==4.8.0 # via ipython
pickleshare==0.7.5 # via ipython
prompt-toolkit==3.0.7 # via ipython
protobuf==3.13.0 # via flyteidl, flytekit, k8s-proto, tensorboard, tensorflow
prompt-toolkit==3.0.8 # via ipython
protobuf==3.13.0 # via flyteidl, flytekit, k8s-proto
ptyprocess==0.6.0 # via pexpect
py4j==0.10.9 # via pyspark
py==1.9.0 # via retry
pyarrow==0.17.1 # via flytekit
pyasn1-modules==0.2.8 # via google-auth
pyasn1==0.4.8 # via pyasn1-modules, rsa
pygments==2.7.1 # via ipython
pyrsistent==0.17.3 # via jsonschema
pyspark==3.0.1 # via flytekit
Expand All @@ -75,35 +60,25 @@ pytimeparse==1.1.8 # via flytekit
pytz==2018.4 # via flytekit, pandas
pyyaml==5.3.1 # via papermill
pyzmq==19.0.2 # via jupyter-client
regex==2020.9.27 # via black
requests-oauthlib==1.3.0 # via google-auth-oauthlib
requests==2.24.0 # via flytekit, papermill, requests-oauthlib, responses, tensorboard
regex==2020.10.15 # via black
requests==2.24.0 # via flytekit, papermill, responses
responses==0.12.0 # via flytekit
retry==0.9.2 # via flytekit
rsa==4.6 # via google-auth
s3transfer==0.3.3 # via boto3
six==1.15.0 # via absl-py, astunparse, flytekit, google-auth, google-pasta, grpcio, h5py, jsonschema, keras-preprocessing, protobuf, python-dateutil, responses, tenacity, tensorboard, tensorflow, thrift
six==1.15.0 # via flytekit, grpcio, jsonschema, protobuf, python-dateutil, responses, tenacity, thrift
sortedcontainers==2.2.2 # via flytekit
statsd==3.3.0 # via flytekit
tenacity==6.2.0 # via papermill
tensorboard-plugin-wit==1.7.0 # via tensorboard
tensorboard==2.3.0 # via tensorflow
tensorflow-estimator==2.3.0 # via tensorflow
tensorflow==2.3.1 # via flytekit
termcolor==1.1.0 # via tensorflow
textwrap3==0.9.2 # via ansiwrap
thrift==0.13.0 # via hmsclient
toml==0.10.1 # via black
torch==1.6.0 # via flytekit
tornado==6.0.4 # via ipykernel, jupyter-client
tqdm==4.50.1 # via papermill
traitlets==5.0.4 # via ipykernel, ipython, jupyter-client, jupyter-core, nbclient, nbformat
tqdm==4.50.2 # via papermill
traitlets==5.0.5 # via ipykernel, ipython, jupyter-client, jupyter-core, nbclient, nbformat
typed-ast==1.4.1 # via black
urllib3==1.25.10 # via botocore, flytekit, requests, responses
urllib3==1.25.11 # via botocore, flytekit, requests, responses
wcwidth==0.2.5 # via prompt-toolkit
werkzeug==1.0.1 # via tensorboard
wheel==0.35.1 # via astunparse, tensorboard, tensorflow
wrapt==1.12.1 # via deprecated, flytekit, tensorflow
wrapt==1.12.1 # via deprecated, flytekit

# The following packages are considered to be unsafe in a requirements file:
# setuptools
57 changes: 16 additions & 41 deletions requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,33 +5,24 @@
# make requirements.txt
#
-e file:.#egg=flytekit # via -r requirements.in
absl-py==0.10.0 # via tensorboard, tensorflow
ansiwrap==0.8.4 # via papermill
appdirs==1.4.4 # via black
appnope==0.1.0 # via ipykernel, ipython
astunparse==1.6.3 # via tensorflow
async-generator==1.10 # via nbclient
attrs==20.2.0 # via black, jsonschema
backcall==0.2.0 # via ipython
black==19.10b0 # via flytekit, papermill
boto3==1.15.14 # via flytekit
botocore==1.18.14 # via boto3, s3transfer
cachetools==4.1.1 # via google-auth
boto3==1.16.1 # via flytekit
botocore==1.19.1 # via boto3, s3transfer
certifi==2020.6.20 # via requests
chardet==3.0.4 # via requests
click==7.1.2 # via black, flytekit, hmsclient, papermill
croniter==0.3.34 # via flytekit
croniter==0.3.35 # via flytekit
decorator==4.4.2 # via ipython, retry
deprecated==1.2.10 # via flytekit
entrypoints==0.3 # via papermill
flyteidl==0.18.9 # via flytekit
future==0.18.2 # via torch
gast==0.3.3 # via tensorflow
google-auth-oauthlib==0.4.1 # via tensorboard
google-auth==1.22.1 # via google-auth-oauthlib, tensorboard
google-pasta==0.2.0 # via tensorflow
grpcio==1.32.0 # via flytekit, tensorboard, tensorflow
h5py==2.10.0 # via tensorflow
grpcio==1.33.1 # via flytekit
hmsclient==0.1.1 # via flytekit
idna==2.10 # via requests
ipykernel==5.3.4 # via flytekit
Expand All @@ -43,30 +34,24 @@ jsonschema==3.2.0 # via nbformat
jupyter-client==6.1.7 # via ipykernel, nbclient
jupyter-core==4.6.3 # via jupyter-client, nbformat
k8s-proto==0.0.3 # via flytekit
keras-preprocessing==1.1.2 # via tensorflow
keyring==21.4.0 # via flytekit
markdown==3.3 # via tensorboard
natsort==7.0.1 # via croniter
nbclient==0.5.0 # via papermill
nbformat==5.0.7 # via nbclient, papermill
nbclient==0.5.1 # via papermill
nbformat==5.0.8 # via nbclient, papermill
nest-asyncio==1.4.1 # via nbclient
numpy==1.18.5 # via flytekit, h5py, keras-preprocessing, opt-einsum, pandas, pyarrow, tensorboard, tensorflow, torch
oauthlib==3.1.0 # via requests-oauthlib
opt-einsum==3.3.0 # via tensorflow
numpy==1.19.2 # via flytekit, pandas, pyarrow
pandas==1.1.3 # via flytekit
papermill==2.2.0 # via flytekit
parso==0.7.1 # via jedi
pathspec==0.8.0 # via black
pexpect==4.8.0 # via ipython
pickleshare==0.7.5 # via ipython
prompt-toolkit==3.0.7 # via ipython
protobuf==3.13.0 # via flyteidl, flytekit, k8s-proto, tensorboard, tensorflow
prompt-toolkit==3.0.8 # via ipython
protobuf==3.13.0 # via flyteidl, flytekit, k8s-proto
ptyprocess==0.6.0 # via pexpect
py4j==0.10.7 # via pyspark
py==1.9.0 # via retry
pyarrow==0.17.1 # via flytekit
pyasn1-modules==0.2.8 # via google-auth
pyasn1==0.4.8 # via pyasn1-modules, rsa
pygments==2.7.1 # via ipython
pyrsistent==0.17.3 # via jsonschema
pyspark==2.4.7 # via flytekit
Expand All @@ -75,35 +60,25 @@ pytimeparse==1.1.8 # via flytekit
pytz==2018.4 # via flytekit, pandas
pyyaml==5.3.1 # via papermill
pyzmq==19.0.2 # via jupyter-client
regex==2020.9.27 # via black
requests-oauthlib==1.3.0 # via google-auth-oauthlib
requests==2.24.0 # via flytekit, papermill, requests-oauthlib, responses, tensorboard
regex==2020.10.15 # via black
requests==2.24.0 # via flytekit, papermill, responses
responses==0.12.0 # via flytekit
retry==0.9.2 # via flytekit
rsa==4.6 # via google-auth
s3transfer==0.3.3 # via boto3
six==1.15.0 # via absl-py, astunparse, flytekit, google-auth, google-pasta, grpcio, h5py, jsonschema, keras-preprocessing, protobuf, python-dateutil, responses, tenacity, tensorboard, tensorflow, thrift
six==1.15.0 # via flytekit, grpcio, jsonschema, protobuf, python-dateutil, responses, tenacity, thrift
sortedcontainers==2.2.2 # via flytekit
statsd==3.3.0 # via flytekit
tenacity==6.2.0 # via papermill
tensorboard-plugin-wit==1.7.0 # via tensorboard
tensorboard==2.3.0 # via tensorflow
tensorflow-estimator==2.3.0 # via tensorflow
tensorflow==2.3.1 # via flytekit
termcolor==1.1.0 # via tensorflow
textwrap3==0.9.2 # via ansiwrap
thrift==0.13.0 # via hmsclient
toml==0.10.1 # via black
torch==1.6.0 # via flytekit
tornado==6.0.4 # via ipykernel, jupyter-client
tqdm==4.50.1 # via papermill
traitlets==5.0.4 # via ipykernel, ipython, jupyter-client, jupyter-core, nbclient, nbformat
tqdm==4.50.2 # via papermill
traitlets==5.0.5 # via ipykernel, ipython, jupyter-client, jupyter-core, nbclient, nbformat
typed-ast==1.4.1 # via black
urllib3==1.25.10 # via botocore, flytekit, requests, responses
urllib3==1.25.11 # via botocore, flytekit, requests, responses
wcwidth==0.2.5 # via prompt-toolkit
werkzeug==1.0.1 # via tensorboard
wheel==0.35.1 # via astunparse, tensorboard, tensorflow
wrapt==1.12.1 # via deprecated, flytekit, tensorflow
wrapt==1.12.1 # via deprecated, flytekit

# The following packages are considered to be unsafe in a requirements file:
# setuptools