[v1.1] - Backporting v1.2.0b2 changes (#1167)

* Add deck to papermill plugin task (#1111) Signed-off-by: Calvin Leather <[email protected]> * Run compilation even in local execution for dynamic tasks to early detect errors (#1121) Signed-off-by: Yee Hing Tong <[email protected]> * Set to pyflyte run blob object remote when dealing with remote files (#1128) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Eduardo Apolinario <[email protected]> * Override voidPromise resource (#1127) * override void promise resource Signed-off-by: Kevin Su <[email protected]> * override void promise resource Signed-off-by: Kevin Su <[email protected]> * Fix how ShellTask retrieves the Pod class name (#1132) * Fix how ShellTask retrieves the Pod class name Signed-off-by: Matheus Moreno <[email protected]> * Set Pod class name as a constant Signed-off-by: Matheus Moreno <[email protected]> * Revert last commit Signed-off-by: Matheus Moreno <[email protected]> * Execute automatic linting Signed-off-by: Matheus Moreno <[email protected]> Signed-off-by: Matheus Moreno <[email protected]> * Add restriction for pandas to be >=1.2 for fsspec plugin (#1136) Signed-off-by: Yee Hing Tong <[email protected]> * Use joblib hashing to generate cache key to ensure repeatability (#1126) * cherry pick 97b454b Signed-off-by: Yee Hing Tong <[email protected]> * requirements Signed-off-by: Yee Hing Tong <[email protected]> * Fix usage of save in ProtoJoblibHasher Signed-off-by: Eduardo Apolinario <[email protected]> * Regenerate requirements using python 3.7 Signed-off-by: Eduardo Apolinario <[email protected]> * Add test_stable_cache_key Signed-off-by: Eduardo Apolinario <[email protected]> Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> * Allow None protocol to mean all data persistence supported storage options in Structured Dataset (#1134) Signed-off-by: Yee Hing Tong <[email protected]> * handle ImportError and OSError in extras.pytorch (#1141) * handle ImportError and OSError in extras.pytorch Signed-off-by: Niels Bantilan <[email protected]> * isolate exception to torch import Signed-off-by: Niels Bantilan <[email protected]> Signed-off-by: Niels Bantilan <[email protected]> * Register dataframe renderers in structured dataset (#1140) * Register dataframe renderers in structured dataset Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * fix test Signed-off-by: Kevin Su <[email protected]> * more tests Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Kevin Su <[email protected]> * pyflyte run imperative workflows (#1131) Signed-off-by: Kevin Su <[email protected]> * Using sidecar handler to run Papermill task (#1143) * remove nb prefix Signed-off-by: Kevin Su <[email protected]> * add tests Signed-off-by: Kevin Su <[email protected]> * Update requirements.in Signed-off-by: Kevin Su <[email protected]> * remove _ Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Kevin Su <[email protected]> * Properly raise error in NumpyArrayTransformer (#1146) Signed-off-by: Rahul Mehta <[email protected]> Signed-off-by: Rahul Mehta <[email protected]> * Add assert_type in dataclass transformer (#1149) * Add assert_type in dataclassTransformer Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * fix tests Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * fix tests Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * more tests Signed-off-by: Kevin Su <[email protected]> * fix lint Signed-off-by: Kevin Su <[email protected]> * Add one more test Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Kevin Su <[email protected]> * Pickle in Union Type (#1147) * Pickel in Union type Signed-off-by: Kevin Su <[email protected]> * Pickel in Union type Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * fix tests Signed-off-by: Kevin Su <[email protected]> * update tests Signed-off-by: Kevin Su <[email protected]> * fix tests Signed-off-by: Kevin Su <[email protected]> * fix tests Signed-off-by: Kevin Su <[email protected]> * fix tests Signed-off-by: Kevin Su <[email protected]> * fix tests Signed-off-by: Kevin Su <[email protected]> * fix tests Signed-off-by: Kevin Su <[email protected]> * fix tests Signed-off-by: Kevin Su <[email protected]> * fix tests Signed-off-by: Kevin Su <[email protected]> * fix tests Signed-off-by: Kevin Su <[email protected]> * Address comment Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Kevin Su <[email protected]> * Bump max docker version to 7.0.0 (#1138) Signed-off-by: Rahul Mehta <[email protected]> Signed-off-by: Rahul Mehta <[email protected]> * Set flytekit<2.0 in plugins (#1152) Signed-off-by: Eduardo Apolinario <[email protected]> Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> * Add literal type to union literal (#1144) * Add literal type to union literal Signed-off-by: Kevin Su <[email protected]> * fix test Signed-off-by: Kevin Su <[email protected]> * Add tests Signed-off-by: Kevin Su <[email protected]> * more tests Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Kevin Su <[email protected]> * Fix the type of optional[int] in nested dataclass (#1148) * Fix the type of optional[int] in nested dataclass Signed-off-by: Kevin Su <[email protected]> * update tests Signed-off-by: Kevin Su <[email protected]> * update comments Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Kevin Su <[email protected]> * Added symlink dereferencing in fast packaging and tests (#1151) * Added symlink dereferencing and tests Signed-off-by: Vanshika Chowdhary <[email protected]> * Added flag to register as well Signed-off-by: Vanshika Chowdhary <[email protected]> * More flag propagation Signed-off-by: Vanshika Chowdhary <[email protected]> Signed-off-by: Vanshika Chowdhary <[email protected]> Co-authored-by: Vanshika Chowdhary <[email protected]> * Strip newline from client secret (#1163) * Strip newline from client secret * Add logging and rework the secret file comparison to work on windows Signed-off-by: Eduardo Apolinario <[email protected]> Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> * Fix the type of optional[int] in dataclass (#1135) Signed-off-by: Kevin Su <[email protected]> * fix: plugins/flytekit-papermill/dev-requirements.txt to reduce vulnerabilities (#1154) The following vulnerabilities are fixed by pinning transitive dependencies: - https://snyk.io/vuln/SNYK-PYTHON-OAUTHLIB-3021142 - https://snyk.io/vuln/SNYK-PYTHON-PYSPARK-3021131 Signed-off-by: Eduardo Apolinario <[email protected]> * Using sidecar handler to run Papermill task (#1143) * remove nb prefix Signed-off-by: Kevin Su <[email protected]> * add tests Signed-off-by: Kevin Su <[email protected]> * Update requirements.in Signed-off-by: Kevin Su <[email protected]> * remove _ Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Kevin Su <[email protected]> * fix: plugins/flytekit-papermill/dev-requirements.txt to reduce vulnerabilities (#1145) The following vulnerabilities are fixed by pinning transitive dependencies: - https://snyk.io/vuln/SNYK-PYTHON-COOKIECUTTER-2414281 * Bump pyspark from 3.2.1 to 3.2.2 in /plugins/flytekit-papermill (#1130) Bumps [pyspark](https://github.com/apache/spark) from 3.2.1 to 3.2.2. - [Release notes](https://github.com/apache/spark/releases) - [Commits](apache/spark@v3.2.1...v3.2.2) --- updated-dependencies: - dependency-name: pyspark dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix: plugins/flytekit-papermill/dev-requirements.txt to reduce vulnerabilities (#1154) The following vulnerabilities are fixed by pinning transitive dependencies: - https://snyk.io/vuln/SNYK-PYTHON-OAUTHLIB-3021142 - https://snyk.io/vuln/SNYK-PYTHON-PYSPARK-3021131 Signed-off-by: Calvin Leather <[email protected]> Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Eduardo Apolinario <[email protected]> Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Matheus Moreno <[email protected]> Signed-off-by: Niels Bantilan <[email protected]> Signed-off-by: Rahul Mehta <[email protected]> Signed-off-by: Vanshika Chowdhary <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Calvin Leather <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]> Co-authored-by: Kevin Su <[email protected]> Co-authored-by: Matheus Moreno <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Co-authored-by: Niels Bantilan <[email protected]> Co-authored-by: Rahul Mehta <[email protected]> Co-authored-by: Vanshika Chowdhary <[email protected]> Co-authored-by: Vanshika Chowdhary <[email protected]> Co-authored-by: Snyk bot <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
flyteorg · Sep 16, 2022 · c58cfb4 · c58cfb4
1 parent 3596ee4
commit c58cfb4
Show file tree

Hide file tree

Showing 77 changed files with 1,248 additions and 437 deletions.
diff --git a/dev-requirements.txt b/dev-requirements.txt
@@ -8,6 +8,8 @@
     # via
     #   -c requirements.txt
     #   pytest-flyte
+appnope==0.1.3
+    # via ipython
 arrow==1.2.2
     # via
     #   -c requirements.txt
@@ -20,7 +22,7 @@ attrs==20.3.0
     #   pytest-docker
 backcall==0.2.0
     # via ipython
-bcrypt==3.2.2
+bcrypt==4.0.0
     # via paramiko
 binaryornot==0.4.4
     # via
@@ -37,7 +39,6 @@ certifi==2022.6.15
 cffi==1.15.1
     # via
     #   -c requirements.txt
-    #   bcrypt
     #   cryptography
     #   pynacl
 cfgv==3.3.1
@@ -46,7 +47,7 @@ chardet==5.0.0
     # via
     #   -c requirements.txt
     #   binaryornot
-charset-normalizer==2.1.0
+charset-normalizer==2.1.1
     # via
     #   -c requirements.txt
     #   requests
@@ -59,13 +60,13 @@ cloudpickle==2.1.0
     # via
     #   -c requirements.txt
     #   flytekit
-codespell==2.1.0
+codespell==2.2.1
     # via -r dev-requirements.in
 cookiecutter==2.1.1
     # via
     #   -c requirements.txt
     #   flytekit
-coverage[toml]==6.4.1
+coverage[toml]==6.4.4
     # via
     #   -r dev-requirements.in
     #   pytest-cov
@@ -78,7 +79,6 @@ cryptography==37.0.4
     #   -c requirements.txt
     #   paramiko
     #   pyopenssl
-    #   secretstorage
 dataclasses-json==0.5.7
     # via
     #   -c requirements.txt
@@ -96,19 +96,17 @@ diskcache==5.4.0
     # via
     #   -c requirements.txt
     #   flytekit
-distlib==0.3.4
+distlib==0.3.6
     # via virtualenv
 distro==1.7.0
     # via docker-compose
-docker[ssh]==5.0.3
+docker[ssh]==6.0.0
     # via
     #   -c requirements.txt
     #   docker-compose
     #   flytekit
 docker-compose==1.29.2
-    # via
-    #   pytest-docker
-    #   pytest-flyte
+    # via pytest-flyte
 docker-image-py==0.1.12
     # via
     #   -c requirements.txt
@@ -121,9 +119,9 @@ docstring-parser==0.14.1
     # via
     #   -c requirements.txt
     #   flytekit
-filelock==3.7.1
+filelock==3.8.0
     # via virtualenv
-flyteidl==1.1.8
+flyteidl==1.1.12
     # via
     #   -c requirements.txt
     #   flytekit
@@ -132,23 +130,23 @@ google-api-core[grpc]==2.8.2
     #   google-cloud-bigquery
     #   google-cloud-bigquery-storage
     #   google-cloud-core
-google-auth==2.9.0
+google-auth==2.11.0
     # via
     #   google-api-core
     #   google-cloud-core
-google-cloud-bigquery==3.2.0
+google-cloud-bigquery==3.3.2
     # via -r dev-requirements.in
-google-cloud-bigquery-storage==2.14.0
+google-cloud-bigquery-storage==2.14.2
     # via
     #   -r dev-requirements.in
     #   google-cloud-bigquery
-google-cloud-core==2.3.1
+google-cloud-core==2.3.2
     # via google-cloud-bigquery
 google-crc32c==1.3.0
     # via google-resumable-media
 google-resumable-media==2.3.3
     # via google-cloud-bigquery
-googleapis-common-protos==1.56.3
+googleapis-common-protos==1.56.4
     # via
     #   -c requirements.txt
     #   flyteidl
@@ -166,7 +164,7 @@ grpcio-status==1.47.0
     #   -c requirements.txt
     #   flytekit
     #   google-api-core
-identify==2.5.1
+identify==2.5.3
     # via pre-commit
 idna==3.3
     # via
@@ -189,11 +187,6 @@ ipython==7.34.0
     # via -r dev-requirements.in
 jedi==0.18.1
     # via ipython
-jeepney==0.8.0
-    # via
-    #   -c requirements.txt
-    #   keyring
-    #   secretstorage
 jinja2==3.1.2
     # via
     #   -c requirements.txt
@@ -205,20 +198,23 @@ jinja2-time==0.2.0
     #   -c requirements.txt
     #   cookiecutter
 joblib==1.1.0
-    # via -r dev-requirements.in
+    # via
+    #   -c requirements.txt
+    #   -r dev-requirements.in
+    #   flytekit
 jsonschema==3.2.0
     # via
     #   -c requirements.txt
     #   docker-compose
-keyring==23.6.0
+keyring==23.8.2
     # via
     #   -c requirements.txt
     #   flytekit
 markupsafe==2.1.1
     # via
     #   -c requirements.txt
     #   jinja2
-marshmallow==3.17.0
+marshmallow==3.17.1
     # via
     #   -c requirements.txt
     #   dataclasses-json
@@ -232,11 +228,11 @@ marshmallow-jsonschema==0.13.0
     # via
     #   -c requirements.txt
     #   flytekit
-matplotlib-inline==0.1.3
+matplotlib-inline==0.1.6
     # via ipython
 mock==4.0.3
     # via -r dev-requirements.in
-mypy==0.961
+mypy==0.971
     # via -r dev-requirements.in
 mypy-extensions==0.4.3
     # via
@@ -258,6 +254,7 @@ numpy==1.21.6
 packaging==21.3
     # via
     #   -c requirements.txt
+    #   docker
     #   google-cloud-bigquery
     #   marshmallow
     #   pytest
@@ -281,7 +278,7 @@ pre-commit==2.20.0
     # via -r dev-requirements.in
 prompt-toolkit==3.0.30
     # via ipython
-proto-plus==1.20.6
+proto-plus==1.22.1
     # via
     #   google-cloud-bigquery
     #   google-cloud-bigquery-storage
@@ -323,7 +320,7 @@ pycparser==2.21
     # via
     #   -c requirements.txt
     #   cffi
-pygments==2.12.0
+pygments==2.13.0
     # via ipython
 pynacl==1.5.0
     # via paramiko
@@ -347,7 +344,7 @@ pytest==7.1.2
     #   pytest-flyte
 pytest-cov==3.0.0
     # via -r dev-requirements.in
-pytest-docker==0.12.0
+pytest-docker==1.0.0
     # via pytest-flyte
 pytest-flyte @ git+https://github.com/flyteorg/pytest-flyte@main
     # via -r dev-requirements.in
@@ -373,7 +370,7 @@ pytimeparse==1.1.8
     # via
     #   -c requirements.txt
     #   flytekit
-pytz==2022.1
+pytz==2022.2.1
     # via
     #   -c requirements.txt
     #   flytekit
@@ -385,7 +382,7 @@ pyyaml==5.4.1
     #   docker-compose
     #   flytekit
     #   pre-commit
-regex==2022.7.9
+regex==2022.8.17
     # via
     #   -c requirements.txt
     #   docker-image-py
@@ -407,12 +404,8 @@ retry==0.9.2
     # via
     #   -c requirements.txt
     #   flytekit
-rsa==4.8
+rsa==4.9
     # via google-auth
-secretstorage==3.3.2
-    # via
-    #   -c requirements.txt
-    #   keyring
 singledispatchmethod==1.0
     # via
     #   -c requirements.txt
@@ -426,7 +419,6 @@ six==1.16.0
     #   jsonschema
     #   paramiko
     #   python-dateutil
-    #   virtualenv
     #   websocket-client
 sortedcontainers==2.4.0
     # via
@@ -449,7 +441,7 @@ tomli==2.0.1
     #   coverage
     #   mypy
     #   pytest
-torch==1.12.0
+torch==1.12.1
     # via -r dev-requirements.in
 traitlets==5.3.0
     # via
@@ -467,17 +459,18 @@ typing-extensions==4.3.0
     #   responses
     #   torch
     #   typing-inspect
-typing-inspect==0.7.1
+typing-inspect==0.8.0
     # via
     #   -c requirements.txt
     #   dataclasses-json
-urllib3==1.26.10
+urllib3==1.26.12
     # via
     #   -c requirements.txt
+    #   docker
     #   flytekit
     #   requests
     #   responses
-virtualenv==20.15.1
+virtualenv==20.16.4
     # via pre-commit
 wcwidth==0.2.5
     # via prompt-toolkit
@@ -495,7 +488,7 @@ wrapt==1.14.1
     #   -c requirements.txt
     #   deprecated
     #   flytekit
-zipp==3.8.0
+zipp==3.8.1
     # via
     #   -c requirements.txt
     #   importlib-metadata