[release v1.2] Backport v1.6 #1645

eapolinario · 2023-05-17T00:57:23Z

TL;DR

Please replace this text with a description of what this PR accomplishes.

Type

Bug Fix
Feature
Plugin

Are all requirements met?

Complete description

How did you fix the bug, make the feature etc. Link to any design docs etc

Tracking Issue

https://github.com/flyteorg/flyte/issues/

Follow-up issue

NA
OR
https://github.com/flyteorg/flyte/issues/

Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Yee Hing Tong <[email protected]>

* Remove retry from flytekit's setup.py and regenerate requirements Signed-off-by: Eduardo Apolinario <[email protected]> * Add to sagemaker Signed-off-by: Eduardo Apolinario <[email protected]> * Remove retry from sagemaker plugin requirements file Signed-off-by: Eduardo Apolinario <[email protected]> * Restore doc-requirements.txt Signed-off-by: eduardo apolinario <[email protected]> * Fix bad merge Signed-off-by: eduardo apolinario <[email protected]> --------- Signed-off-by: Eduardo Apolinario <[email protected]> Signed-off-by: eduardo apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]>

Signed-off-by: Yee Hing Tong <[email protected]>

Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Niels Bantilan <[email protected]> Signed-off-by: Yee Hing Tong <[email protected]>

* fix: Make FlyteFile compatible with Annotated[..., HashMethod] See issue #3424 Signed-off-by: Adrian Rumpold <[email protected]> * tests: Add test case for FlyteFile with HashMethod annotation Issue: #3424 Signed-off-by: Adrian Rumpold <[email protected]> * fix: Use typing_extensions.Annotated for py3.8 compatibility Issue: #3424 Signed-off-by: Adrian Rumpold <[email protected]> * fix: Use `get_args` and `get_origin` from typing_extensions for py3.8 compatibility Issue: #3424 Signed-off-by: Adrian Rumpold <[email protected]> * fix(tests): Use fixture for local dummy file Signed-off-by: Adrian Rumpold <[email protected]> --------- Signed-off-by: Adrian Rumpold <[email protected]>

* move FlyteSchema deprecation warning to initialization method Signed-off-by: Niels Bantilan <[email protected]> * update Signed-off-by: Niels Bantilan <[email protected]> --------- Signed-off-by: Niels Bantilan <[email protected]>

…1515) * add pod_template and pod_template_name arguments for ContainerTask Signed-off-by: Felix Ruess <[email protected]> * factor out _serialize_pod_spec into separate util function Signed-off-by: Felix Ruess <[email protected]> * model file changes, couple other changes Signed-off-by: Yee Hing Tong <[email protected]> * minor cleanup Signed-off-by: Felix Ruess <[email protected]> * add unit test for container_task pod_template Signed-off-by: Felix Ruess <[email protected]> * bump min version of flyteidl to 1.3.12 for pod template data config support Signed-off-by: Felix Ruess <[email protected]> * require flyteidl==1.3.12 in doc-requirements.txt Signed-off-by: Felix Ruess <[email protected]> --------- Signed-off-by: Felix Ruess <[email protected]> Signed-off-by: Yee Hing Tong <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]>

Signed-off-by: franco-bocci <[email protected]>

* TypeTransformer for TensorFlow model Signed-off-by: Samhita Alla <[email protected]> * clean up Signed-off-by: Samhita Alla <[email protected]> * clean up Signed-off-by: Samhita Alla <[email protected]> * fix lint Signed-off-by: Samhita Alla <[email protected]> --------- Signed-off-by: Samhita Alla <[email protected]>

Signed-off-by: Yee Hing Tong <[email protected]>

* Device auth flow Signed-off-by: Ketan Umare <[email protected]> * Device AuthFlow is now available in flytekit Signed-off-by: Ketan Umare <[email protected]> * unit tests Signed-off-by: Ketan Umare <[email protected]> * test added Signed-off-by: Ketan Umare <[email protected]> * updated Signed-off-by: Ketan Umare <[email protected]> * Fixed test Signed-off-by: Ketan Umare <[email protected]> * Fixed unit test Signed-off-by: Ketan Umare <[email protected]> * Fixed lint errors Signed-off-by: Ketan Umare <[email protected]> --------- Signed-off-by: Ketan Umare <[email protected]>

* url encode secret first Signed-off-by: Yee Hing Tong <[email protected]> * nit Signed-off-by: Yee Hing Tong <[email protected]> --------- Signed-off-by: Yee Hing Tong <[email protected]>

* wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> * update comment Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]>

#1556) Signed-off-by: Ketan Umare <[email protected]>

Co-authored-by: ggydush-fn <[email protected]>

Signed-off-by: Yee Hing Tong <[email protected]>

* Automatically remove unused import Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]>

* Disallow partial lists in map tasks Signed-off-by: eduardo apolinario <[email protected]> * Lint Signed-off-by: eduardo apolinario <[email protected]> --------- Signed-off-by: eduardo apolinario <[email protected]> Co-authored-by: eduardo apolinario <[email protected]>

Signed-off-by: Yee Hing Tong <[email protected]>

* Convert List[Any] to a single pickle file * remove redundant code * keep batchSize only if type contain flytePickle * fix error * add batch support to translate_inputs_to_literals * add ci test Signed-off-by: Yicheng-Lu-llll <[email protected]> * improve comment Signed-off-by: Yicheng-Lu-llll <[email protected]> * add more ci test Signed-off-by: Yicheng-Lu-llll <[email protected]> * improve Signed-off-by: Yicheng-Lu-llll <[email protected]> * handle HashMethod case Signed-off-by: Yicheng-Lu-llll <[email protected]> * improve format Signed-off-by: Yicheng-Lu-llll <[email protected]> * improve Signed-off-by: Yicheng-Lu-llll <[email protected]> * improve Signed-off-by: Yicheng-Lu-llll <[email protected]> * improve Signed-off-by: Yicheng-Lu-llll <[email protected]> * add test_is_batchable Signed-off-by: Yicheng-Lu-llll <[email protected]> * Add BatchSize Signed-off-by: Kevin Su <[email protected]> * test_batch_size Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Yicheng-Lu-llll <[email protected]> Signed-off-by: Kevin Su <[email protected]> Co-authored-by: root <[email protected]> Co-authored-by: Kevin Su <[email protected]>

Signed-off-by: Samhita Alla <[email protected]>

* Unify sqlalchemy Dockerfiles Signed-off-by: eduardo apolinario <[email protected]> * Simplifyte sqlalchemy Dockerfile Signed-off-by: eduardo apolinario <[email protected]> * Set pythonpath to /root Signed-off-by: eduardo apolinario <[email protected]> --------- Signed-off-by: eduardo apolinario <[email protected]> Co-authored-by: eduardo apolinario <[email protected]>

* wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * pyflyte build Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * Support serialize and package Signed-off-by: Kevin Su <[email protected]> * more tests Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * move to plugin Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * test Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * fixed tested Signed-off-by: Kevin Su <[email protected]> * fixed tested Signed-off-by: Kevin Su <[email protected]> * more tests Signed-off-by: Kevin Su <[email protected]> * Add support passing yaml in pyflyte run Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]>

…ror messages (#1582) * improve input type conversion error Signed-off-by: Niels Bantilan <[email protected]> * fix lint Signed-off-by: Niels Bantilan <[email protected]> * fix Signed-off-by: Niels Bantilan <[email protected]> * add tests Signed-off-by: Niels Bantilan <[email protected]> * add tests Signed-off-by: Niels Bantilan <[email protected]> * add rich Signed-off-by: Niels Bantilan <[email protected]> * fix lint Signed-off-by: Niels Bantilan <[email protected]> * remove prototyping script, update loggers Signed-off-by: Niels Bantilan <[email protected]> * update __init__.py Signed-off-by: Niels Bantilan <[email protected]> * update logger Signed-off-by: Niels Bantilan <[email protected]> * update logger Signed-off-by: Niels Bantilan <[email protected]> * fix GE and pandera tests Signed-off-by: Niels Bantilan <[email protected]> * fix lint Signed-off-by: Niels Bantilan <[email protected]> --------- Signed-off-by: Niels Bantilan <[email protected]>

* Add support nested FlyteFile in pyflyte run Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]>

* Set cache_regions=True in the case of s3fs * Linting Signed-off-by: eduardo apolinario <[email protected]> * Fix tests Signed-off-by: eduardo apolinario <[email protected]> * More linting Signed-off-by: eduardo apolinario <[email protected]> --------- Signed-off-by: eduardo apolinario <[email protected]> Co-authored-by: eduardo apolinario <[email protected]>

* set default ComSpec when running on Windows Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]>

* add activate-launchplan command to pyflyte Signed-off-by: Artem Petrov <[email protected]> * fix linting Signed-off-by: Artem Petrov <[email protected]> * rework launchplan command Signed-off-by: Artem Petrov <[email protected]> * fix linting Signed-off-by: Artem Petrov <[email protected]> --------- Signed-off-by: Artem Petrov <[email protected]>

Signed-off-by: Yee Hing Tong <[email protected]>

* checkpoint Signed-off-by: Mike Zhong <[email protected]> * Experimental implementation works well. Instead of messing with the class, we utilize the interpolation and context to dynamically generate a wrapper around the desired script. In the wrapper, we cd to the ctx.working_directory, export the env variables (handled by an additional method to convert dict to str), and then pass the arguments to the script Signed-off-by: Mike Zhong <[email protected]> * fix spacing Signed-off-by: Mike Zhong <[email protected]> * remove breakpoint Signed-off-by: Mike Zhong <[email protected]> * Output ctx.working_directory as single output Signed-off-by: Mike Zhong <[email protected]> * more doc strings Signed-off-by: Mike Zhong <[email protected]> * more comments Signed-off-by: Mike Zhong <[email protected]> * Added comments Signed-off-by: Mike Zhong <[email protected]> * Add tests and test files from other branch Signed-off-by: Mike Zhong <[email protected]> * minor fix, have function return the instance rather than create. It seems to get registered when flyte packages your project Signed-off-by: Mike Zhong <[email protected]> * fix tests Signed-off-by: Mike Zhong <[email protected]> * remove set flags not supported by sh Signed-off-by: Mike Zhong <[email protected]> * fix spellcheck lint errors Signed-off-by: Mike Zhong <[email protected]> * don't have a windows equivalent test script so bypassing those tests for now Signed-off-by: Mike Zhong <[email protected]> * Add typing to make_export_string_from_env_dict params Signed-off-by: Mike Zhong <[email protected]> * fixup doc string Signed-off-by: Mike Zhong <[email protected]> * Refactored the new behavior out into a new class. Did not change implementation details at all Signed-off-by: Mike Zhong <[email protected]> * Address linter issues and up test cov Signed-off-by: Mike Zhong <[email protected]> * Skip test on windows, no equivalent script Signed-off-by: Mike Zhong <[email protected]> * Run black and isort, address SC2236 Signed-off-by: Mike Zhong <[email protected]> * Refactored class name to remove _, make utility function require name so multiple uses in a workflow don't break Signed-off-by: Mike Zhong <[email protected]> * fix utility function Signed-off-by: Mike Zhong <[email protected]> * fix utility function call in tests Signed-off-by: Mike Zhong <[email protected]> * Fix isort on test_shell.py Signed-off-by: Mike Zhong <[email protected]> * adding logging settings to papermill plugin Signed-off-by: Calvin Leather <[email protected]> * set level to info logging * improved comments/docs --------- Signed-off-by: Mike Zhong <[email protected]> Signed-off-by: Calvin Leather <[email protected]> Co-authored-by: Mike Zhong <[email protected]> Co-authored-by: Mike Zhong <[email protected]>

* wip Signed-off-by: Kevin Su <[email protected]> * Fix mypy errors Signed-off-by: Kevin Su <[email protected]> * Fix mypy errors Signed-off-by: Kevin Su <[email protected]> * Fix tests Signed-off-by: Kevin Su <[email protected]> * Fix tests Signed-off-by: Kevin Su <[email protected]> * Fix tests Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * fix tests Signed-off-by: Kevin Su <[email protected]> * fix tests Signed-off-by: Kevin Su <[email protected]> * fix test Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * Update type Signed-off-by: Kevin Su <[email protected]> * Fix tests Signed-off-by: Kevin Su <[email protected]> * Fix tests Signed-off-by: Kevin Su <[email protected]> * Fix tests Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * update dev-requirements.txt Signed-off-by: Kevin Su <[email protected]> * Address comment Signed-off-by: Kevin Su <[email protected]> * upgrade torch Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * lint Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]>

Signed-off-by: eduardo apolinario <[email protected]>

codecov · 2023-05-17T01:12:10Z

Codecov Report

❗ No coverage uploaded for pull request base (release-v1.2@f7b59bc). Click here to learn what that means.
The diff coverage is n/a.

@@               Coverage Diff               @@
##             release-v1.2    #1645   +/-   ##
===============================================
  Coverage                ?   71.13%           
===============================================
  Files                   ?      331           
  Lines                   ?    30348           
  Branches                ?     2907           
===============================================
  Hits                    ?    21588           
  Misses                  ?     8208           
  Partials                ?      552

Signed-off-by: eduardo apolinario <[email protected]>

This reverts commit af49155. Signed-off-by: Eduardo Apolinario <[email protected]>

Signed-off-by: eduardo apolinario <[email protected]>

wild-endeavor and others added 30 commits May 16, 2023 14:00

Data subsystem (#1526)

4555578

Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Yee Hing Tong <[email protected]>

Update the pypi wait (#1554)

2731766

Signed-off-by: Yee Hing Tong <[email protected]>

Stream Directories and Files using Flyte (#1512)

0c47d46

Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Niels Bantilan <[email protected]> Signed-off-by: Yee Hing Tong <[email protected]>

Pass locally defined scopes to RemoteClientConfigStore (#1553)

b22e5e3

Signed-off-by: franco-bocci <[email protected]>

Remove fsspec flytekit plugin from main Dockerfile (#1561)

bd6a7e5

Signed-off-by: Yee Hing Tong <[email protected]>

url encode secret in client credentials flow (#1566)

7e76d4b

* url encode secret first Signed-off-by: Yee Hing Tong <[email protected]> * nit Signed-off-by: Yee Hing Tong <[email protected]> --------- Signed-off-by: Yee Hing Tong <[email protected]>

General Partial support in flytekit and multi-list support in flytekit (

9a31a01

#1556) Signed-off-by: Ketan Umare <[email protected]>

fix: Silence keyring warnings by changing to debug (#1568)

d8c10c6

Co-authored-by: ggydush-fn <[email protected]>

Support GCP secrets (#1571)

06ea83b

Signed-off-by: Yee Hing Tong <[email protected]>

Automatically remove unused import (#1574)

57e21c2

* Automatically remove unused import Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]>

Remove duplicate reporting logic (#1578)

0c0c2ac

Signed-off-by: Yee Hing Tong <[email protected]>

Improve authoring structure documentation (#1572)

8941718

Signed-off-by: Samhita Alla <[email protected]>

nit (#1596)

34ea031

Signed-off-by: Yee Hing Tong <[email protected]>

pingsutw and others added 5 commits May 16, 2023 15:54

type-protobuf<4

725cdb7

Signed-off-by: eduardo apolinario <[email protected]>

Allow for python 3.7

02f7830

Signed-off-by: eduardo apolinario <[email protected]>

Fix protobuf transformer merge error

6836d62

Fix bad merge of s/_task_models/task_models/

bbda9ad

eapolinario requested review from wild-endeavor, kumare3, pingsutw and cosmicBboy as code owners May 17, 2023 00:57

eapolinario mentioned this pull request May 17, 2023

[release v1.2] Backport v1.6 #1644

Closed

8 tasks

Add maxsize=128 to @lru_cache to cover python 3.7

8914849

Signed-off-by: eduardo apolinario <[email protected]>

eapolinario added 16 commits May 16, 2023 18:39

Lint

2178128

Signed-off-by: eduardo apolinario <[email protected]>

Regenerate doc-requirements.txt

6059a60

Signed-off-by: eduardo apolinario <[email protected]>

Add missing call to lru_cache

f5e4ae5

Signed-off-by: eduardo apolinario <[email protected]>

Bump version and regenerate requirements in a few plugins

f24f261

Signed-off-by: eduardo apolinario <[email protected]>

Revert "Fix mypy errors (#1313)"

14fb5b5

This reverts commit af49155. Signed-off-by: Eduardo Apolinario <[email protected]>

Remove docs

184b173

Signed-off-by: eduardo apolinario <[email protected]>

More linting

61d0afc

Signed-off-by: eduardo apolinario <[email protected]>

Fix type of get_type_for_output_var

5ee2067

Signed-off-by: eduardo apolinario <[email protected]>

Remove tensorflow test_model

2fd8094

Replace use of shutil.copytree

a947a45

Signed-off-by: eduardo apolinario <[email protected]>

Other use of copytree

e36e4f6

Signed-off-by: eduardo apolinario <[email protected]>

Stringify paths in test_script_mode

d5547e7

Signed-off-by: eduardo apolinario <[email protected]>

os.path.isfile instead in imagespec

7a01c0a

Signed-off-by: eduardo apolinario <[email protected]>

Remove tests

f1c71bb

Signed-off-by: eduardo apolinario <[email protected]>

More linting

d10cac7

Signed-off-by: eduardo apolinario <[email protected]>

Remove flaky test

1540d1c

Signed-off-by: eduardo apolinario <[email protected]>

wild-endeavor approved these changes May 17, 2023

View reviewed changes

eapolinario merged commit 4fd5d81 into release-v1.2 May 17, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[release v1.2] Backport v1.6 #1645

[release v1.2] Backport v1.6 #1645

eapolinario commented May 17, 2023

codecov bot commented May 17, 2023 •

edited

Loading

[release v1.2] Backport v1.6 #1645

[release v1.2] Backport v1.6 #1645

Conversation

eapolinario commented May 17, 2023

TL;DR

Type

Are all requirements met?

Complete description

Tracking Issue

Follow-up issue

codecov bot commented May 17, 2023 • edited Loading

Codecov Report

codecov bot commented May 17, 2023 •

edited

Loading