Get raw input/output from remote execution #675
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## master #675 +/- ##
==========================================
- Coverage 85.73% 85.72% -0.01%
==========================================
Files 343 343
Lines 29207 29260 +53
Branches 2403 2408 +5
==========================================
+ Hits 25041 25084 +43
- Misses 3525 3531 +6
- Partials 641 645 +4
Continue to review full report at Codecov.
|
kumare3
commented
Oct 1, 2021
| execution._inputs = TypeEngine.literal_map_to_kwargs( | ||
| ctx=ctx, | ||
| lm=self._get_input_literal_map(execution_data), | ||
| lm=input_literal_map, | ||
| python_types=TypeEngine.guess_python_types(interface.inputs), |
There was a problem hiding this comment.
This is a beta API, can we drop the outputs?
There was a problem hiding this comment.
i think I just want to support, if as_type is None then we can simply return the guessed type?
This will also prevent us from converting the type too eagerly?
8 tasks
|
cc @pingsutw / @wild-endeavor / @eapolinario can you guys drive this PR please? |
pingsutw
force-pushed
the
working-feast-demo
branch
from
31b517c to
b9f80c2
Compare
pingsutw
changed the title
Signed-off-by: Ketan Umare <[email protected]>
Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Kevin Su <[email protected]>
pingsutw
force-pushed
the
working-feast-demo
branch
from
8a1d09f to
66ab5e7
Compare
Signed-off-by: Kevin Su <[email protected]>
eapolinario
previously approved these changes
Dec 1, 2021
Signed-off-by: Kevin Su <[email protected]>
eapolinario
approved these changes
Dec 2, 2021
pingsutw
added a commit
that referenced
this pull request
Dec 6, 2021
* [wip] for feast demo Signed-off-by: Ketan Umare <[email protected]> * clean up a bit Signed-off-by: Yee Hing Tong <[email protected]> * add a test and move where constructor is called Signed-off-by: Yee Hing Tong <[email protected]> * remove unneeded import Signed-off-by: Yee Hing Tong <[email protected]> * add a part of a test Signed-off-by: Yee Hing Tong <[email protected]> * Added tests Signed-off-by: Kevin Su <[email protected]> * Fixed lint Signed-off-by: Kevin Su <[email protected]> * typo Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]> Co-authored-by: Kevin Su <[email protected]> Signed-off-by: Kevin Su <[email protected]>
pingsutw
added a commit
that referenced
this pull request
Dec 6, 2021
* [wip] for feast demo Signed-off-by: Ketan Umare <[email protected]> * clean up a bit Signed-off-by: Yee Hing Tong <[email protected]> * add a test and move where constructor is called Signed-off-by: Yee Hing Tong <[email protected]> * remove unneeded import Signed-off-by: Yee Hing Tong <[email protected]> * add a part of a test Signed-off-by: Yee Hing Tong <[email protected]> * Added tests Signed-off-by: Kevin Su <[email protected]> * Fixed lint Signed-off-by: Kevin Su <[email protected]> * typo Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]> Co-authored-by: Kevin Su <[email protected]> Signed-off-by: Kevin Su <[email protected]>
eapolinario
pushed a commit
that referenced
this pull request
Jan 28, 2022
* [wip] for feast demo Signed-off-by: Ketan Umare <[email protected]> * clean up a bit Signed-off-by: Yee Hing Tong <[email protected]> * add a test and move where constructor is called Signed-off-by: Yee Hing Tong <[email protected]> * remove unneeded import Signed-off-by: Yee Hing Tong <[email protected]> * add a part of a test Signed-off-by: Yee Hing Tong <[email protected]> * Added tests Signed-off-by: Kevin Su <[email protected]> * Fixed lint Signed-off-by: Kevin Su <[email protected]> * typo Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]> Co-authored-by: Kevin Su <[email protected]> Signed-off-by: Eduardo Apolinario <[email protected]>
eapolinario
added a commit
that referenced
this pull request
Mar 21, 2022
* Add support union type Signed-off-by: Kevin Su <[email protected]> Signed-off-by: maximsmol <[email protected]> * Fixed test Signed-off-by: Kevin Su <[email protected]> Signed-off-by: maximsmol <[email protected]> * Fixed test Signed-off-by: Kevin Su <[email protected]> Signed-off-by: maximsmol <[email protected]> * Fixed lint Signed-off-by: Kevin Su <[email protected]> Signed-off-by: maximsmol <[email protected]> * Fixed tests Signed-off-by: Kevin Su <[email protected]> Signed-off-by: maximsmol <[email protected]> * Updated tests Signed-off-by: Kevin Su <[email protected]> Signed-off-by: maximsmol <[email protected]> * Update Union to use tagged Unions Signed-off-by: maximsmol <[email protected]> * Update to use string tags (part 1) Signed-off-by: maximsmol <[email protected]> * Working implementation, update tests Signed-off-by: maximsmol <[email protected]> * Fixes, more tests Signed-off-by: maximsmol <[email protected]> * Fix incorrect unwrapped literal-union matching, update test Signed-off-by: maximsmol <[email protected]> * Fix duplicate tag handling, add tests for collections containing unions Signed-off-by: maximsmol <[email protected]> * Fix type hint test Signed-off-by: maximsmol <[email protected]> * Add implicit wrapping union type tests Signed-off-by: maximsmol <[email protected]> * Add union ambiguity tests Signed-off-by: maximsmol <[email protected]> * Fixup tests, make TypeTransformerFailed compatible with all old exception types Signed-off-by: maximsmol <[email protected]> * Fixup models + add tests Signed-off-by: maximsmol <[email protected]> * Implement changed design Signed-off-by: maximsmol <[email protected]> * Fix PR comments Signed-off-by: maximsmol <[email protected]> * Remote entrypoint serialize (#733) Signed-off-by: Emirhan Karagül <[email protected]> Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: maximsmol <[email protected]> * Fix lint error in remote.py (#755) Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Signed-off-by: maximsmol <[email protected]> * Support enum in dataclass (#753) * Add support enum in dataclass Signed-off-by: Kevin Su <[email protected]> * Update test Signed-off-by: Kevin Su <[email protected]> * Fixed lint Signed-off-by: Kevin Su <[email protected]> * Fix subworkflow and launch plan FlyteRemote behavior (#751) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: maximsmol <[email protected]> * Extras: Shell task (#747) Signed-off-by: maximsmol <[email protected]> * Add support FlyteSchema in dataclass (#722) * schema in dataclass Signed-off-by: Kevin Su <[email protected]> * Added tests Signed-off-by: Kevin Su <[email protected]> * Fixed lint Signed-off-by: Kevin Su <[email protected]> * Updated tests Signed-off-by: Kevin Su <[email protected]> * Updated tests Signed-off-by: Kevin Su <[email protected]> * Fixed lint Signed-off-by: Kevin Su <[email protected]> * updated Signed-off-by: Kevin Su <[email protected]> * updated Signed-off-by: Kevin Su <[email protected]> * updated Signed-off-by: Kevin Su <[email protected]> * Remove workflow_execution.py (#758) Signed-off-by: Kevin Su <[email protected]> Signed-off-by: maximsmol <[email protected]> * Get raw input/output from remote execution (#675) * [wip] for feast demo Signed-off-by: Ketan Umare <[email protected]> * clean up a bit Signed-off-by: Yee Hing Tong <[email protected]> * add a test and move where constructor is called Signed-off-by: Yee Hing Tong <[email protected]> * remove unneeded import Signed-off-by: Yee Hing Tong <[email protected]> * add a part of a test Signed-off-by: Yee Hing Tong <[email protected]> * Added tests Signed-off-by: Kevin Su <[email protected]> * Fixed lint Signed-off-by: Kevin Su <[email protected]> * typo Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]> Co-authored-by: Kevin Su <[email protected]> Signed-off-by: maximsmol <[email protected]> * Fix mypy errors in flytekit/types (#757) Signed-off-by: Lisa <[email protected]> Signed-off-by: Kevin Su <[email protected]> * Remote client failed to fetch FlytePickle object (#764) * Fetch pickle value from flytekit remote Signed-off-by: Kevin Su <[email protected]> * Fix tests Signed-off-by: Kevin Su <[email protected]> * Remove default value Signed-off-by: Kevin Su <[email protected]> * Add support FlyteFile in dataclass (#725) * Add support Flyte File and directory in dataclass Signed-off-by: Kevin Su <[email protected]> * Fixed tests Signed-off-by: Kevin Su <[email protected]> * Fixed tests Signed-off-by: Kevin Su <[email protected]> * Updated Signed-off-by: Kevin Su <[email protected]> * Updated Signed-off-by: Kevin Su <[email protected]> * Fixed tests Signed-off-by: Kevin Su <[email protected]> * add task_resolver arg to @task decorator (#765) Signed-off-by: Niels Bantilan <[email protected]> Signed-off-by: maximsmol <[email protected]> * Copy metadata into map task from underlying (#766) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: maximsmol <[email protected]> * Support for delayed annotations (#760) Signed-off-by: Stefan Nelson-Lindall <[email protected]> Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: maximsmol <[email protected]> * Complex dataclass unit tests (#773) Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: maximsmol <[email protected]> * remote_source lost on serialization of @dataclass_json with FlyteFile (#774) * remote_source lost on serialization of @dataclass_json with FlyteFile Signed-off-by: Kevin Su <[email protected]> * updated tests Signed-off-by: Kevin Su <[email protected]> * updated tests Signed-off-by: Kevin Su <[email protected]> Signed-off-by: maximsmol <[email protected]> * Single-task execution FlyteRemote sync (#778) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: maximsmol <[email protected]> * Logging updates (#775) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: maximsmol <[email protected]> * fix: doc-requirements.txt to reduce vulnerabilities (#779) The following vulnerabilities are fixed by pinning transitive dependencies: - https://snyk.io/vuln/SNYK-PYTHON-LXML-2316995 Signed-off-by: maximsmol <[email protected]> * Add cache_serialize parameter to tasks (#673) * added cache_serialize parameter for basic tasks Signed-off-by: Daniel Rammer <[email protected]> * fixed typo Signed-off-by: Daniel Rammer <[email protected]> * updated flyteidl version Signed-off-by: Daniel Rammer <[email protected]> * changed flyteidl requirements everywhere Signed-off-by: Daniel Rammer <[email protected]> * remove flyteidl version requires in setup.py so we can use develop Signed-off-by: Daniel Rammer <[email protected]> * removed flyteidl git repos from a variety of requirements packages Signed-off-by: Daniel Rammer <[email protected]> * updated variable discovery_serializable to cache_serializable Signed-off-by: Daniel Rammer <[email protected]> * updated requirements Signed-off-by: Daniel Rammer <[email protected]> * fixed TaskMetadata _cache_serializable variable name Signed-off-by: Daniel Rammer <[email protected]> * propgating cache_serialize parameter through to tasks Signed-off-by: Daniel Rammer <[email protected]> * added cache_serializable to SdkRawContainerTask Signed-off-by: Daniel Rammer <[email protected]> * fixing cache_serializable variable propogation issues Signed-off-by: Daniel Rammer <[email protected]> * added documentation Signed-off-by: Daniel Rammer <[email protected]> * added unit tests for cache_serialize metadata Signed-off-by: Daniel Rammer <[email protected]> * linter added spaces in unit tests Signed-off-by: Daniel Rammer <[email protected]> Signed-off-by: maximsmol <[email protected]> * When using the `task` and `workflow` decorator, correctly wrap the fu… (#780) * When using the `task` and `workflow` decorator, correctly wrap the function This enables tooling such as docstring search tools to unwrap the object and show the correct docstring. Signed-off-by: Bernhard Stadlbauer <[email protected]> * Remove blackshark copyright header Signed-off-by: Bernhard Stadlbauer <[email protected]> * Fix broken great expectations test Signed-off-by: Bernhard Stadlbauer <[email protected]> * Add test for stacked decorators Signed-off-by: Bernhard Stadlbauer <[email protected]> Co-authored-by: Bernhard Stadlbauer <[email protected]> Co-authored-by: Bernhard Stadlbauer <[email protected]> Signed-off-by: maximsmol <[email protected]> * Add option to flyte-cli for specifying root certificate (#783) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: maximsmol <[email protected]> * Add validation check to cacert switch (#787) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: maximsmol <[email protected]> * Remove pytz constraint (#786) * Remove pytz constraint from setup.py Signed-off-by: Eduardo Apolinario <[email protected]> * Regenerate requirements files Signed-off-by: Eduardo Apolinario <[email protected]> * Put pytz back Signed-off-by: Eduardo Apolinario <[email protected]> * make requirements.txt Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Signed-off-by: maximsmol <[email protected]> * Lint: remove f-string misuse (#788) Signed-off-by: Eduardo Apolinario <[email protected]> Signed-off-by: maximsmol <[email protected]> * Pyarrow greater than 4.0.0 (#790) * pyarrow>=4.0.0 Signed-off-by: Eduardo Apolinario <[email protected]> * Regenerate requirements. Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Signed-off-by: maximsmol <[email protected]> * Support python 3.10 (#791) * [wip] Support python 3.10 Signed-off-by: Eduardo Apolinario <[email protected]> * Add Dockerfile.py310 Signed-off-by: Eduardo Apolinario <[email protected]> * Stringify python version Signed-off-by: Eduardo Apolinario <[email protected]> * Skip flytekit-modin plugin tests on 3.10 Signed-off-by: Eduardo Apolinario <[email protected]> * Add 3.9 and 3.10 to list of supported version in plugins Signed-off-by: Eduardo Apolinario <[email protected]> * Comment why flytekit-modin is not running on 3.10 and disable fail-fast in plugin tests Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> * add `with_overrides` to map task (#794) * add with_overrides Signed-off-by: Samhita Alla <[email protected]> * remove Resources Signed-off-by: Samhita Alla <[email protected]> Signed-off-by: maximsmol <[email protected]> * bump docsearch version (#805) Signed-off-by: Samhita Alla <[email protected]> Signed-off-by: maximsmol <[email protected]> * update docs for new navbar theme (#806) Signed-off-by: maximsmol <[email protected]> * fix requirment.txt github issue (#810) Signed-off-by: Yuvraj <[email protected]> Signed-off-by: maximsmol <[email protected]> * Add sphinx panels (#815) Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: maximsmol <[email protected]> * Schema overhaul (#785) Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Yee Hing Tong <[email protected]> * Parent workflow serialization fails when calling a launch plan with fixed inputs (#814) Signed-off-by: Kevin Su <[email protected]> Signed-off-by: maximsmol <[email protected]> * Fix sagemaker plugin (#817) Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Signed-off-by: maximsmol <[email protected]> * Remove legacy API (#807) Signed-off-by: Yee Hing Tong <[email protected]> * Add Bigquery plugin (#789) * Add bigquery plugin Signed-off-by: Kevin Su <[email protected]> * Update dependency Signed-off-by: Kevin Su <[email protected]> * update get_custom Signed-off-by: Kevin Su <[email protected]> * Add structured dataset Signed-off-by: Kevin Su <[email protected]> * Add structured dataset Signed-off-by: Kevin Su <[email protected]> * Updated comment Signed-off-by: Kevin Su <[email protected]> * Add BQ in GA Signed-off-by: Kevin Su <[email protected]> * alphabetical order Signed-off-by: Kevin Su <[email protected]> Signed-off-by: maximsmol <[email protected]> * Fixed flytekit-papermill ImportError (#818) Signed-off-by: Kevin Su <[email protected]> * Add support for string-format-like sytax for shell task (#792) * POC: Add support for f-string like sytax for shell task This commit is a proof of concept adding f-string like syntax for shell_tasks. This supports using nested types for script inputs, such as data classes. This change was motivated by the desire to combine shell_tasks that have multiple inputs with map_tasks which only support tasks with a single input. This commit is only a starting point, since it makes some changes to the shell_task API (adds a template_style field), and modifies some of the default behavior for ease of implementation (e.g. throwing an error when there are unused input arguments). Signed-off-by: Zach Palchick <[email protected]> * Drop support for old/regex style for doing string interpolation Signed-off-by: Zach Palchick <[email protected]> Signed-off-by: maximsmol <[email protected]> * Expose configured RawOutputPrefix during execution (#813) * Expose configured RawOutputPrefix during execution Signed-off-by: Kevin Su <[email protected]> * Remove sdk_runnable.py and spark_task.py Signed-off-by: Kevin Su <[email protected]> Signed-off-by: maximsmol <[email protected]> * Add SecretsManager back to old import location (#820) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: maximsmol <[email protected]> * Add some tests (#819) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: maximsmol <[email protected]> * Fixed flaky spark test (#821) Signed-off-by: Kevin Su <[email protected]> Signed-off-by: maximsmol <[email protected]> * fix: plugins/flytekit-greatexpectations/requirements.txt to reduce vulnerabilities (#823) The following vulnerabilities are fixed by pinning transitive dependencies: - https://snyk.io/vuln/SNYK-PYTHON-IPYTHON-2348630 Signed-off-by: maximsmol <[email protected]> * fix: plugins/flytekit-papermill/requirements.txt to reduce vulnerabilities (#825) The following vulnerabilities are fixed by pinning transitive dependencies: - https://snyk.io/vuln/SNYK-PYTHON-IPYTHON-2348630 Signed-off-by: maximsmol <[email protected]> * fix: requirements-spark2.txt to reduce vulnerabilities (#826) The following vulnerabilities are fixed by pinning transitive dependencies: - https://snyk.io/vuln/SNYK-PYTHON-IPYTHON-2348630 Signed-off-by: maximsmol <[email protected]> * fix: requirements.txt to reduce vulnerabilities (#824) The following vulnerabilities are fixed by pinning transitive dependencies: - https://snyk.io/vuln/SNYK-PYTHON-IPYTHON-2348630 Signed-off-by: maximsmol <[email protected]> * Intratask checkpointing (#771) * wip - intratask checkpointing Signed-off-by: Ketan Umare <[email protected]> * sync checkpointer with tests Signed-off-by: Ketan Umare <[email protected]> * Checkpinter in entrypoint Signed-off-by: Ketan Umare <[email protected]> * checkpoint in progress Signed-off-by: Ketan Umare <[email protected]> * wip Signed-off-by: Ketan Umare <[email protected]> * Intratask checkpointer Signed-off-by: Ketan Umare <[email protected]> * Checkpoint updated Signed-off-by: Ketan Umare <[email protected]> * Intra-task checkpointing Signed-off-by: Ketan Umare <[email protected]> * Test and entrypoint updated Signed-off-by: Ketan Umare <[email protected]> * lint fixed Signed-off-by: Ketan Umare <[email protected]> * test fixes Signed-off-by: Ketan Umare <[email protected]> * fmt Signed-off-by: Ketan Umare <[email protected]> * updated entrypoint Signed-off-by: Ketan Umare <[email protected]> * updated Signed-off-by: Ketan Umare <[email protected]> * update Signed-off-by: Ketan Umare <[email protected]> * print Signed-off-by: Ketan Umare <[email protected]> * updated Signed-off-by: Ketan Umare <[email protected]> * SyncCheckpointer working Signed-off-by: Ketan Umare <[email protected]> * updated Signed-off-by: Ketan Umare <[email protected]> * update Signed-off-by: Ketan Umare <[email protected]> * fixed import problems Signed-off-by: Ketan Umare <[email protected]> * fixed test Signed-off-by: Ketan Umare <[email protected]> * fixed imports Signed-off-by: Ketan Umare <[email protected]> * fixed lints and errors Signed-off-by: Ketan Umare <[email protected]> * lint fix Signed-off-by: Ketan Umare <[email protected]> * addressed comments Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: maximsmol <[email protected]> * Support reading subset column (#822) * Support StructuredDatasetDecoder read subset column Signed-off-by: Kevin Su <[email protected]> * Added tests Signed-off-by: Kevin Su <[email protected]> * Fixed tests Signed-off-by: Kevin Su <[email protected]> * Fixed typo Signed-off-by: Kevin Su <[email protected]> * Updated tests Signed-off-by: Kevin Su <[email protected]> * [pr into #822] (#827) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Kevin Su <[email protected]> * [pr into #822] Final update to structured dataset column subsetting (#828) Signed-off-by: Yee Hing Tong <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]> * Fix spark regression (#830) Signed-off-by: Kevin Su <[email protected]> Signed-off-by: maximsmol <[email protected]> * Update argument setting for in fast registered, dynamically generated, pod tasks (#835) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: maximsmol <[email protected]> * `ctx` Context can be used within shell tasks - to access context vars and secrets (#832) * Adding context to a substitutable parameter in shell task Signed-off-by: Ketan Umare <[email protected]> * Support for secrets in context Signed-off-by: Ketan Umare <[email protected]> * addressed comments Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: maximsmol <[email protected]> * Expose Checkpoint as a top-level interface in flytekit (#839) Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: maximsmol <[email protected]> * Parse duration field from flyteidl to `flytekit.models.execution.ExecutionClosure` (#829) * Parse duration field from flyteidl to `flytekit.models.execution.ExecutionClosure` Signed-off-by: Bernhard Stadlbauer <[email protected]> * Add test for execution closure Signed-off-by: Bernhard Stadlbauer <[email protected]> * Add tests to Flyte remote Signed-off-by: Bernhard Stadlbauer <[email protected]> * Split execution test into with output and with error Signed-off-by: Bernhard Stadlbauer <[email protected]> Co-authored-by: Bernhard Stadlbauer <[email protected]> Signed-off-by: maximsmol <[email protected]> * Gate new Structured Dataset feature & remove old config objects (#831) Signed-off-by: Yee Hing Tong <[email protected]> * Fixing out of order for conditional outputs (#843) Signed-off-by: maximsmol <[email protected]> * Set default values to map task template (#841) * Set sane defaults in map task templates Signed-off-by: Eduardo Apolinario <[email protected]> * Remove unused method Signed-off-by: Eduardo Apolinario <[email protected]> * Put ArrayJob.from_dict back Signed-off-by: Eduardo Apolinario <[email protected]> * Define parallelism=0 as unbounded Signed-off-by: Eduardo Apolinario <[email protected]> * Remove special case to handle 0 Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Signed-off-by: maximsmol <[email protected]> * Regenerate requirements files - dependencies dropping support for python 3.7 (#838) * Regenerate requirements files Signed-off-by: Eduardo Apolinario <[email protected]> * Put restrictions on numpy and pandas versions Signed-off-by: Eduardo Apolinario <[email protected]> * Use --use-deprecated=legacy-resolver Signed-off-by: Eduardo Apolinario <[email protected]> * use pip==22.0.3 everywhere Signed-off-by: Eduardo Apolinario <[email protected]> * Remove --use-deprecated=legacy-resolver Signed-off-by: Eduardo Apolinario <[email protected]> * Relax click Signed-off-by: Eduardo Apolinario <[email protected]> * Regenerated plugins requirements Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> * TypeAnnotation (#759) * feat: support for annotated simple + list Signed-off-by: Kenny Workman <[email protected]> * feat: addition of annotation att to Signed-off-by: Kenny Workman <[email protected]> * feat: core obj Signed-off-by: Kenny Workman <[email protected]> * feat: proto model Signed-off-by: Kenny Workman <[email protected]> * feat: testing suite Signed-off-by: Kenny Workman <[email protected]> * fix: more stable typing introspection Signed-off-by: Kenny Workman <[email protected]> * fix: strip legacy Signed-off-by: Kenny Workman <[email protected]> * fix: explicitly allow only one annotation Signed-off-by: Kenny Workman <[email protected]> * feat: direct type transformer tests Signed-off-by: Kenny Workman <[email protected]> * fix: there and back test Signed-off-by: Kenny Workman <[email protected]> * fix: typing_extensions for get_origin Signed-off-by: Kenny Workman <[email protected]> * fix: more semantic list generic unwrap Signed-off-by: Kenny Workman <[email protected]> * fix: tmp requirements file with custom idl Signed-off-by: Kenny Workman <[email protected]> * fix: nits Signed-off-by: Kenny Workman <[email protected]> * feat: semantic error for unsupported complex literals Signed-off-by: Kenny Workman <[email protected]> * fix: but Signed-off-by: Kenny Workman <[email protected]> * feat: more tests ;) Signed-off-by: Kenny Workman <[email protected]> * fix: imports Signed-off-by: Kenny Workman <[email protected]> * fix: complex annotations Signed-off-by: Kenny Workman <[email protected]> * fix: temp requirements files for unit tests Signed-off-by: Kenny Workman <[email protected]> * fix: lint bug Signed-off-by: Kenny Workman <[email protected]> * fix: tmp setup.py Signed-off-by: Kenny Workman <[email protected]> * fix: use typing_extensions Signed-off-by: Kenny Workman <[email protected]> * fix: typing_extensions for annotated Signed-off-by: Kenny Workman <[email protected]> * fix: typing_ext Signed-off-by: Kenny Workman <[email protected]> * fix: plugin tmp requirements Signed-off-by: Kenny Workman <[email protected]> * fix: bump requirements Signed-off-by: Kenny Workman <[email protected]> * fix: doc requirements Signed-off-by: Kenny Workman <[email protected]> * fix: whitespace Signed-off-by: Kenny Workman <[email protected]> * fix: bump flytekit Signed-off-by: Kenny Workman <[email protected]> * fix: numpy version Signed-off-by: Kenny Workman <[email protected]> * fix: lint Signed-off-by: Kenny Workman <[email protected]> * fix: pandas version Signed-off-by: Kenny Workman <[email protected]> * fix: bump requirements Signed-off-by: Kenny Workman <[email protected]> * fix: test import Signed-off-by: Kenny Workman <[email protected]> * fix: flake8 lint Signed-off-by: Kenny Workman <[email protected]> * fix: merge Signed-off-by: Kenny Workman <[email protected]> * fix: requirements Signed-off-by: Kenny Workman <[email protected]> * fix: requirements Signed-off-by: Kenny Workman <[email protected]> * fix: lint Signed-off-by: Kenny Workman <[email protected]> * fix: papermill req Signed-off-by: Kenny Workman <[email protected]> * fix: req Signed-off-by: Kenny Workman <[email protected]> * Remove singleton from structured dataset transformer engine (#848) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Kevin Su <[email protected]> * Assign input and output to FlyteWorkflowExecution (#842) Signed-off-by: Kevin Su <[email protected]> Signed-off-by: maximsmol <[email protected]> * Add reference entities to FlyteTask and FlyteLaunchPlan (#850) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: maximsmol <[email protected]> * Fix fast registration error (#851) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Samhita Alla <[email protected]> Signed-off-by: maximsmol <[email protected]> * Add support for local execute in pod task (#852) Signed-off-by: ggydush-fn <[email protected]> Signed-off-by: maximsmol <[email protected]> * Add anonymous retry (#854) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: maximsmol <[email protected]> * fix: formatting, linting, typing_extensions * fix: do not use SDK types Signed-off-by: maximsmol <[email protected]> * fix: update test comment Signed-off-by: maximsmol <[email protected]> * fix: also check literal type castability when tags match Signed-off-by: maximsmol <[email protected]> * Point flyteidl to maxim's fork in CI and requirements files Signed-off-by: Eduardo Apolinario <[email protected]> * [Core feature] Add Raw AWS Batch Task (#782) * Init plugin Signed-off-by: Kevin Su <[email protected]> * Fixed lint Signed-off-by: Kevin Su <[email protected]> * address comment Signed-off-by: Kevin Su <[email protected]> * Fixed typo Signed-off-by: Kevin Su <[email protected]> * Updated AWS config * Fixed lint Signed-off-by: Kevin Su <[email protected]> * Added comment Signed-off-by: Kevin Su <[email protected]> * Update config Signed-off-by: Kevin Su <[email protected]> * Fixed tests Signed-off-by: Kevin Su <[email protected]> * Fixed tests Signed-off-by: Kevin Su <[email protected]> * Fixed tests Signed-off-by: Kevin Su <[email protected]> * use pyflyte execute Signed-off-by: Kevin Su <[email protected]> * Fixed tests Signed-off-by: Kevin Su <[email protected]> * Fixed tests Signed-off-by: Kevin Su <[email protected]> * Added comment Signed-off-by: Kevin Su <[email protected]> Signed-off-by: maximsmol <[email protected]> * Add structured dataset encoder/decoder in fsspec plugin (#849) Signed-off-by: Kevin Su <[email protected]> Signed-off-by: maximsmol <[email protected]> * Delete unnecessary auth configuration (#858) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: maximsmol <[email protected]> * Fixed format alias in Flytekit docs (#844) * Fixed format alias Fixed docs for file format alias that weren't rendered properly. A warning popped up stating 'ignore' is deprecated, and to use 'ignore_paths' instead. Signed-off-by: SmritiSatyanV <[email protected]> * Bump idl (#862) Signed-off-by: Yee Hing Tong <[email protected]> * Updated authoring.rst (#863) * Updated authoring.rst Added directive Rephrased sentence * Fixed build error Signed-off-by: SmritiSatyanV <[email protected]> * test-build-1 Signed-off-by: SmritiSatyanV <[email protected]> Signed-off-by: maximsmol <[email protected]> * Updated authoring.rst (#866) Added the directive correctly Signed-off-by: SmritiSatyanV <[email protected]> Signed-off-by: maximsmol <[email protected]> * Change docs for HTMLPage type to say HTMLPage instead of PNGImage (#868) Signed-off-by: maximsmol <[email protected]> * Revisit StructuredDatasetDecoder interface (#865) Signed-off-by: Kevin Su <[email protected]> Signed-off-by: maximsmol <[email protected]> * Remove legacy mentions in contributing guide (#870) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: maximsmol <[email protected]> * Add GCS protocol in the structured dataset (#869) Signed-off-by: Kevin Su <[email protected]> Signed-off-by: maximsmol <[email protected]> * Make fetched entities callable within workflows (#867) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: maximsmol <[email protected]> * Caching of offloaded objects (#762) * Remove flyteidl from install_requires Signed-off-by: Eduardo Apolinario <[email protected]> * Expose hash in Literal Signed-off-by: Eduardo Apolinario <[email protected]> * Set hash in TypeEngine Signed-off-by: Eduardo Apolinario <[email protected]> * Modify cache key calculation to take hash into account Signed-off-by: Eduardo Apolinario <[email protected]> * Opt-in PandasDataFrameTransformer Signed-off-by: Eduardo Apolinario <[email protected]> * Add unit tests Signed-off-by: Eduardo Apolinario <[email protected]> * Iterate using a flyteidl branch Signed-off-by: Eduardo Apolinario <[email protected]> * Regenerate requirements files Signed-off-by: Eduardo Apolinario <[email protected]> * Regenerate requirements files Signed-off-by: Eduardo Apolinario <[email protected]> * Move _hash_overridable to StructureDatasetTransformerEngine Signed-off-by: Eduardo Apolinario <[email protected]> * Move HashMethod to flytekit.core.hash Signed-off-by: Eduardo Apolinario <[email protected]> * Fix `unit_test` make target Signed-off-by: Eduardo Apolinario <[email protected]> * Split `unit_test` make target in two lines Signed-off-by: Eduardo Apolinario <[email protected]> * Add assert to structured dataset compatibility test Signed-off-by: Eduardo Apolinario <[email protected]> * Remove TODO Signed-off-by: Eduardo Apolinario <[email protected]> * Regenerate plugins requirements files pointing to the right version of flyteidl. Signed-off-by: Eduardo Apolinario <[email protected]> * Set hash as a property of the literal Signed-off-by: Eduardo Apolinario <[email protected]> * Install plugins requirements in CI. Signed-off-by: Eduardo Apolinario <[email protected]> * Add hash.setter Signed-off-by: Eduardo Apolinario <[email protected]> * Install flyteidl directly Signed-off-by: Eduardo Apolinario <[email protected]> * Revert "Regenerate plugins requirements files pointing to the right version of flyteidl." This reverts commit c2dbb54. Signed-off-by: Eduardo Apolinario <[email protected]> * wip - Add support for univariate lists Signed-off-by: Eduardo Apolinario <[email protected]> * Add support for lists of annotated objects Signed-off-by: Eduardo Apolinario <[email protected]> * Revamp generation of cache key (to cover case of literals collections and maps) Signed-off-by: Eduardo Apolinario <[email protected]> * Leave TODO for warning Signed-off-by: Eduardo Apolinario <[email protected]> * Revert "Add support for lists of annotated objects" This reverts commit 4b5f608. Signed-off-by: Eduardo Apolinario <[email protected]> * Revert "wip - Add support for univariate lists" This reverts commit adaa448. Signed-off-by: Eduardo Apolinario <[email protected]> * Remove docstring Signed-off-by: Eduardo Apolinario <[email protected]> * Add flyteidl>=0.23.0 Signed-off-by: Eduardo Apolinario <[email protected]> * Remove mentions to branch flyteidl@add-hash-to-literal Signed-off-by: Eduardo Apolinario <[email protected]> * Bump flyteidl in plugins requirements Signed-off-by: Eduardo Apolinario <[email protected]> * Regenerate plugins requirements again Signed-off-by: Eduardo Apolinario <[email protected]> * Restore papermill/requirements.txt Signed-off-by: Eduardo Apolinario <[email protected]> * Point flytekitplugins-spark to the offloaded-objects-caching branch in papermill tests Signed-off-by: Eduardo Apolinario <[email protected]> * Set flyteidl>=0.23.0 in papermill dev-requirements Co-authored-by: Eduardo Apolinario <[email protected]> * FlyteRemote fetch of conditional nodes (#772) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: maximsmol <[email protected]> * Removed root logger (#871) * Removed root logger Signed-off-by: Kevin Su <[email protected]> * Updated logger Signed-off-by: Kevin Su <[email protected]> * Fixed lint Signed-off-by: Kevin Su <[email protected]> * Updated log level Signed-off-by: Kevin Su <[email protected]> * Updated logger Signed-off-by: Kevin Su <[email protected]> * Updated logger Signed-off-by: Kevin Su <[email protected]> * Updated logger Signed-off-by: Kevin Su <[email protected]> * lint fixed Signed-off-by: Kevin Su <[email protected]> Signed-off-by: maximsmol <[email protected]> * Fix flytekit_compatibility/test_schema_types.py test Signed-off-by: Eduardo Apolinario <[email protected]> Signed-off-by: maximsmol <[email protected]> * make lint Signed-off-by: Eduardo Apolinario <[email protected]> * fix: annotated type conversion error Signed-off-by: maximsmol <[email protected]> * fix: _are_types_castable based on tests Signed-off-by: maximsmol <[email protected]> * fix: test failing if using random order Signed-off-by: maximsmol <[email protected]> * Merge branch 'master' into maximsmol/union_type Signed-off-by: maximsmol <[email protected]> * fix: merge issue Signed-off-by: maximsmol <[email protected]> * fix: requirements Signed-off-by: maximsmol <[email protected]> * fix: schema transformer error Signed-off-by: maximsmol <[email protected]> * fix: test Signed-off-by: maximsmol <[email protected]> * fix: merge issue Signed-off-by: maximsmol <[email protected]> * fix: union + annotated behavior Signed-off-by: maximsmol <[email protected]> * Regenerate requirements Signed-off-by: Eduardo Apolinario <[email protected]> * Bump requirements in plugins Signed-off-by: Eduardo Apolinario <[email protected]> * Handle nested Annotated Signed-off-by: Eduardo Apolinario <[email protected]> * Leave TODO re: strucutured dataset type castability Signed-off-by: Eduardo Apolinario <[email protected]> * Remove mention to flyteidl@union_type in doc-requirements Signed-off-by: Eduardo Apolinario <[email protected]> * Linting Signed-off-by: Eduardo Apolinario <[email protected]> * Use tempfile.mkdtemp to create a temporary directory for local data persistence. Signed-off-by: Eduardo Apolinario <[email protected]> * Revert "Use tempfile.mkdtemp to create a temporary directory for local data persistence." This reverts commit 861b157. Signed-off-by: eduardo apolinario <[email protected]> * Force temporary file to not be deleted in test Signed-off-by: eduardo apolinario <[email protected]> * Regenerate papermill dev-requirements Signed-off-by: eduardo apolinario <[email protected]> * Remove duplicate code Signed-off-by: Eduardo Apolinario <[email protected]> * Put a lower bound on the pip version installed in CI Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Kevin Su <[email protected]> Co-authored-by: Emirhan Karagül <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Co-authored-by: Kevin Su <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]> Co-authored-by: Ketan Umare <[email protected]> Co-authored-by: Lisa <[email protected]> Co-authored-by: Niels Bantilan <[email protected]> Co-authored-by: Stef Nelson-Lindall <[email protected]> Co-authored-by: Snyk bot <[email protected]> Co-authored-by: Dan Rammer <[email protected]> Co-authored-by: bstadlbauer <[email protected]> Co-authored-by: Bernhard Stadlbauer <[email protected]> Co-authored-by: Bernhard Stadlbauer <[email protected]> Co-authored-by: Samhita Alla <[email protected]> Co-authored-by: Yuvraj <[email protected]> Co-authored-by: Zach Palchick <[email protected]> Co-authored-by: Snyk bot <[email protected]> Co-authored-by: Kenny Workman <[email protected]> Co-authored-by: ggydush-fn <[email protected]> Co-authored-by: SmritiSatyanV <[email protected]> Co-authored-by: Matthew Griffin <[email protected]> Co-authored-by: eduardo apolinario <[email protected]>
myz540
pushed a commit
to ProjectAussie/flytekit
that referenced
this pull request
Apr 11, 2022
* Add support union type Signed-off-by: Kevin Su <[email protected]> Signed-off-by: maximsmol <[email protected]> * Fixed test Signed-off-by: Kevin Su <[email protected]> Signed-off-by: maximsmol <[email protected]> * Fixed test Signed-off-by: Kevin Su <[email protected]> Signed-off-by: maximsmol <[email protected]> * Fixed lint Signed-off-by: Kevin Su <[email protected]> Signed-off-by: maximsmol <[email protected]> * Fixed tests Signed-off-by: Kevin Su <[email protected]> Signed-off-by: maximsmol <[email protected]> * Updated tests Signed-off-by: Kevin Su <[email protected]> Signed-off-by: maximsmol <[email protected]> * Update Union to use tagged Unions Signed-off-by: maximsmol <[email protected]> * Update to use string tags (part 1) Signed-off-by: maximsmol <[email protected]> * Working implementation, update tests Signed-off-by: maximsmol <[email protected]> * Fixes, more tests Signed-off-by: maximsmol <[email protected]> * Fix incorrect unwrapped literal-union matching, update test Signed-off-by: maximsmol <[email protected]> * Fix duplicate tag handling, add tests for collections containing unions Signed-off-by: maximsmol <[email protected]> * Fix type hint test Signed-off-by: maximsmol <[email protected]> * Add implicit wrapping union type tests Signed-off-by: maximsmol <[email protected]> * Add union ambiguity tests Signed-off-by: maximsmol <[email protected]> * Fixup tests, make TypeTransformerFailed compatible with all old exception types Signed-off-by: maximsmol <[email protected]> * Fixup models + add tests Signed-off-by: maximsmol <[email protected]> * Implement changed design Signed-off-by: maximsmol <[email protected]> * Fix PR comments Signed-off-by: maximsmol <[email protected]> * Remote entrypoint serialize (flyteorg#733) Signed-off-by: Emirhan Karagül <[email protected]> Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: maximsmol <[email protected]> * Fix lint error in remote.py (flyteorg#755) Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Signed-off-by: maximsmol <[email protected]> * Support enum in dataclass (flyteorg#753) * Add support enum in dataclass Signed-off-by: Kevin Su <[email protected]> * Update test Signed-off-by: Kevin Su <[email protected]> * Fixed lint Signed-off-by: Kevin Su <[email protected]> * Fix subworkflow and launch plan FlyteRemote behavior (flyteorg#751) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: maximsmol <[email protected]> * Extras: Shell task (flyteorg#747) Signed-off-by: maximsmol <[email protected]> * Add support FlyteSchema in dataclass (flyteorg#722) * schema in dataclass Signed-off-by: Kevin Su <[email protected]> * Added tests Signed-off-by: Kevin Su <[email protected]> * Fixed lint Signed-off-by: Kevin Su <[email protected]> * Updated tests Signed-off-by: Kevin Su <[email protected]> * Updated tests Signed-off-by: Kevin Su <[email protected]> * Fixed lint Signed-off-by: Kevin Su <[email protected]> * updated Signed-off-by: Kevin Su <[email protected]> * updated Signed-off-by: Kevin Su <[email protected]> * updated Signed-off-by: Kevin Su <[email protected]> * Remove workflow_execution.py (flyteorg#758) Signed-off-by: Kevin Su <[email protected]> Signed-off-by: maximsmol <[email protected]> * Get raw input/output from remote execution (flyteorg#675) * [wip] for feast demo Signed-off-by: Ketan Umare <[email protected]> * clean up a bit Signed-off-by: Yee Hing Tong <[email protected]> * add a test and move where constructor is called Signed-off-by: Yee Hing Tong <[email protected]> * remove unneeded import Signed-off-by: Yee Hing Tong <[email protected]> * add a part of a test Signed-off-by: Yee Hing Tong <[email protected]> * Added tests Signed-off-by: Kevin Su <[email protected]> * Fixed lint Signed-off-by: Kevin Su <[email protected]> * typo Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]> Co-authored-by: Kevin Su <[email protected]> Signed-off-by: maximsmol <[email protected]> * Fix mypy errors in flytekit/types (flyteorg#757) Signed-off-by: Lisa <[email protected]> Signed-off-by: Kevin Su <[email protected]> * Remote client failed to fetch FlytePickle object (flyteorg#764) * Fetch pickle value from flytekit remote Signed-off-by: Kevin Su <[email protected]> * Fix tests Signed-off-by: Kevin Su <[email protected]> * Remove default value Signed-off-by: Kevin Su <[email protected]> * Add support FlyteFile in dataclass (flyteorg#725) * Add support Flyte File and directory in dataclass Signed-off-by: Kevin Su <[email protected]> * Fixed tests Signed-off-by: Kevin Su <[email protected]> * Fixed tests Signed-off-by: Kevin Su <[email protected]> * Updated Signed-off-by: Kevin Su <[email protected]> * Updated Signed-off-by: Kevin Su <[email protected]> * Fixed tests Signed-off-by: Kevin Su <[email protected]> * add task_resolver arg to @task decorator (flyteorg#765) Signed-off-by: Niels Bantilan <[email protected]> Signed-off-by: maximsmol <[email protected]> * Copy metadata into map task from underlying (flyteorg#766) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: maximsmol <[email protected]> * Support for delayed annotations (flyteorg#760) Signed-off-by: Stefan Nelson-Lindall <[email protected]> Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: maximsmol <[email protected]> * Complex dataclass unit tests (flyteorg#773) Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: maximsmol <[email protected]> * remote_source lost on serialization of @dataclass_json with FlyteFile (flyteorg#774) * remote_source lost on serialization of @dataclass_json with FlyteFile Signed-off-by: Kevin Su <[email protected]> * updated tests Signed-off-by: Kevin Su <[email protected]> * updated tests Signed-off-by: Kevin Su <[email protected]> Signed-off-by: maximsmol <[email protected]> * Single-task execution FlyteRemote sync (flyteorg#778) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: maximsmol <[email protected]> * Logging updates (flyteorg#775) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: maximsmol <[email protected]> * fix: doc-requirements.txt to reduce vulnerabilities (flyteorg#779) The following vulnerabilities are fixed by pinning transitive dependencies: - https://snyk.io/vuln/SNYK-PYTHON-LXML-2316995 Signed-off-by: maximsmol <[email protected]> * Add cache_serialize parameter to tasks (flyteorg#673) * added cache_serialize parameter for basic tasks Signed-off-by: Daniel Rammer <[email protected]> * fixed typo Signed-off-by: Daniel Rammer <[email protected]> * updated flyteidl version Signed-off-by: Daniel Rammer <[email protected]> * changed flyteidl requirements everywhere Signed-off-by: Daniel Rammer <[email protected]> * remove flyteidl version requires in setup.py so we can use develop Signed-off-by: Daniel Rammer <[email protected]> * removed flyteidl git repos from a variety of requirements packages Signed-off-by: Daniel Rammer <[email protected]> * updated variable discovery_serializable to cache_serializable Signed-off-by: Daniel Rammer <[email protected]> * updated requirements Signed-off-by: Daniel Rammer <[email protected]> * fixed TaskMetadata _cache_serializable variable name Signed-off-by: Daniel Rammer <[email protected]> * propgating cache_serialize parameter through to tasks Signed-off-by: Daniel Rammer <[email protected]> * added cache_serializable to SdkRawContainerTask Signed-off-by: Daniel Rammer <[email protected]> * fixing cache_serializable variable propogation issues Signed-off-by: Daniel Rammer <[email protected]> * added documentation Signed-off-by: Daniel Rammer <[email protected]> * added unit tests for cache_serialize metadata Signed-off-by: Daniel Rammer <[email protected]> * linter added spaces in unit tests Signed-off-by: Daniel Rammer <[email protected]> Signed-off-by: maximsmol <[email protected]> * When using the `task` and `workflow` decorator, correctly wrap the fu… (flyteorg#780) * When using the `task` and `workflow` decorator, correctly wrap the function This enables tooling such as docstring search tools to unwrap the object and show the correct docstring. Signed-off-by: Bernhard Stadlbauer <[email protected]> * Remove blackshark copyright header Signed-off-by: Bernhard Stadlbauer <[email protected]> * Fix broken great expectations test Signed-off-by: Bernhard Stadlbauer <[email protected]> * Add test for stacked decorators Signed-off-by: Bernhard Stadlbauer <[email protected]> Co-authored-by: Bernhard Stadlbauer <[email protected]> Co-authored-by: Bernhard Stadlbauer <[email protected]> Signed-off-by: maximsmol <[email protected]> * Add option to flyte-cli for specifying root certificate (flyteorg#783) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: maximsmol <[email protected]> * Add validation check to cacert switch (flyteorg#787) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: maximsmol <[email protected]> * Remove pytz constraint (flyteorg#786) * Remove pytz constraint from setup.py Signed-off-by: Eduardo Apolinario <[email protected]> * Regenerate requirements files Signed-off-by: Eduardo Apolinario <[email protected]> * Put pytz back Signed-off-by: Eduardo Apolinario <[email protected]> * make requirements.txt Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Signed-off-by: maximsmol <[email protected]> * Lint: remove f-string misuse (flyteorg#788) Signed-off-by: Eduardo Apolinario <[email protected]> Signed-off-by: maximsmol <[email protected]> * Pyarrow greater than 4.0.0 (flyteorg#790) * pyarrow>=4.0.0 Signed-off-by: Eduardo Apolinario <[email protected]> * Regenerate requirements. Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Signed-off-by: maximsmol <[email protected]> * Support python 3.10 (flyteorg#791) * [wip] Support python 3.10 Signed-off-by: Eduardo Apolinario <[email protected]> * Add Dockerfile.py310 Signed-off-by: Eduardo Apolinario <[email protected]> * Stringify python version Signed-off-by: Eduardo Apolinario <[email protected]> * Skip flytekit-modin plugin tests on 3.10 Signed-off-by: Eduardo Apolinario <[email protected]> * Add 3.9 and 3.10 to list of supported version in plugins Signed-off-by: Eduardo Apolinario <[email protected]> * Comment why flytekit-modin is not running on 3.10 and disable fail-fast in plugin tests Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> * add `with_overrides` to map task (flyteorg#794) * add with_overrides Signed-off-by: Samhita Alla <[email protected]> * remove Resources Signed-off-by: Samhita Alla <[email protected]> Signed-off-by: maximsmol <[email protected]> * bump docsearch version (flyteorg#805) Signed-off-by: Samhita Alla <[email protected]> Signed-off-by: maximsmol <[email protected]> * update docs for new navbar theme (flyteorg#806) Signed-off-by: maximsmol <[email protected]> * fix requirment.txt github issue (flyteorg#810) Signed-off-by: Yuvraj <[email protected]> Signed-off-by: maximsmol <[email protected]> * Add sphinx panels (flyteorg#815) Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: maximsmol <[email protected]> * Schema overhaul (flyteorg#785) Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Yee Hing Tong <[email protected]> * Parent workflow serialization fails when calling a launch plan with fixed inputs (flyteorg#814) Signed-off-by: Kevin Su <[email protected]> Signed-off-by: maximsmol <[email protected]> * Fix sagemaker plugin (flyteorg#817) Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Signed-off-by: maximsmol <[email protected]> * Remove legacy API (flyteorg#807) Signed-off-by: Yee Hing Tong <[email protected]> * Add Bigquery plugin (flyteorg#789) * Add bigquery plugin Signed-off-by: Kevin Su <[email protected]> * Update dependency Signed-off-by: Kevin Su <[email protected]> * update get_custom Signed-off-by: Kevin Su <[email protected]> * Add structured dataset Signed-off-by: Kevin Su <[email protected]> * Add structured dataset Signed-off-by: Kevin Su <[email protected]> * Updated comment Signed-off-by: Kevin Su <[email protected]> * Add BQ in GA Signed-off-by: Kevin Su <[email protected]> * alphabetical order Signed-off-by: Kevin Su <[email protected]> Signed-off-by: maximsmol <[email protected]> * Fixed flytekit-papermill ImportError (flyteorg#818) Signed-off-by: Kevin Su <[email protected]> * Add support for string-format-like sytax for shell task (flyteorg#792) * POC: Add support for f-string like sytax for shell task This commit is a proof of concept adding f-string like syntax for shell_tasks. This supports using nested types for script inputs, such as data classes. This change was motivated by the desire to combine shell_tasks that have multiple inputs with map_tasks which only support tasks with a single input. This commit is only a starting point, since it makes some changes to the shell_task API (adds a template_style field), and modifies some of the default behavior for ease of implementation (e.g. throwing an error when there are unused input arguments). Signed-off-by: Zach Palchick <[email protected]> * Drop support for old/regex style for doing string interpolation Signed-off-by: Zach Palchick <[email protected]> Signed-off-by: maximsmol <[email protected]> * Expose configured RawOutputPrefix during execution (flyteorg#813) * Expose configured RawOutputPrefix during execution Signed-off-by: Kevin Su <[email protected]> * Remove sdk_runnable.py and spark_task.py Signed-off-by: Kevin Su <[email protected]> Signed-off-by: maximsmol <[email protected]> * Add SecretsManager back to old import location (flyteorg#820) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: maximsmol <[email protected]> * Add some tests (flyteorg#819) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: maximsmol <[email protected]> * Fixed flaky spark test (flyteorg#821) Signed-off-by: Kevin Su <[email protected]> Signed-off-by: maximsmol <[email protected]> * fix: plugins/flytekit-greatexpectations/requirements.txt to reduce vulnerabilities (flyteorg#823) The following vulnerabilities are fixed by pinning transitive dependencies: - https://snyk.io/vuln/SNYK-PYTHON-IPYTHON-2348630 Signed-off-by: maximsmol <[email protected]> * fix: plugins/flytekit-papermill/requirements.txt to reduce vulnerabilities (flyteorg#825) The following vulnerabilities are fixed by pinning transitive dependencies: - https://snyk.io/vuln/SNYK-PYTHON-IPYTHON-2348630 Signed-off-by: maximsmol <[email protected]> * fix: requirements-spark2.txt to reduce vulnerabilities (flyteorg#826) The following vulnerabilities are fixed by pinning transitive dependencies: - https://snyk.io/vuln/SNYK-PYTHON-IPYTHON-2348630 Signed-off-by: maximsmol <[email protected]> * fix: requirements.txt to reduce vulnerabilities (flyteorg#824) The following vulnerabilities are fixed by pinning transitive dependencies: - https://snyk.io/vuln/SNYK-PYTHON-IPYTHON-2348630 Signed-off-by: maximsmol <[email protected]> * Intratask checkpointing (flyteorg#771) * wip - intratask checkpointing Signed-off-by: Ketan Umare <[email protected]> * sync checkpointer with tests Signed-off-by: Ketan Umare <[email protected]> * Checkpinter in entrypoint Signed-off-by: Ketan Umare <[email protected]> * checkpoint in progress Signed-off-by: Ketan Umare <[email protected]> * wip Signed-off-by: Ketan Umare <[email protected]> * Intratask checkpointer Signed-off-by: Ketan Umare <[email protected]> * Checkpoint updated Signed-off-by: Ketan Umare <[email protected]> * Intra-task checkpointing Signed-off-by: Ketan Umare <[email protected]> * Test and entrypoint updated Signed-off-by: Ketan Umare <[email protected]> * lint fixed Signed-off-by: Ketan Umare <[email protected]> * test fixes Signed-off-by: Ketan Umare <[email protected]> * fmt Signed-off-by: Ketan Umare <[email protected]> * updated entrypoint Signed-off-by: Ketan Umare <[email protected]> * updated Signed-off-by: Ketan Umare <[email protected]> * update Signed-off-by: Ketan Umare <[email protected]> * print Signed-off-by: Ketan Umare <[email protected]> * updated Signed-off-by: Ketan Umare <[email protected]> * SyncCheckpointer working Signed-off-by: Ketan Umare <[email protected]> * updated Signed-off-by: Ketan Umare <[email protected]> * update Signed-off-by: Ketan Umare <[email protected]> * fixed import problems Signed-off-by: Ketan Umare <[email protected]> * fixed test Signed-off-by: Ketan Umare <[email protected]> * fixed imports Signed-off-by: Ketan Umare <[email protected]> * fixed lints and errors Signed-off-by: Ketan Umare <[email protected]> * lint fix Signed-off-by: Ketan Umare <[email protected]> * addressed comments Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: maximsmol <[email protected]> * Support reading subset column (flyteorg#822) * Support StructuredDatasetDecoder read subset column Signed-off-by: Kevin Su <[email protected]> * Added tests Signed-off-by: Kevin Su <[email protected]> * Fixed tests Signed-off-by: Kevin Su <[email protected]> * Fixed typo Signed-off-by: Kevin Su <[email protected]> * Updated tests Signed-off-by: Kevin Su <[email protected]> * [pr into flyteorg#822] (flyteorg#827) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Kevin Su <[email protected]> * [pr into flyteorg#822] Final update to structured dataset column subsetting (flyteorg#828) Signed-off-by: Yee Hing Tong <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]> * Fix spark regression (flyteorg#830) Signed-off-by: Kevin Su <[email protected]> Signed-off-by: maximsmol <[email protected]> * Update argument setting for in fast registered, dynamically generated, pod tasks (flyteorg#835) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: maximsmol <[email protected]> * `ctx` Context can be used within shell tasks - to access context vars and secrets (flyteorg#832) * Adding context to a substitutable parameter in shell task Signed-off-by: Ketan Umare <[email protected]> * Support for secrets in context Signed-off-by: Ketan Umare <[email protected]> * addressed comments Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: maximsmol <[email protected]> * Expose Checkpoint as a top-level interface in flytekit (flyteorg#839) Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: maximsmol <[email protected]> * Parse duration field from flyteidl to `flytekit.models.execution.ExecutionClosure` (flyteorg#829) * Parse duration field from flyteidl to `flytekit.models.execution.ExecutionClosure` Signed-off-by: Bernhard Stadlbauer <[email protected]> * Add test for execution closure Signed-off-by: Bernhard Stadlbauer <[email protected]> * Add tests to Flyte remote Signed-off-by: Bernhard Stadlbauer <[email protected]> * Split execution test into with output and with error Signed-off-by: Bernhard Stadlbauer <[email protected]> Co-authored-by: Bernhard Stadlbauer <[email protected]> Signed-off-by: maximsmol <[email protected]> * Gate new Structured Dataset feature & remove old config objects (flyteorg#831) Signed-off-by: Yee Hing Tong <[email protected]> * Fixing out of order for conditional outputs (flyteorg#843) Signed-off-by: maximsmol <[email protected]> * Set default values to map task template (flyteorg#841) * Set sane defaults in map task templates Signed-off-by: Eduardo Apolinario <[email protected]> * Remove unused method Signed-off-by: Eduardo Apolinario <[email protected]> * Put ArrayJob.from_dict back Signed-off-by: Eduardo Apolinario <[email protected]> * Define parallelism=0 as unbounded Signed-off-by: Eduardo Apolinario <[email protected]> * Remove special case to handle 0 Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Signed-off-by: maximsmol <[email protected]> * Regenerate requirements files - dependencies dropping support for python 3.7 (flyteorg#838) * Regenerate requirements files Signed-off-by: Eduardo Apolinario <[email protected]> * Put restrictions on numpy and pandas versions Signed-off-by: Eduardo Apolinario <[email protected]> * Use --use-deprecated=legacy-resolver Signed-off-by: Eduardo Apolinario <[email protected]> * use pip==22.0.3 everywhere Signed-off-by: Eduardo Apolinario <[email protected]> * Remove --use-deprecated=legacy-resolver Signed-off-by: Eduardo Apolinario <[email protected]> * Relax click Signed-off-by: Eduardo Apolinario <[email protected]> * Regenerated plugins requirements Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> * TypeAnnotation (flyteorg#759) * feat: support for annotated simple + list Signed-off-by: Kenny Workman <[email protected]> * feat: addition of annotation att to Signed-off-by: Kenny Workman <[email protected]> * feat: core obj Signed-off-by: Kenny Workman <[email protected]> * feat: proto model Signed-off-by: Kenny Workman <[email protected]> * feat: testing suite Signed-off-by: Kenny Workman <[email protected]> * fix: more stable typing introspection Signed-off-by: Kenny Workman <[email protected]> * fix: strip legacy Signed-off-by: Kenny Workman <[email protected]> * fix: explicitly allow only one annotation Signed-off-by: Kenny Workman <[email protected]> * feat: direct type transformer tests Signed-off-by: Kenny Workman <[email protected]> * fix: there and back test Signed-off-by: Kenny Workman <[email protected]> * fix: typing_extensions for get_origin Signed-off-by: Kenny Workman <[email protected]> * fix: more semantic list generic unwrap Signed-off-by: Kenny Workman <[email protected]> * fix: tmp requirements file with custom idl Signed-off-by: Kenny Workman <[email protected]> * fix: nits Signed-off-by: Kenny Workman <[email protected]> * feat: semantic error for unsupported complex literals Signed-off-by: Kenny Workman <[email protected]> * fix: but Signed-off-by: Kenny Workman <[email protected]> * feat: more tests ;) Signed-off-by: Kenny Workman <[email protected]> * fix: imports Signed-off-by: Kenny Workman <[email protected]> * fix: complex annotations Signed-off-by: Kenny Workman <[email protected]> * fix: temp requirements files for unit tests Signed-off-by: Kenny Workman <[email protected]> * fix: lint bug Signed-off-by: Kenny Workman <[email protected]> * fix: tmp setup.py Signed-off-by: Kenny Workman <[email protected]> * fix: use typing_extensions Signed-off-by: Kenny Workman <[email protected]> * fix: typing_extensions for annotated Signed-off-by: Kenny Workman <[email protected]> * fix: typing_ext Signed-off-by: Kenny Workman <[email protected]> * fix: plugin tmp requirements Signed-off-by: Kenny Workman <[email protected]> * fix: bump requirements Signed-off-by: Kenny Workman <[email protected]> * fix: doc requirements Signed-off-by: Kenny Workman <[email protected]> * fix: whitespace Signed-off-by: Kenny Workman <[email protected]> * fix: bump flytekit Signed-off-by: Kenny Workman <[email protected]> * fix: numpy version Signed-off-by: Kenny Workman <[email protected]> * fix: lint Signed-off-by: Kenny Workman <[email protected]> * fix: pandas version Signed-off-by: Kenny Workman <[email protected]> * fix: bump requirements Signed-off-by: Kenny Workman <[email protected]> * fix: test import Signed-off-by: Kenny Workman <[email protected]> * fix: flake8 lint Signed-off-by: Kenny Workman <[email protected]> * fix: merge Signed-off-by: Kenny Workman <[email protected]> * fix: requirements Signed-off-by: Kenny Workman <[email protected]> * fix: requirements Signed-off-by: Kenny Workman <[email protected]> * fix: lint Signed-off-by: Kenny Workman <[email protected]> * fix: papermill req Signed-off-by: Kenny Workman <[email protected]> * fix: req Signed-off-by: Kenny Workman <[email protected]> * Remove singleton from structured dataset transformer engine (flyteorg#848) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Kevin Su <[email protected]> * Assign input and output to FlyteWorkflowExecution (flyteorg#842) Signed-off-by: Kevin Su <[email protected]> Signed-off-by: maximsmol <[email protected]> * Add reference entities to FlyteTask and FlyteLaunchPlan (flyteorg#850) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: maximsmol <[email protected]> * Fix fast registration error (flyteorg#851) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Samhita Alla <[email protected]> Signed-off-by: maximsmol <[email protected]> * Add support for local execute in pod task (flyteorg#852) Signed-off-by: ggydush-fn <[email protected]> Signed-off-by: maximsmol <[email protected]> * Add anonymous retry (flyteorg#854) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: maximsmol <[email protected]> * fix: formatting, linting, typing_extensions * fix: do not use SDK types Signed-off-by: maximsmol <[email protected]> * fix: update test comment Signed-off-by: maximsmol <[email protected]> * fix: also check literal type castability when tags match Signed-off-by: maximsmol <[email protected]> * Point flyteidl to maxim's fork in CI and requirements files Signed-off-by: Eduardo Apolinario <[email protected]> * [Core feature] Add Raw AWS Batch Task (flyteorg#782) * Init plugin Signed-off-by: Kevin Su <[email protected]> * Fixed lint Signed-off-by: Kevin Su <[email protected]> * address comment Signed-off-by: Kevin Su <[email protected]> * Fixed typo Signed-off-by: Kevin Su <[email protected]> * Updated AWS config * Fixed lint Signed-off-by: Kevin Su <[email protected]> * Added comment Signed-off-by: Kevin Su <[email protected]> * Update config Signed-off-by: Kevin Su <[email protected]> * Fixed tests Signed-off-by: Kevin Su <[email protected]> * Fixed tests Signed-off-by: Kevin Su <[email protected]> * Fixed tests Signed-off-by: Kevin Su <[email protected]> * use pyflyte execute Signed-off-by: Kevin Su <[email protected]> * Fixed tests Signed-off-by: Kevin Su <[email protected]> * Fixed tests Signed-off-by: Kevin Su <[email protected]> * Added comment Signed-off-by: Kevin Su <[email protected]> Signed-off-by: maximsmol <[email protected]> * Add structured dataset encoder/decoder in fsspec plugin (flyteorg#849) Signed-off-by: Kevin Su <[email protected]> Signed-off-by: maximsmol <[email protected]> * Delete unnecessary auth configuration (flyteorg#858) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: maximsmol <[email protected]> * Fixed format alias in Flytekit docs (flyteorg#844) * Fixed format alias Fixed docs for file format alias that weren't rendered properly. A warning popped up stating 'ignore' is deprecated, and to use 'ignore_paths' instead. Signed-off-by: SmritiSatyanV <[email protected]> * Bump idl (flyteorg#862) Signed-off-by: Yee Hing Tong <[email protected]> * Updated authoring.rst (flyteorg#863) * Updated authoring.rst Added directive Rephrased sentence * Fixed build error Signed-off-by: SmritiSatyanV <[email protected]> * test-build-1 Signed-off-by: SmritiSatyanV <[email protected]> Signed-off-by: maximsmol <[email protected]> * Updated authoring.rst (flyteorg#866) Added the directive correctly Signed-off-by: SmritiSatyanV <[email protected]> Signed-off-by: maximsmol <[email protected]> * Change docs for HTMLPage type to say HTMLPage instead of PNGImage (flyteorg#868) Signed-off-by: maximsmol <[email protected]> * Revisit StructuredDatasetDecoder interface (flyteorg#865) Signed-off-by: Kevin Su <[email protected]> Signed-off-by: maximsmol <[email protected]> * Remove legacy mentions in contributing guide (flyteorg#870) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: maximsmol <[email protected]> * Add GCS protocol in the structured dataset (flyteorg#869) Signed-off-by: Kevin Su <[email protected]> Signed-off-by: maximsmol <[email protected]> * Make fetched entities callable within workflows (flyteorg#867) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: maximsmol <[email protected]> * Caching of offloaded objects (flyteorg#762) * Remove flyteidl from install_requires Signed-off-by: Eduardo Apolinario <[email protected]> * Expose hash in Literal Signed-off-by: Eduardo Apolinario <[email protected]> * Set hash in TypeEngine Signed-off-by: Eduardo Apolinario <[email protected]> * Modify cache key calculation to take hash into account Signed-off-by: Eduardo Apolinario <[email protected]> * Opt-in PandasDataFrameTransformer Signed-off-by: Eduardo Apolinario <[email protected]> * Add unit tests Signed-off-by: Eduardo Apolinario <[email protected]> * Iterate using a flyteidl branch Signed-off-by: Eduardo Apolinario <[email protected]> * Regenerate requirements files Signed-off-by: Eduardo Apolinario <[email protected]> * Regenerate requirements files Signed-off-by: Eduardo Apolinario <[email protected]> * Move _hash_overridable to StructureDatasetTransformerEngine Signed-off-by: Eduardo Apolinario <[email protected]> * Move HashMethod to flytekit.core.hash Signed-off-by: Eduardo Apolinario <[email protected]> * Fix `unit_test` make target Signed-off-by: Eduardo Apolinario <[email protected]> * Split `unit_test` make target in two lines Signed-off-by: Eduardo Apolinario <[email protected]> * Add assert to structured dataset compatibility test Signed-off-by: Eduardo Apolinario <[email protected]> * Remove TODO Signed-off-by: Eduardo Apolinario <[email protected]> * Regenerate plugins requirements files pointing to the right version of flyteidl. Signed-off-by: Eduardo Apolinario <[email protected]> * Set hash as a property of the literal Signed-off-by: Eduardo Apolinario <[email protected]> * Install plugins requirements in CI. Signed-off-by: Eduardo Apolinario <[email protected]> * Add hash.setter Signed-off-by: Eduardo Apolinario <[email protected]> * Install flyteidl directly Signed-off-by: Eduardo Apolinario <[email protected]> * Revert "Regenerate plugins requirements files pointing to the right version of flyteidl." This reverts commit c2dbb54. Signed-off-by: Eduardo Apolinario <[email protected]> * wip - Add support for univariate lists Signed-off-by: Eduardo Apolinario <[email protected]> * Add support for lists of annotated objects Signed-off-by: Eduardo Apolinario <[email protected]> * Revamp generation of cache key (to cover case of literals collections and maps) Signed-off-by: Eduardo Apolinario <[email protected]> * Leave TODO for warning Signed-off-by: Eduardo Apolinario <[email protected]> * Revert "Add support for lists of annotated objects" This reverts commit 4b5f608. Signed-off-by: Eduardo Apolinario <[email protected]> * Revert "wip - Add support for univariate lists" This reverts commit adaa448. Signed-off-by: Eduardo Apolinario <[email protected]> * Remove docstring Signed-off-by: Eduardo Apolinario <[email protected]> * Add flyteidl>=0.23.0 Signed-off-by: Eduardo Apolinario <[email protected]> * Remove mentions to branch flyteidl@add-hash-to-literal Signed-off-by: Eduardo Apolinario <[email protected]> * Bump flyteidl in plugins requirements Signed-off-by: Eduardo Apolinario <[email protected]> * Regenerate plugins requirements again Signed-off-by: Eduardo Apolinario <[email protected]> * Restore papermill/requirements.txt Signed-off-by: Eduardo Apolinario <[email protected]> * Point flytekitplugins-spark to the offloaded-objects-caching branch in papermill tests Signed-off-by: Eduardo Apolinario <[email protected]> * Set flyteidl>=0.23.0 in papermill dev-requirements Co-authored-by: Eduardo Apolinario <[email protected]> * FlyteRemote fetch of conditional nodes (flyteorg#772) Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: maximsmol <[email protected]> * Removed root logger (flyteorg#871) * Removed root logger Signed-off-by: Kevin Su <[email protected]> * Updated logger Signed-off-by: Kevin Su <[email protected]> * Fixed lint Signed-off-by: Kevin Su <[email protected]> * Updated log level Signed-off-by: Kevin Su <[email protected]> * Updated logger Signed-off-by: Kevin Su <[email protected]> * Updated logger Signed-off-by: Kevin Su <[email protected]> * Updated logger Signed-off-by: Kevin Su <[email protected]> * lint fixed Signed-off-by: Kevin Su <[email protected]> Signed-off-by: maximsmol <[email protected]> * Fix flytekit_compatibility/test_schema_types.py test Signed-off-by: Eduardo Apolinario <[email protected]> Signed-off-by: maximsmol <[email protected]> * make lint Signed-off-by: Eduardo Apolinario <[email protected]> * fix: annotated type conversion error Signed-off-by: maximsmol <[email protected]> * fix: _are_types_castable based on tests Signed-off-by: maximsmol <[email protected]> * fix: test failing if using random order Signed-off-by: maximsmol <[email protected]> * Merge branch 'master' into maximsmol/union_type Signed-off-by: maximsmol <[email protected]> * fix: merge issue Signed-off-by: maximsmol <[email protected]> * fix: requirements Signed-off-by: maximsmol <[email protected]> * fix: schema transformer error Signed-off-by: maximsmol <[email protected]> * fix: test Signed-off-by: maximsmol <[email protected]> * fix: merge issue Signed-off-by: maximsmol <[email protected]> * fix: union + annotated behavior Signed-off-by: maximsmol <[email protected]> * Regenerate requirements Signed-off-by: Eduardo Apolinario <[email protected]> * Bump requirements in plugins Signed-off-by: Eduardo Apolinario <[email protected]> * Handle nested Annotated Signed-off-by: Eduardo Apolinario <[email protected]> * Leave TODO re: strucutured dataset type castability Signed-off-by: Eduardo Apolinario <[email protected]> * Remove mention to flyteidl@union_type in doc-requirements Signed-off-by: Eduardo Apolinario <[email protected]> * Linting Signed-off-by: Eduardo Apolinario <[email protected]> * Use tempfile.mkdtemp to create a temporary directory for local data persistence. Signed-off-by: Eduardo Apolinario <[email protected]> * Revert "Use tempfile.mkdtemp to create a temporary directory for local data persistence." This reverts commit 861b157. Signed-off-by: eduardo apolinario <[email protected]> * Force temporary file to not be deleted in test Signed-off-by: eduardo apolinario <[email protected]> * Regenerate papermill dev-requirements Signed-off-by: eduardo apolinario <[email protected]> * Remove duplicate code Signed-off-by: Eduardo Apolinario <[email protected]> * Put a lower bound on the pip version installed in CI Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: Kevin Su <[email protected]> Co-authored-by: Emirhan Karagül <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]> Co-authored-by: Kevin Su <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]> Co-authored-by: Ketan Umare <[email protected]> Co-authored-by: Lisa <[email protected]> Co-authored-by: Niels Bantilan <[email protected]> Co-authored-by: Stef Nelson-Lindall <[email protected]> Co-authored-by: Snyk bot <[email protected]> Co-authored-by: Dan Rammer <[email protected]> Co-authored-by: bstadlbauer <[email protected]> Co-authored-by: Bernhard Stadlbauer <[email protected]> Co-authored-by: Bernhard Stadlbauer <[email protected]> Co-authored-by: Samhita Alla <[email protected]> Co-authored-by: Yuvraj <[email protected]> Co-authored-by: Zach Palchick <[email protected]> Co-authored-by: Snyk bot <[email protected]> Co-authored-by: Kenny Workman <[email protected]> Co-authored-by: ggydush-fn <[email protected]> Co-authored-by: SmritiSatyanV <[email protected]> Co-authored-by: Matthew Griffin <[email protected]> Co-authored-by: eduardo apolinario <[email protected]> Signed-off-by: Mike Zhong <[email protected]>
reverson
pushed a commit
to reverson/flytekit
that referenced
this pull request
May 27, 2022
* [wip] for feast demo Signed-off-by: Ketan Umare <[email protected]> * clean up a bit Signed-off-by: Yee Hing Tong <[email protected]> * add a test and move where constructor is called Signed-off-by: Yee Hing Tong <[email protected]> * remove unneeded import Signed-off-by: Yee Hing Tong <[email protected]> * add a part of a test Signed-off-by: Yee Hing Tong <[email protected]> * Added tests Signed-off-by: Kevin Su <[email protected]> * Fixed lint Signed-off-by: Kevin Su <[email protected]> * typo Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]> Co-authored-by: Kevin Su <[email protected]> Signed-off-by: Robert Everson <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Ketan Umare [email protected]
TL;DR
Get raw input/output (Literal) from remote execution.
Type
Are all requirements met?
Complete description
How did you fix the bug, make the feature etc. Link to any design docs etc
Tracking Issue
https://github.com/lyft/flyte/issues/
Follow-up issue
NA
OR
https://github.com/lyft/flyte/issues/