Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Snyk] Security upgrade ipython from 7.30.1 to 7.31.1 #824

Merged
merged 1 commit into from
Jan 21, 2022

Conversation

snyk-bot
Copy link
Contributor

Snyk has created this PR to fix one or more vulnerable packages in the `pip` dependencies of this project.

Changes included in this PR

  • Changes to the following files to upgrade the vulnerable dependencies to a fixed version:
    • requirements.txt
⚠️ Warning
sagemaker-training 3.9.2 requires scipy, which is not installed.
sagemaker-training 3.9.2 requires werkzeug, which is not installed.
jupyterlab-pygments 0.1.2 requires pygments, which is not installed.
jsonschema 3.2.0 requires pyrsistent, which is not installed.
cookiecutter 1.7.3 requires python-slugify, which is not installed.

Vulnerabilities that will be fixed

By pinning:
Severity Priority Score (*) Issue Upgrade Breaking Change Exploit Maturity
high severity 696/1000
Why? Recently disclosed, Has a fix available, CVSS 8.2
Arbitrary Code Execution
SNYK-PYTHON-IPYTHON-2348630
ipython:
7.30.1 -> 7.31.1
No No Known Exploit

(*) Note that the real score may have changed since the PR was raised.

Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the effected dependencies could be upgraded.

Check the changes in this PR to ensure they won't cause issues with your project.


Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.

For more information:
🧐 View latest project report

🛠 Adjust project settings

📚 Read more about Snyk's upgrade and patch logic

The following vulnerabilities are fixed by pinning transitive dependencies:
- https://snyk.io/vuln/SNYK-PYTHON-IPYTHON-2348630
@codecov
Copy link

codecov bot commented Jan 21, 2022

Codecov Report

Merging #824 (05943a5) into master (88c7d38) will not change coverage.
The diff coverage is n/a.

Impacted file tree graph

@@           Coverage Diff           @@
##           master     #824   +/-   ##
=======================================
  Coverage   86.76%   86.76%           
=======================================
  Files         244      244           
  Lines       21505    21505           
  Branches     2399     2399           
=======================================
  Hits        18659    18659           
  Misses       2427     2427           
  Partials      419      419           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 88c7d38...05943a5. Read the comment docs.

@eapolinario eapolinario merged commit 3d3c995 into master Jan 21, 2022
eapolinario pushed a commit that referenced this pull request Jan 28, 2022
The following vulnerabilities are fixed by pinning transitive dependencies:
- https://snyk.io/vuln/SNYK-PYTHON-IPYTHON-2348630

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
kennyworkman pushed a commit to kennyworkman/flytekit that referenced this pull request Feb 8, 2022
The following vulnerabilities are fixed by pinning transitive dependencies:
- https://snyk.io/vuln/SNYK-PYTHON-IPYTHON-2348630
Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>
kennyworkman pushed a commit to kennyworkman/flytekit that referenced this pull request Feb 8, 2022
The following vulnerabilities are fixed by pinning transitive dependencies:
- https://snyk.io/vuln/SNYK-PYTHON-IPYTHON-2348630
eapolinario added a commit that referenced this pull request Mar 21, 2022
* Add support union type

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Fixed test

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Fixed test

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Fixed lint

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Fixed tests

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Updated tests

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Update Union to use tagged Unions

Signed-off-by: maximsmol <maximsmol@gmail.com>

* Update to use string tags (part 1)

Signed-off-by: maximsmol <maximsmol@gmail.com>

* Working implementation, update tests

Signed-off-by: maximsmol <maximsmol@gmail.com>

* Fixes, more tests

Signed-off-by: maximsmol <maximsmol@gmail.com>

* Fix incorrect unwrapped literal-union matching, update test

Signed-off-by: maximsmol <maximsmol@gmail.com>

* Fix duplicate tag handling, add tests for collections containing unions

Signed-off-by: maximsmol <maximsmol@gmail.com>

* Fix type hint test

Signed-off-by: maximsmol <maximsmol@gmail.com>

* Add implicit wrapping union type tests

Signed-off-by: maximsmol <maximsmol@gmail.com>

* Add union ambiguity tests

Signed-off-by: maximsmol <maximsmol@gmail.com>

* Fixup tests, make TypeTransformerFailed compatible with all old exception types

Signed-off-by: maximsmol <maximsmol@gmail.com>

* Fixup models + add tests

Signed-off-by: maximsmol <maximsmol@gmail.com>

* Implement changed design

Signed-off-by: maximsmol <maximsmol@gmail.com>

* Fix PR comments

Signed-off-by: maximsmol <maximsmol@gmail.com>

* Remote entrypoint serialize (#733)

Signed-off-by: Emirhan Karagül <emirhan350z@gmail.com>
Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Fix lint error in remote.py (#755)

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Support enum in dataclass (#753)

* Add support enum in dataclass

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Update test

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Fixed lint

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Fix subworkflow and launch plan FlyteRemote behavior (#751)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Extras: Shell task (#747)

Signed-off-by: maximsmol <maximsmol@gmail.com>

* Add support FlyteSchema in dataclass (#722)

* schema in dataclass

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Added tests

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Fixed lint

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Updated tests

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Updated tests

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Fixed lint

Signed-off-by: Kevin Su <pingsutw@apache.org>

* updated

Signed-off-by: Kevin Su <pingsutw@apache.org>

* updated

Signed-off-by: Kevin Su <pingsutw@apache.org>

* updated

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Remove workflow_execution.py (#758)

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Get raw input/output from remote execution (#675)

* [wip] for feast demo

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* clean up a bit

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* add a test and move where constructor is called

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* remove unneeded import

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* add a part of a test

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* Added tests

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Fixed lint

Signed-off-by: Kevin Su <pingsutw@apache.org>

* typo

Signed-off-by: Kevin Su <pingsutw@apache.org>

Co-authored-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Co-authored-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Fix mypy errors in flytekit/types (#757)

Signed-off-by: Lisa <aeioulisa@gmail.com>
Signed-off-by: Kevin Su <pingsutw@apache.org>

* Remote client failed to fetch FlytePickle object (#764)

* Fetch pickle value from flytekit remote

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Fix tests

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Remove default value

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Add support FlyteFile in dataclass (#725)

* Add support Flyte File and directory in dataclass

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Fixed tests

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Fixed tests

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Updated

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Updated

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Fixed tests

Signed-off-by: Kevin Su <pingsutw@apache.org>

* add task_resolver arg to @task decorator (#765)

Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Copy metadata into map task from underlying (#766)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Support for delayed annotations (#760)

Signed-off-by: Stefan Nelson-Lindall <stef@stripe.com>
Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Complex dataclass unit tests (#773)

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>
Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* remote_source lost on serialization of @dataclass_json with FlyteFile (#774)

* remote_source lost on serialization of @dataclass_json with FlyteFile

Signed-off-by: Kevin Su <pingsutw@apache.org>

* updated tests

Signed-off-by: Kevin Su <pingsutw@apache.org>

* updated tests

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Single-task execution FlyteRemote sync (#778)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Logging updates (#775)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* fix: doc-requirements.txt to reduce vulnerabilities (#779)

The following vulnerabilities are fixed by pinning transitive dependencies:
- https://snyk.io/vuln/SNYK-PYTHON-LXML-2316995

Signed-off-by: maximsmol <maximsmol@gmail.com>

* Add cache_serialize parameter to tasks (#673)

* added cache_serialize parameter for basic tasks

Signed-off-by: Daniel Rammer <daniel@union.ai>

* fixed typo

Signed-off-by: Daniel Rammer <daniel@union.ai>

* updated flyteidl version

Signed-off-by: Daniel Rammer <daniel@union.ai>

* changed flyteidl requirements everywhere

Signed-off-by: Daniel Rammer <daniel@union.ai>

* remove flyteidl version requires in setup.py so we can use develop

Signed-off-by: Daniel Rammer <daniel@union.ai>

* removed flyteidl git repos from a variety of requirements packages

Signed-off-by: Daniel Rammer <daniel@union.ai>

* updated variable discovery_serializable to cache_serializable

Signed-off-by: Daniel Rammer <daniel@union.ai>

* updated requirements

Signed-off-by: Daniel Rammer <daniel@union.ai>

* fixed TaskMetadata _cache_serializable variable name

Signed-off-by: Daniel Rammer <daniel@union.ai>

* propgating cache_serialize parameter through to tasks

Signed-off-by: Daniel Rammer <daniel@union.ai>

* added cache_serializable to SdkRawContainerTask

Signed-off-by: Daniel Rammer <daniel@union.ai>

* fixing cache_serializable variable propogation issues

Signed-off-by: Daniel Rammer <daniel@union.ai>

* added documentation

Signed-off-by: Daniel Rammer <daniel@union.ai>

* added unit tests for cache_serialize metadata

Signed-off-by: Daniel Rammer <daniel@union.ai>

* linter added spaces in unit tests

Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* When using the `task` and `workflow` decorator, correctly wrap the fu… (#780)

* When using the `task` and `workflow` decorator, correctly wrap the function

This enables tooling such as docstring search tools to unwrap the object and show the correct docstring.

Signed-off-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai>

* Remove blackshark copyright header

Signed-off-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai>

* Fix broken great expectations test

Signed-off-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai>

* Add test for stacked decorators

Signed-off-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai>

Co-authored-by: Bernhard Stadlbauer <bstadlbauer@gmx.net>
Co-authored-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Add option to flyte-cli for specifying root certificate (#783)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Add validation check to cacert switch (#787)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Remove pytz constraint (#786)

* Remove pytz constraint from setup.py

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Regenerate requirements files

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Put pytz back

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* make requirements.txt

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Lint: remove f-string misuse (#788)

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Pyarrow greater than 4.0.0 (#790)

* pyarrow>=4.0.0

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Regenerate requirements.

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Support python 3.10 (#791)

* [wip] Support python 3.10

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Add Dockerfile.py310

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Stringify python version

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Skip flytekit-modin plugin tests on 3.10

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Add 3.9 and 3.10 to list of supported version in plugins

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Comment why flytekit-modin is not running on 3.10 and disable fail-fast in plugin tests

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* add `with_overrides` to map task (#794)

* add with_overrides

Signed-off-by: Samhita Alla <aallasamhita@gmail.com>

* remove Resources

Signed-off-by: Samhita Alla <aallasamhita@gmail.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* bump docsearch version (#805)

Signed-off-by: Samhita Alla <aallasamhita@gmail.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* update docs for new navbar theme (#806)

Signed-off-by: maximsmol <maximsmol@gmail.com>

* fix requirment.txt github issue (#810)

Signed-off-by: Yuvraj <code@evalsocket.dev>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Add sphinx panels (#815)

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Schema overhaul (#785)

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* Parent workflow serialization fails when calling a launch plan with fixed inputs (#814)

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Fix sagemaker plugin (#817)

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Remove legacy API (#807)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* Add Bigquery plugin (#789)

* Add bigquery plugin

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Update dependency

Signed-off-by: Kevin Su <pingsutw@apache.org>

* update get_custom

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Add structured dataset

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Add structured dataset

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Updated comment

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Add BQ in GA

Signed-off-by: Kevin Su <pingsutw@apache.org>

* alphabetical order

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Fixed flytekit-papermill ImportError (#818)

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Add support for string-format-like sytax for shell task (#792)

* POC: Add support for f-string like sytax for shell task

This commit is a proof of concept adding f-string like syntax for
shell_tasks. This supports using nested types for script inputs, such
as data classes. This change was motivated by the desire to combine
shell_tasks that have multiple inputs with map_tasks which only support
tasks with a single input.

This commit is only a starting point, since it makes some changes to the
shell_task API (adds a template_style field), and modifies some of the default
behavior for ease of implementation (e.g. throwing an error when there
are unused input arguments).

Signed-off-by: Zach Palchick <palchicz@zymergen.com>

* Drop support for old/regex style for doing string interpolation

Signed-off-by: Zach Palchick <palchicz@zymergen.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Expose configured RawOutputPrefix during execution (#813)

* Expose configured RawOutputPrefix during execution

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Remove sdk_runnable.py and spark_task.py

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Add SecretsManager back to old import location (#820)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Add some tests (#819)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Fixed flaky spark test (#821)

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* fix: plugins/flytekit-greatexpectations/requirements.txt to reduce vulnerabilities (#823)

The following vulnerabilities are fixed by pinning transitive dependencies:
- https://snyk.io/vuln/SNYK-PYTHON-IPYTHON-2348630

Signed-off-by: maximsmol <maximsmol@gmail.com>

* fix: plugins/flytekit-papermill/requirements.txt to reduce vulnerabilities (#825)

The following vulnerabilities are fixed by pinning transitive dependencies:
- https://snyk.io/vuln/SNYK-PYTHON-IPYTHON-2348630

Signed-off-by: maximsmol <maximsmol@gmail.com>

* fix: requirements-spark2.txt to reduce vulnerabilities (#826)

The following vulnerabilities are fixed by pinning transitive dependencies:
- https://snyk.io/vuln/SNYK-PYTHON-IPYTHON-2348630

Signed-off-by: maximsmol <maximsmol@gmail.com>

* fix: requirements.txt to reduce vulnerabilities (#824)

The following vulnerabilities are fixed by pinning transitive dependencies:
- https://snyk.io/vuln/SNYK-PYTHON-IPYTHON-2348630

Signed-off-by: maximsmol <maximsmol@gmail.com>

* Intratask checkpointing  (#771)

* wip - intratask checkpointing

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* sync checkpointer with tests

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* Checkpinter in entrypoint

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* checkpoint in progress

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* wip

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* Intratask checkpointer

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* Checkpoint updated

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* Intra-task checkpointing

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* Test and entrypoint updated

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* lint fixed

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* test fixes

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* fmt

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* updated entrypoint

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* updated

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* update

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* print

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* updated

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* SyncCheckpointer working

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* updated

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* update

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* fixed import problems

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* fixed test

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* fixed imports

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* fixed lints and errors

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* lint fix

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* addressed comments

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Support reading subset column (#822)

* Support StructuredDatasetDecoder read subset column

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Added tests

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Fixed tests

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Fixed typo

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Updated tests

Signed-off-by: Kevin Su <pingsutw@apache.org>

* [pr into #822]  (#827)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

Signed-off-by: Kevin Su <pingsutw@apache.org>

* [pr into #822] Final update to structured dataset column subsetting (#828)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

Co-authored-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* Fix spark regression (#830)

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Update argument setting for in fast registered, dynamically generated, pod tasks (#835)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* `ctx` Context can be used within shell tasks - to access context vars and secrets (#832)

* Adding context to a substitutable parameter in shell task

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* Support for secrets in context

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* addressed comments

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Expose Checkpoint as a top-level interface in flytekit (#839)

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Parse duration field from flyteidl to `flytekit.models.execution.ExecutionClosure` (#829)

* Parse duration field from flyteidl to `flytekit.models.execution.ExecutionClosure`

Signed-off-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai>

* Add test for execution closure

Signed-off-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai>

* Add tests to Flyte remote

Signed-off-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai>

* Split execution test into with output and with error

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

Co-authored-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Gate new Structured Dataset feature & remove old config objects (#831)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* Fixing out of order for conditional outputs (#843)

Signed-off-by: maximsmol <maximsmol@gmail.com>

* Set default values to map task template (#841)

* Set sane defaults in map task templates

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Remove unused method

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Put ArrayJob.from_dict back

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Define parallelism=0 as unbounded

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Remove special case to handle 0

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Regenerate requirements files - dependencies dropping support for python 3.7 (#838)

* Regenerate requirements files

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Put restrictions on numpy and pandas versions

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Use --use-deprecated=legacy-resolver

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* use pip==22.0.3 everywhere

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Remove --use-deprecated=legacy-resolver

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Relax click

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Regenerated plugins requirements

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* TypeAnnotation (#759)

* feat:  support for annotated simple + list

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* feat: addition of annotation att to

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* feat: core  obj

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* feat:  proto model

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* feat: testing suite

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: more stable typing introspection

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: strip legacy

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: explicitly allow only one annotation

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* feat: direct type transformer tests

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: there and back test

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: typing_extensions for get_origin

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: more semantic list generic unwrap

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: tmp requirements file with custom idl

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: nits

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* feat: semantic error for unsupported complex literals

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: but

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* feat: more tests ;)

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: imports

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: complex annotations

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: temp requirements files for unit tests

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: lint bug

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: tmp setup.py

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: use typing_extensions

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: typing_extensions for annotated

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: typing_ext

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: plugin tmp requirements

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: bump requirements

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: doc requirements

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: whitespace

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: bump flytekit

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: numpy version

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: lint

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: pandas version

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: bump requirements

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: test import

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: flake8 lint

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: merge

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: requirements

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: requirements

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: lint

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: papermill req

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: req

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* Remove singleton from structured dataset transformer engine (#848)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: Kevin Su <pingsutw@apache.org>

* Assign input and output to FlyteWorkflowExecution (#842)

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Add reference entities to FlyteTask and FlyteLaunchPlan (#850)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Fix fast registration error (#851)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

Signed-off-by: Samhita Alla <aallasamhita@gmail.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Add support for local execute in pod task (#852)

Signed-off-by: ggydush-fn <greg.gydush@freenome.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Add anonymous retry (#854)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* fix: formatting, linting, typing_extensions

* fix: do not use SDK types

Signed-off-by: maximsmol <maximsmol@gmail.com>

* fix: update test comment

Signed-off-by: maximsmol <maximsmol@gmail.com>

* fix: also check literal type castability when tags match

Signed-off-by: maximsmol <maximsmol@gmail.com>

* Point flyteidl to maxim's fork in CI and requirements files

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* [Core feature] Add Raw AWS Batch Task (#782)

* Init plugin

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Fixed lint

Signed-off-by: Kevin Su <pingsutw@apache.org>

* address comment

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Fixed typo

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Updated AWS config

* Fixed lint

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Added comment

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Update config

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Fixed tests

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Fixed tests

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Fixed tests

Signed-off-by: Kevin Su <pingsutw@apache.org>

* use pyflyte execute

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Fixed tests

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Fixed tests

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Added comment

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Add structured dataset encoder/decoder in fsspec plugin (#849)

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Delete unnecessary auth configuration (#858)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Fixed format alias in Flytekit docs (#844)

* Fixed format alias

Fixed docs for file format alias that weren't rendered properly.
A warning popped up stating 'ignore'  is deprecated, and to use 'ignore_paths' instead.
Signed-off-by: SmritiSatyanV <smriti@union.ai>

* Bump idl (#862)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* Updated authoring.rst (#863)

* Updated authoring.rst

Added directive
Rephrased sentence

* Fixed build error

Signed-off-by: SmritiSatyanV <smriti@union.ai>

* test-build-1

Signed-off-by: SmritiSatyanV <smriti@union.ai>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Updated authoring.rst (#866)

Added the directive correctly
Signed-off-by: SmritiSatyanV <smriti@union.ai>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Change docs for HTMLPage type to say HTMLPage instead of PNGImage (#868)

Signed-off-by: maximsmol <maximsmol@gmail.com>

* Revisit StructuredDatasetDecoder interface (#865)

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Remove legacy mentions in contributing guide (#870)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Add GCS protocol in the structured dataset (#869)

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Make fetched entities callable within workflows (#867)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Caching of offloaded objects (#762)

* Remove flyteidl from install_requires

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Expose hash in Literal

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Set hash in TypeEngine

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Modify cache key calculation to take hash into account

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Opt-in PandasDataFrameTransformer

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Add unit tests

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Iterate using a flyteidl branch

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Regenerate requirements files

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Regenerate requirements files

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Move _hash_overridable to StructureDatasetTransformerEngine

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Move HashMethod to flytekit.core.hash

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Fix `unit_test` make target

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Split `unit_test` make target in two lines

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Add assert to structured dataset compatibility test

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Remove TODO

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Regenerate plugins requirements files pointing to the right version of flyteidl.

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Set hash as a property of the literal

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Install plugins requirements in CI.

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Add hash.setter

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Install flyteidl directly

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Revert "Regenerate plugins requirements files pointing to the right version of flyteidl."

This reverts commit c2dbb54.

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* wip - Add support for univariate lists

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Add support for lists of annotated objects

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Revamp generation of cache key (to cover case of literals collections and maps)

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Leave TODO for warning

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Revert "Add support for lists of annotated objects"

This reverts commit 4b5f608.

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Revert "wip - Add support for univariate lists"

This reverts commit adaa448.

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Remove docstring

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Add flyteidl>=0.23.0

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Remove mentions to branch flyteidl@add-hash-to-literal

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Bump flyteidl in plugins requirements

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Regenerate plugins requirements again

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Restore papermill/requirements.txt

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Point flytekitplugins-spark to the offloaded-objects-caching branch in papermill tests

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Set flyteidl>=0.23.0 in papermill dev-requirements

Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* FlyteRemote fetch of conditional nodes (#772)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Removed root logger (#871)

* Removed root logger

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Updated logger

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Fixed lint

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Updated log level

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Updated logger

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Updated logger

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Updated logger

Signed-off-by: Kevin Su <pingsutw@apache.org>

* lint fixed

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Fix flytekit_compatibility/test_schema_types.py test

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* make lint

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* fix: annotated type conversion error

Signed-off-by: maximsmol <maximsmol@gmail.com>

* fix: _are_types_castable based on tests

Signed-off-by: maximsmol <maximsmol@gmail.com>

* fix: test failing if using random order

Signed-off-by: maximsmol <maximsmol@gmail.com>

* Merge branch 'master' into maximsmol/union_type

Signed-off-by: maximsmol <maximsmol@gmail.com>

* fix: merge issue

Signed-off-by: maximsmol <maximsmol@gmail.com>

* fix: requirements

Signed-off-by: maximsmol <maximsmol@gmail.com>

* fix: schema transformer error

Signed-off-by: maximsmol <maximsmol@gmail.com>

* fix: test

Signed-off-by: maximsmol <maximsmol@gmail.com>

* fix: merge issue

Signed-off-by: maximsmol <maximsmol@gmail.com>

* fix: union + annotated behavior

Signed-off-by: maximsmol <maximsmol@gmail.com>

* Regenerate requirements

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Bump requirements in plugins

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Handle nested Annotated

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Leave TODO re: strucutured dataset type castability

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Remove mention to flyteidl@union_type in doc-requirements

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Linting

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Use tempfile.mkdtemp to create a temporary directory for local data persistence.

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Revert "Use tempfile.mkdtemp to create a temporary directory for local data persistence."

This reverts commit 861b157.

Signed-off-by: eduardo apolinario <eduardo@eduardos-MacBook-Pro.local>

* Force temporary file to not be deleted in test

Signed-off-by: eduardo apolinario <eduardo@eduardos-MacBook-Pro.local>

* Regenerate papermill dev-requirements

Signed-off-by: eduardo apolinario <eduardo@eduardos-MacBook-Pro.local>

* Remove duplicate code

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Put a lower bound on the pip version installed in CI

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

Co-authored-by: Kevin Su <pingsutw@apache.org>
Co-authored-by: Emirhan Karagül <emirhan350z@gmail.com>
Co-authored-by: Eduardo Apolinario <653394+eapolinario@users.noreply.github.com>
Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
Co-authored-by: Kevin Su <pingsutw@gmail.com>
Co-authored-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Co-authored-by: Ketan Umare <16888709+kumare3@users.noreply.github.com>
Co-authored-by: Lisa <30621230+aeioulisa@users.noreply.github.com>
Co-authored-by: Niels Bantilan <niels.bantilan@gmail.com>
Co-authored-by: Stef Nelson-Lindall <bethebunny@gmail.com>
Co-authored-by: Snyk bot <github+bot@snyk.io>
Co-authored-by: Dan Rammer <hamersaw@protonmail.com>
Co-authored-by: bstadlbauer <11799671+bstadlbauer@users.noreply.github.com>
Co-authored-by: Bernhard Stadlbauer <bstadlbauer@gmx.net>
Co-authored-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai>
Co-authored-by: Samhita Alla <aallasamhita@gmail.com>
Co-authored-by: Yuvraj <code@evalsocket.dev>
Co-authored-by: Zach Palchick <palchicz@users.noreply.github.com>
Co-authored-by: Snyk bot <snyk-bot@snyk.io>
Co-authored-by: Kenny Workman <31255434+kennyworkman@users.noreply.github.com>
Co-authored-by: ggydush-fn <69013027+ggydush-fn@users.noreply.github.com>
Co-authored-by: SmritiSatyanV <94349093+SmritiSatyanV@users.noreply.github.com>
Co-authored-by: Matthew Griffin <1matthewgriffin@gmail.com>
Co-authored-by: eduardo apolinario <eduardo@eduardos-MacBook-Pro.local>
myz540 pushed a commit to ProjectAussie/flytekit that referenced this pull request Apr 11, 2022
* Add support union type

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Fixed test

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Fixed test

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Fixed lint

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Fixed tests

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Updated tests

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Update Union to use tagged Unions

Signed-off-by: maximsmol <maximsmol@gmail.com>

* Update to use string tags (part 1)

Signed-off-by: maximsmol <maximsmol@gmail.com>

* Working implementation, update tests

Signed-off-by: maximsmol <maximsmol@gmail.com>

* Fixes, more tests

Signed-off-by: maximsmol <maximsmol@gmail.com>

* Fix incorrect unwrapped literal-union matching, update test

Signed-off-by: maximsmol <maximsmol@gmail.com>

* Fix duplicate tag handling, add tests for collections containing unions

Signed-off-by: maximsmol <maximsmol@gmail.com>

* Fix type hint test

Signed-off-by: maximsmol <maximsmol@gmail.com>

* Add implicit wrapping union type tests

Signed-off-by: maximsmol <maximsmol@gmail.com>

* Add union ambiguity tests

Signed-off-by: maximsmol <maximsmol@gmail.com>

* Fixup tests, make TypeTransformerFailed compatible with all old exception types

Signed-off-by: maximsmol <maximsmol@gmail.com>

* Fixup models + add tests

Signed-off-by: maximsmol <maximsmol@gmail.com>

* Implement changed design

Signed-off-by: maximsmol <maximsmol@gmail.com>

* Fix PR comments

Signed-off-by: maximsmol <maximsmol@gmail.com>

* Remote entrypoint serialize (flyteorg#733)

Signed-off-by: Emirhan Karagül <emirhan350z@gmail.com>
Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Fix lint error in remote.py (flyteorg#755)

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Support enum in dataclass (flyteorg#753)

* Add support enum in dataclass

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Update test

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Fixed lint

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Fix subworkflow and launch plan FlyteRemote behavior (flyteorg#751)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Extras: Shell task (flyteorg#747)

Signed-off-by: maximsmol <maximsmol@gmail.com>

* Add support FlyteSchema in dataclass (flyteorg#722)

* schema in dataclass

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Added tests

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Fixed lint

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Updated tests

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Updated tests

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Fixed lint

Signed-off-by: Kevin Su <pingsutw@apache.org>

* updated

Signed-off-by: Kevin Su <pingsutw@apache.org>

* updated

Signed-off-by: Kevin Su <pingsutw@apache.org>

* updated

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Remove workflow_execution.py (flyteorg#758)

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Get raw input/output from remote execution (flyteorg#675)

* [wip] for feast demo

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* clean up a bit

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* add a test and move where constructor is called

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* remove unneeded import

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* add a part of a test

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* Added tests

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Fixed lint

Signed-off-by: Kevin Su <pingsutw@apache.org>

* typo

Signed-off-by: Kevin Su <pingsutw@apache.org>

Co-authored-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Co-authored-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Fix mypy errors in flytekit/types (flyteorg#757)

Signed-off-by: Lisa <aeioulisa@gmail.com>
Signed-off-by: Kevin Su <pingsutw@apache.org>

* Remote client failed to fetch FlytePickle object (flyteorg#764)

* Fetch pickle value from flytekit remote

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Fix tests

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Remove default value

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Add support FlyteFile in dataclass (flyteorg#725)

* Add support Flyte File and directory in dataclass

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Fixed tests

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Fixed tests

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Updated

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Updated

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Fixed tests

Signed-off-by: Kevin Su <pingsutw@apache.org>

* add task_resolver arg to @task decorator (flyteorg#765)

Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Copy metadata into map task from underlying (flyteorg#766)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Support for delayed annotations (flyteorg#760)

Signed-off-by: Stefan Nelson-Lindall <stef@stripe.com>
Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Complex dataclass unit tests (flyteorg#773)

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>
Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* remote_source lost on serialization of @dataclass_json with FlyteFile (flyteorg#774)

* remote_source lost on serialization of @dataclass_json with FlyteFile

Signed-off-by: Kevin Su <pingsutw@apache.org>

* updated tests

Signed-off-by: Kevin Su <pingsutw@apache.org>

* updated tests

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Single-task execution FlyteRemote sync (flyteorg#778)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Logging updates (flyteorg#775)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* fix: doc-requirements.txt to reduce vulnerabilities (flyteorg#779)

The following vulnerabilities are fixed by pinning transitive dependencies:
- https://snyk.io/vuln/SNYK-PYTHON-LXML-2316995

Signed-off-by: maximsmol <maximsmol@gmail.com>

* Add cache_serialize parameter to tasks (flyteorg#673)

* added cache_serialize parameter for basic tasks

Signed-off-by: Daniel Rammer <daniel@union.ai>

* fixed typo

Signed-off-by: Daniel Rammer <daniel@union.ai>

* updated flyteidl version

Signed-off-by: Daniel Rammer <daniel@union.ai>

* changed flyteidl requirements everywhere

Signed-off-by: Daniel Rammer <daniel@union.ai>

* remove flyteidl version requires in setup.py so we can use develop

Signed-off-by: Daniel Rammer <daniel@union.ai>

* removed flyteidl git repos from a variety of requirements packages

Signed-off-by: Daniel Rammer <daniel@union.ai>

* updated variable discovery_serializable to cache_serializable

Signed-off-by: Daniel Rammer <daniel@union.ai>

* updated requirements

Signed-off-by: Daniel Rammer <daniel@union.ai>

* fixed TaskMetadata _cache_serializable variable name

Signed-off-by: Daniel Rammer <daniel@union.ai>

* propgating cache_serialize parameter through to tasks

Signed-off-by: Daniel Rammer <daniel@union.ai>

* added cache_serializable to SdkRawContainerTask

Signed-off-by: Daniel Rammer <daniel@union.ai>

* fixing cache_serializable variable propogation issues

Signed-off-by: Daniel Rammer <daniel@union.ai>

* added documentation

Signed-off-by: Daniel Rammer <daniel@union.ai>

* added unit tests for cache_serialize metadata

Signed-off-by: Daniel Rammer <daniel@union.ai>

* linter added spaces in unit tests

Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* When using the `task` and `workflow` decorator, correctly wrap the fu… (flyteorg#780)

* When using the `task` and `workflow` decorator, correctly wrap the function

This enables tooling such as docstring search tools to unwrap the object and show the correct docstring.

Signed-off-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai>

* Remove blackshark copyright header

Signed-off-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai>

* Fix broken great expectations test

Signed-off-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai>

* Add test for stacked decorators

Signed-off-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai>

Co-authored-by: Bernhard Stadlbauer <bstadlbauer@gmx.net>
Co-authored-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Add option to flyte-cli for specifying root certificate (flyteorg#783)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Add validation check to cacert switch (flyteorg#787)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Remove pytz constraint (flyteorg#786)

* Remove pytz constraint from setup.py

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Regenerate requirements files

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Put pytz back

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* make requirements.txt

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Lint: remove f-string misuse (flyteorg#788)

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Pyarrow greater than 4.0.0 (flyteorg#790)

* pyarrow>=4.0.0

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Regenerate requirements.

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Support python 3.10 (flyteorg#791)

* [wip] Support python 3.10

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Add Dockerfile.py310

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Stringify python version

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Skip flytekit-modin plugin tests on 3.10

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Add 3.9 and 3.10 to list of supported version in plugins

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Comment why flytekit-modin is not running on 3.10 and disable fail-fast in plugin tests

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* add `with_overrides` to map task (flyteorg#794)

* add with_overrides

Signed-off-by: Samhita Alla <aallasamhita@gmail.com>

* remove Resources

Signed-off-by: Samhita Alla <aallasamhita@gmail.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* bump docsearch version (flyteorg#805)

Signed-off-by: Samhita Alla <aallasamhita@gmail.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* update docs for new navbar theme (flyteorg#806)

Signed-off-by: maximsmol <maximsmol@gmail.com>

* fix requirment.txt github issue (flyteorg#810)

Signed-off-by: Yuvraj <code@evalsocket.dev>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Add sphinx panels (flyteorg#815)

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Schema overhaul (flyteorg#785)

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* Parent workflow serialization fails when calling a launch plan with fixed inputs (flyteorg#814)

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Fix sagemaker plugin (flyteorg#817)

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Remove legacy API (flyteorg#807)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* Add Bigquery plugin (flyteorg#789)

* Add bigquery plugin

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Update dependency

Signed-off-by: Kevin Su <pingsutw@apache.org>

* update get_custom

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Add structured dataset

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Add structured dataset

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Updated comment

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Add BQ in GA

Signed-off-by: Kevin Su <pingsutw@apache.org>

* alphabetical order

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Fixed flytekit-papermill ImportError (flyteorg#818)

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Add support for string-format-like sytax for shell task (flyteorg#792)

* POC: Add support for f-string like sytax for shell task

This commit is a proof of concept adding f-string like syntax for
shell_tasks. This supports using nested types for script inputs, such
as data classes. This change was motivated by the desire to combine
shell_tasks that have multiple inputs with map_tasks which only support
tasks with a single input.

This commit is only a starting point, since it makes some changes to the
shell_task API (adds a template_style field), and modifies some of the default
behavior for ease of implementation (e.g. throwing an error when there
are unused input arguments).

Signed-off-by: Zach Palchick <palchicz@zymergen.com>

* Drop support for old/regex style for doing string interpolation

Signed-off-by: Zach Palchick <palchicz@zymergen.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Expose configured RawOutputPrefix during execution (flyteorg#813)

* Expose configured RawOutputPrefix during execution

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Remove sdk_runnable.py and spark_task.py

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Add SecretsManager back to old import location (flyteorg#820)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Add some tests (flyteorg#819)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Fixed flaky spark test (flyteorg#821)

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* fix: plugins/flytekit-greatexpectations/requirements.txt to reduce vulnerabilities (flyteorg#823)

The following vulnerabilities are fixed by pinning transitive dependencies:
- https://snyk.io/vuln/SNYK-PYTHON-IPYTHON-2348630

Signed-off-by: maximsmol <maximsmol@gmail.com>

* fix: plugins/flytekit-papermill/requirements.txt to reduce vulnerabilities (flyteorg#825)

The following vulnerabilities are fixed by pinning transitive dependencies:
- https://snyk.io/vuln/SNYK-PYTHON-IPYTHON-2348630

Signed-off-by: maximsmol <maximsmol@gmail.com>

* fix: requirements-spark2.txt to reduce vulnerabilities (flyteorg#826)

The following vulnerabilities are fixed by pinning transitive dependencies:
- https://snyk.io/vuln/SNYK-PYTHON-IPYTHON-2348630

Signed-off-by: maximsmol <maximsmol@gmail.com>

* fix: requirements.txt to reduce vulnerabilities (flyteorg#824)

The following vulnerabilities are fixed by pinning transitive dependencies:
- https://snyk.io/vuln/SNYK-PYTHON-IPYTHON-2348630

Signed-off-by: maximsmol <maximsmol@gmail.com>

* Intratask checkpointing  (flyteorg#771)

* wip - intratask checkpointing

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* sync checkpointer with tests

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* Checkpinter in entrypoint

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* checkpoint in progress

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* wip

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* Intratask checkpointer

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* Checkpoint updated

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* Intra-task checkpointing

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* Test and entrypoint updated

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* lint fixed

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* test fixes

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* fmt

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* updated entrypoint

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* updated

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* update

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* print

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* updated

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* SyncCheckpointer working

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* updated

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* update

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* fixed import problems

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* fixed test

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* fixed imports

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* fixed lints and errors

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* lint fix

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* addressed comments

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Support reading subset column (flyteorg#822)

* Support StructuredDatasetDecoder read subset column

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Added tests

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Fixed tests

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Fixed typo

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Updated tests

Signed-off-by: Kevin Su <pingsutw@apache.org>

* [pr into flyteorg#822]  (flyteorg#827)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

Signed-off-by: Kevin Su <pingsutw@apache.org>

* [pr into flyteorg#822] Final update to structured dataset column subsetting (flyteorg#828)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

Co-authored-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* Fix spark regression (flyteorg#830)

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Update argument setting for in fast registered, dynamically generated, pod tasks (flyteorg#835)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* `ctx` Context can be used within shell tasks - to access context vars and secrets (flyteorg#832)

* Adding context to a substitutable parameter in shell task

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* Support for secrets in context

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>

* addressed comments

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Expose Checkpoint as a top-level interface in flytekit (flyteorg#839)

Signed-off-by: Ketan Umare <ketan.umare@gmail.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Parse duration field from flyteidl to `flytekit.models.execution.ExecutionClosure` (flyteorg#829)

* Parse duration field from flyteidl to `flytekit.models.execution.ExecutionClosure`

Signed-off-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai>

* Add test for execution closure

Signed-off-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai>

* Add tests to Flyte remote

Signed-off-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai>

* Split execution test into with output and with error

Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net>

Co-authored-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Gate new Structured Dataset feature & remove old config objects (flyteorg#831)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* Fixing out of order for conditional outputs (flyteorg#843)

Signed-off-by: maximsmol <maximsmol@gmail.com>

* Set default values to map task template (flyteorg#841)

* Set sane defaults in map task templates

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Remove unused method

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Put ArrayJob.from_dict back

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Define parallelism=0 as unbounded

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Remove special case to handle 0

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Regenerate requirements files - dependencies dropping support for python 3.7 (flyteorg#838)

* Regenerate requirements files

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Put restrictions on numpy and pandas versions

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Use --use-deprecated=legacy-resolver

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* use pip==22.0.3 everywhere

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Remove --use-deprecated=legacy-resolver

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Relax click

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Regenerated plugins requirements

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* TypeAnnotation (flyteorg#759)

* feat:  support for annotated simple + list

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* feat: addition of annotation att to

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* feat: core  obj

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* feat:  proto model

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* feat: testing suite

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: more stable typing introspection

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: strip legacy

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: explicitly allow only one annotation

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* feat: direct type transformer tests

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: there and back test

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: typing_extensions for get_origin

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: more semantic list generic unwrap

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: tmp requirements file with custom idl

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: nits

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* feat: semantic error for unsupported complex literals

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: but

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* feat: more tests ;)

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: imports

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: complex annotations

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: temp requirements files for unit tests

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: lint bug

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: tmp setup.py

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: use typing_extensions

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: typing_extensions for annotated

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: typing_ext

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: plugin tmp requirements

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: bump requirements

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: doc requirements

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: whitespace

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: bump flytekit

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: numpy version

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: lint

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: pandas version

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: bump requirements

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: test import

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: flake8 lint

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: merge

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: requirements

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: requirements

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: lint

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: papermill req

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* fix: req

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

* Remove singleton from structured dataset transformer engine (flyteorg#848)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: Kevin Su <pingsutw@apache.org>

* Assign input and output to FlyteWorkflowExecution (flyteorg#842)

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Add reference entities to FlyteTask and FlyteLaunchPlan (flyteorg#850)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Fix fast registration error (flyteorg#851)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

Signed-off-by: Samhita Alla <aallasamhita@gmail.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Add support for local execute in pod task (flyteorg#852)

Signed-off-by: ggydush-fn <greg.gydush@freenome.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Add anonymous retry (flyteorg#854)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* fix: formatting, linting, typing_extensions

* fix: do not use SDK types

Signed-off-by: maximsmol <maximsmol@gmail.com>

* fix: update test comment

Signed-off-by: maximsmol <maximsmol@gmail.com>

* fix: also check literal type castability when tags match

Signed-off-by: maximsmol <maximsmol@gmail.com>

* Point flyteidl to maxim's fork in CI and requirements files

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* [Core feature] Add Raw AWS Batch Task (flyteorg#782)

* Init plugin

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Fixed lint

Signed-off-by: Kevin Su <pingsutw@apache.org>

* address comment

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Fixed typo

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Updated AWS config

* Fixed lint

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Added comment

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Update config

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Fixed tests

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Fixed tests

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Fixed tests

Signed-off-by: Kevin Su <pingsutw@apache.org>

* use pyflyte execute

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Fixed tests

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Fixed tests

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Added comment

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Add structured dataset encoder/decoder in fsspec plugin (flyteorg#849)

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Delete unnecessary auth configuration (flyteorg#858)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Fixed format alias in Flytekit docs (flyteorg#844)

* Fixed format alias

Fixed docs for file format alias that weren't rendered properly.
A warning popped up stating 'ignore'  is deprecated, and to use 'ignore_paths' instead.
Signed-off-by: SmritiSatyanV <smriti@union.ai>

* Bump idl (flyteorg#862)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* Updated authoring.rst (flyteorg#863)

* Updated authoring.rst

Added directive
Rephrased sentence

* Fixed build error

Signed-off-by: SmritiSatyanV <smriti@union.ai>

* test-build-1

Signed-off-by: SmritiSatyanV <smriti@union.ai>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Updated authoring.rst (flyteorg#866)

Added the directive correctly
Signed-off-by: SmritiSatyanV <smriti@union.ai>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Change docs for HTMLPage type to say HTMLPage instead of PNGImage (flyteorg#868)

Signed-off-by: maximsmol <maximsmol@gmail.com>

* Revisit StructuredDatasetDecoder interface (flyteorg#865)

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Remove legacy mentions in contributing guide (flyteorg#870)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Add GCS protocol in the structured dataset (flyteorg#869)

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Make fetched entities callable within workflows (flyteorg#867)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Caching of offloaded objects (flyteorg#762)

* Remove flyteidl from install_requires

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Expose hash in Literal

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Set hash in TypeEngine

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Modify cache key calculation to take hash into account

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Opt-in PandasDataFrameTransformer

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Add unit tests

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Iterate using a flyteidl branch

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Regenerate requirements files

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Regenerate requirements files

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Move _hash_overridable to StructureDatasetTransformerEngine

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Move HashMethod to flytekit.core.hash

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Fix `unit_test` make target

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Split `unit_test` make target in two lines

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Add assert to structured dataset compatibility test

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Remove TODO

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Regenerate plugins requirements files pointing to the right version of flyteidl.

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Set hash as a property of the literal

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Install plugins requirements in CI.

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Add hash.setter

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Install flyteidl directly

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Revert "Regenerate plugins requirements files pointing to the right version of flyteidl."

This reverts commit c2dbb54.

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* wip - Add support for univariate lists

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Add support for lists of annotated objects

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Revamp generation of cache key (to cover case of literals collections and maps)

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Leave TODO for warning

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Revert "Add support for lists of annotated objects"

This reverts commit 4b5f608.

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Revert "wip - Add support for univariate lists"

This reverts commit adaa448.

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Remove docstring

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Add flyteidl>=0.23.0

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Remove mentions to branch flyteidl@add-hash-to-literal

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Bump flyteidl in plugins requirements

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Regenerate plugins requirements again

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Restore papermill/requirements.txt

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Point flytekitplugins-spark to the offloaded-objects-caching branch in papermill tests

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Set flyteidl>=0.23.0 in papermill dev-requirements

Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* FlyteRemote fetch of conditional nodes (flyteorg#772)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Removed root logger (flyteorg#871)

* Removed root logger

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Updated logger

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Fixed lint

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Updated log level

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Updated logger

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Updated logger

Signed-off-by: Kevin Su <pingsutw@apache.org>

* Updated logger

Signed-off-by: Kevin Su <pingsutw@apache.org>

* lint fixed

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* Fix flytekit_compatibility/test_schema_types.py test

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
Signed-off-by: maximsmol <maximsmol@gmail.com>

* make lint

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* fix: annotated type conversion error

Signed-off-by: maximsmol <maximsmol@gmail.com>

* fix: _are_types_castable based on tests

Signed-off-by: maximsmol <maximsmol@gmail.com>

* fix: test failing if using random order

Signed-off-by: maximsmol <maximsmol@gmail.com>

* Merge branch 'master' into maximsmol/union_type

Signed-off-by: maximsmol <maximsmol@gmail.com>

* fix: merge issue

Signed-off-by: maximsmol <maximsmol@gmail.com>

* fix: requirements

Signed-off-by: maximsmol <maximsmol@gmail.com>

* fix: schema transformer error

Signed-off-by: maximsmol <maximsmol@gmail.com>

* fix: test

Signed-off-by: maximsmol <maximsmol@gmail.com>

* fix: merge issue

Signed-off-by: maximsmol <maximsmol@gmail.com>

* fix: union + annotated behavior

Signed-off-by: maximsmol <maximsmol@gmail.com>

* Regenerate requirements

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Bump requirements in plugins

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Handle nested Annotated

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Leave TODO re: strucutured dataset type castability

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Remove mention to flyteidl@union_type in doc-requirements

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Linting

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Use tempfile.mkdtemp to create a temporary directory for local data persistence.

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Revert "Use tempfile.mkdtemp to create a temporary directory for local data persistence."

This reverts commit 861b157.

Signed-off-by: eduardo apolinario <eduardo@eduardos-MacBook-Pro.local>

* Force temporary file to not be deleted in test

Signed-off-by: eduardo apolinario <eduardo@eduardos-MacBook-Pro.local>

* Regenerate papermill dev-requirements

Signed-off-by: eduardo apolinario <eduardo@eduardos-MacBook-Pro.local>

* Remove duplicate code

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Put a lower bound on the pip version installed in CI

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

Co-authored-by: Kevin Su <pingsutw@apache.org>
Co-authored-by: Emirhan Karagül <emirhan350z@gmail.com>
Co-authored-by: Eduardo Apolinario <653394+eapolinario@users.noreply.github.com>
Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
Co-authored-by: Kevin Su <pingsutw@gmail.com>
Co-authored-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Co-authored-by: Ketan Umare <16888709+kumare3@users.noreply.github.com>
Co-authored-by: Lisa <30621230+aeioulisa@users.noreply.github.com>
Co-authored-by: Niels Bantilan <niels.bantilan@gmail.com>
Co-authored-by: Stef Nelson-Lindall <bethebunny@gmail.com>
Co-authored-by: Snyk bot <github+bot@snyk.io>
Co-authored-by: Dan Rammer <hamersaw@protonmail.com>
Co-authored-by: bstadlbauer <11799671+bstadlbauer@users.noreply.github.com>
Co-authored-by: Bernhard Stadlbauer <bstadlbauer@gmx.net>
Co-authored-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai>
Co-authored-by: Samhita Alla <aallasamhita@gmail.com>
Co-authored-by: Yuvraj <code@evalsocket.dev>
Co-authored-by: Zach Palchick <palchicz@users.noreply.github.com>
Co-authored-by: Snyk bot <snyk-bot@snyk.io>
Co-authored-by: Kenny Workman <31255434+kennyworkman@users.noreply.github.com>
Co-authored-by: ggydush-fn <69013027+ggydush-fn@users.noreply.github.com>
Co-authored-by: SmritiSatyanV <94349093+SmritiSatyanV@users.noreply.github.com>
Co-authored-by: Matthew Griffin <1matthewgriffin@gmail.com>
Co-authored-by: eduardo apolinario <eduardo@eduardos-MacBook-Pro.local>
Signed-off-by: Mike Zhong <mzhong@embarkvet.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants