Data plugins yt #565
Merged
Data plugins yt #565
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… output variables (#557) Signed-off-by: Sean Lin <[email protected]> Co-authored-by: wild-endeavor <[email protected]>
* document flytekit remote module Signed-off-by: cosmicBboy <[email protected]> * execute method supports project/domain override - add debug logging info for project/domain override - update docs Signed-off-by: cosmicBboy <[email protected]> * update doc-requirements.txt Signed-off-by: cosmicBboy <[email protected]> * address nits Signed-off-by: cosmicBboy <[email protected]>
Signed-off-by: wild-endeavor <[email protected]>
…e object itself Signed-off-by: wild-endeavor <[email protected]>
Signed-off-by: wild-endeavor <[email protected]>
Signed-off-by: wild-endeavor <[email protected]>
Signed-off-by: wild-endeavor <[email protected]>
Signed-off-by: wild-endeavor <[email protected]>
Signed-off-by: wild-endeavor <[email protected]>
Codecov Report
@@ Coverage Diff @@
## data-plugins #565 +/- ##
================================================
- Coverage 85.44% 85.09% -0.36%
================================================
Files 374 381 +7
Lines 29040 29488 +448
Branches 2332 2379 +47
================================================
+ Hits 24813 25092 +279
- Misses 3597 3751 +154
- Partials 630 645 +15
Continue to review full report at Codecov.
|
kumare3
pushed a commit
that referenced
this pull request
Jul 31, 2021
Signed-off-by: wild-endeavor <[email protected]> Signed-off-by: Ketan Umare <[email protected]>
kumare3
added a commit
that referenced
this pull request
Aug 19, 2021
* Name attribute added Signed-off-by: Ketan Umare <[email protected]> * Data persistence work in progress Signed-off-by: Ketan Umare <[email protected]> * updated data persistence api Signed-off-by: Ketan Umare <[email protected]> * tried my hand at docs - failed! Signed-off-by: Ketan Umare <[email protected]> * wip - pr into #559 (#565) Signed-off-by: wild-endeavor <[email protected]> * simplify code Signed-off-by: wild-endeavor <[email protected]> Signed-off-by: Ketan Umare <[email protected]> * Addressing comments Signed-off-by: Ketan Umare <[email protected]> * black format Signed-off-by: Ketan Umare <[email protected]> * Fixed lint errors Signed-off-by: Ketan Umare <[email protected]> * Added unit tests Signed-off-by: Ketan Umare <[email protected]> * fixed lint and added more tests Signed-off-by: Ketan Umare <[email protected]> * updated Signed-off-by: Ketan Umare <[email protected]> * delete this Signed-off-by: Ketan Umare <[email protected]> * Revert "delete this" This reverts commit efa5234. Signed-off-by: Ketan Umare <[email protected]> * Fsspec DataPersistence plugin (#596) * Support for all fsspec data-persistence plugins Signed-off-by: Ketan Umare <[email protected]> * fixed lint Signed-off-by: Ketan Umare <[email protected]> * FSSpec debugging and defaults Signed-off-by: Ketan Umare <[email protected]> * FSSpec implementation Signed-off-by: Ketan Umare <[email protected]> * updated fsspec plugin Signed-off-by: Ketan Umare <[email protected]> * minio support Signed-off-by: Ketan Umare <[email protected]> * persist fix Signed-off-by: Ketan Umare <[email protected]> * fixed Signed-off-by: Ketan Umare <[email protected]> * updated Signed-off-by: Ketan Umare <[email protected]> * client args Signed-off-by: Ketan Umare <[email protected]> * upload to a dir Signed-off-by: Ketan Umare <[email protected]> * updated Signed-off-by: Ketan Umare <[email protected]> * fixed paths Signed-off-by: Ketan Umare <[email protected]> * updated Signed-off-by: Ketan Umare <[email protected]> * updated Signed-off-by: Ketan Umare <[email protected]> * update put logic Signed-off-by: Ketan Umare <[email protected]> * updated Signed-off-by: Ketan Umare <[email protected]> * remove tempfile Signed-off-by: Yee Hing Tong <[email protected]> * unit tests added Signed-off-by: Ketan Umare <[email protected]> * lint fix Signed-off-by: Ketan Umare <[email protected]> * test Signed-off-by: Yee Hing Tong <[email protected]> * try another setting Signed-off-by: Yee Hing Tong <[email protected]> * just fsspec Signed-off-by: Yee Hing Tong <[email protected]> * missing s Signed-off-by: Yee Hing Tong <[email protected]> * Addressed comments Signed-off-by: Ketan Umare <[email protected]> * test fix Signed-off-by: Ketan Umare <[email protected]> * Great Expectations FlyteFile modifications Signed-off-by: Samhita Alla <[email protected]> * remove shutil package Signed-off-by: Samhita Alla <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]> Co-authored-by: Samhita Alla <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR into #559
Merged in master so way more diffs to look at than usual.
The main change is that I changed the
DataPersistencePluginsobject to hold the DataPersistence classes instead of instances of those classes.Also changed the base class to take a prefix. This is so that when running in local mode, we can have two different DiskPersistence objects pointing to different prefixes.